Images from GUI Bloopers, by Jeff Johnson. Used without permission.
Group boxes in most GUI toolkits can be set to display a variety of different border styles, and drop shadows.
Variation A: Mixing etched-in and etched-out group boxes. This usually occurs because different programmers develop different parts of the application, with no design guidelines, no communication, and no management oversight.
Variation B : Inconsistent border styles. Using entirely different styles of group boxes in different windows in an application, with no systematic difference in meaning for different styles. Again, this is usually due to a lack of communication and coordination between programmers.
Group boxes in an application or family of applications should all have the same border style. By far the most common border style for group boxes is etched (in or out). Either is OK, as long as you are consistent throughout the application or family of applications.
The best way to make sure you do the "right thing" is to simply follow the style guide of the GUI platform on which your software will be running..