Whatever quality settings are choosen, the screen resolution (640x480 for Nightfall, 800x600 for the unsupported NightfallMEGA) remains the same. You can choose thousands or millions of colors on screen. Quality really refers to detail, and how much effort goes into drawing. Lower quality draws faster, giving you a smoother game. Higher quality draws more slowly, but there is more detail, more special effects, and more care taken in drawing each pixel on the screen. Software gradually ramps up the special effects drawn. If you select hardware, the window size alters, but effects such as colored fog and glows are used at all quality levels if available.
Low and Medium quality show graphics in a smaller window. This allows cards with only 2 megabytes of VRAM to use 3D hardware. Medium quality is different in that colored fog and glow effects are used if available.
Fine, Good, Excellent, and Amazing cover more of the screen, so you can see more detail. All these modes need at least 4 megabytes of VRAM to use 3D hardware in thousands of colors. If you are using NightfallMEGA or either game in millions of colors you usually need 6 megabytes of VRAM (it may vary from chip to chip).
Fine uses fog which fades to black. All other modes use colored fog. Good, Excellent and Amazing use more detailed textures. They look crisper. If hardware supports it, they also use filtering, so textures appear less blocky. Finally, in software, the difference between Excellent and Amazing is in the care taken in drawing. There is no difference in millions of colors, but in thousands you will see that fog and shadows are drawn much more smoothly in Amazing quality, but it is also a lot slower. There is no difference between Excellent and Amazing when using 3D hardware.