This is a rough outline of the different options for saving DDS files using the Nvidia Photoshop DDS plugin. I wanted to know what each did, and couldn't find any online documentation for all of them, so I am creating this to help myself and other artists with the same questions. I'll update this as I receive feedback about anything I got wrong, or any additional information.
The dialog box in question. The plugin is available here. The PSD with each compression type as an individual layer so you can compare them is available here. I've skipped the formats that crashed photoshop (the four "palette" entries), as well as the floating point formats, as my image wasn't a floating point to begin with.
Source Image for this series of examples (a dove I captured in Paris) The Targa source file is 1,025 kb. Targas are uncompressed. The alpha is in full 8 bit grayscale.
First option is DXT1, with or without 1 bit alpha. DXT1 file is 171 kb. DXT 1 is an 8 : 1 compression ratio. The alpha is 1 bit (meaning no grayscale, either a pixel is black or it's white) You can see some artifacts in any smooth areas, but nothing too bad on diffuse maps. DXT1 works less well with normal maps. It can create visible artifacts, especially at the terminator shadow, where you can see the squares used in the compression algorithm.
DXT3 has an 8 bit explicit alpha. The DXT3 file is 342 kb. DXT3 is a 4:1 compression ratio. The alpha is 8 bit, but you can clearly see banding. DXT3 is much better for normal maps because of the lesser compression. The difference in diffuse, specular, etc is a less noticeable improvement over DXT1.
DXT5 has an 8 bit interpolated alpha. The DXT5 file is 342 kb. DXT5 is a 4:1 compression ratio. The alpha is 8 bit, but much smoother than DXT3. I honestly cannot tell why someone would use DXT3 over DXT5. It seems like you get the same size, but a smoother alpha without banding. If I figure out any real differences, I'll update this portion. As of now it looks like your real choice is between DXT1 or DXT5, with DXT3 out of the picture completely.
4:4:4:4 has a 4 bit alpha. The 4:4:4:4 file is 683 kb. 4:4:4:4 is basically 4 bits per channel (hence the name). I'm not sure where this format would come in handy, but it has no compression artifacts.
1:5:5:5 has a 1 bit alpha. The 1:5:5:5 file is 683 kb. This format basically steals 3 bits from the alpha channel to distribute to the R,G, and B channels. This results in a smoother diffuse, but a sharper alpha. Again, no compression, so no artifacts, but the file size is much to large to consider in most instances.
5:6:5 has no alpha. The 5:6:5 file is 683 kb. Similar to the 4:4:4:4 and 1:5:5:5, but takes it a step further and eliminates the alpha all together to offer an extra bit to the green channel. The 5:6:5 is getting interesting though. With the extra bit, it looks very similar to the targa source in quality, but is 90% the filesize of the targa (saved as a 24 bit without alpha).
8:8:8:8 has 8-bit alpha. The 8:8:8:8 file is 1,366 kb. This is the uncompressed 32-bit targa (8 bit alpha) equivalent DDS format. However it's larger than the original targa in filesize!
8:8:8 has no alpha. The 8:8:8 file is 1,025 kb. This is the uncompressed 24-bit targa (no alpha channel) equivalent DDS format. However it's larger than the original targa in filesize!
5:5:5 has no alpha. The 5:5:5 file is 683 kb. This format is identical to 1:5:5:5 in every way, except it has no alpha at all. There is no situation I can think of where you would use 5:5:5 over 1:5:5:5. It is a compression artefact-free format, but like 1:5:5:5 it has banding problems in the diffuse.
V8U8 has a red and green channel only. The V8U8 file is 683 kb. This format is for storing two grayscale masks (to control emissive and bump for example) This format has no compression so the red and green are identical to the targa source. I'm also pretty sure that currently ATI cards cannot use this format, but NVidia cards can.
CXV8U8 has no alpha. The V8U8 file is 683 kb. This format is not compressed. I'm unsure of the specifics of this format, but I know it's for normal maps. I believe it's somehow only saving two channels worth of information, despite the fact it's showing three. It uses a Microsoft/Nvidia only algorithm for reconstructing the missing blue channel (the displayed blue channel is not how standard normal map blue channels look) at a very low cost (in terms of gpu power). I'm fairly sure than neither ATI, nor the PS3 is able to take advantage of this format.
A8 is an 8 bit alpha only. The A8 file is 342 kb. As far as I can tell, this is a lossless alpha format. 1,1,1 RGB is fully transparent, 128,128,128 is one step below fully opaque, and 0,0,0 is fully opaque in this format. You still keep your original alpha in standard black to white format, but this is how the map is stored. A8 has solid black for all RGB formats, and keeps itself in the alpha channel.
Q8W8V8U8 has 8-bit alpha. The 8:8:8:8 file is 1,366 kb. I'm not certain what this format is for. It's apparently the same as 8:8:8:8 only this format is "signed". Visually it's brighter, but otherwise lossless like 8:8:8:8.
A8L8 has a red and alpha channel only. The A8L8 file is 683 kb. This format is for storing two grayscale masks (to control alpha and Luminance) As far as I can tell, this is a lossless format. 1,1,1 RGB is fully transparent, 128,128,128 is one step below fully opaque, and 0,0,0 is fully opaque in this format. To set up this file, you have your RGB as a grayscale image to control alpha (transparency), and your alpha channel a grayscale to control Luminance.
L8 is an 8 bit alpha only. The L8 file is 342 kb. 1,1,1 RGB is fully transparent, 128,128,128 is one step below fully opaque, and 0,0,0 is fully opaque in this format. You still keep your original Luminance in standard black to white format, but this is how the map is stored. L8 keeps itself in the only channel, which is the only difference between itself and A8.
DXT5_NM has an 8 bit interpolated alpha. The DXT5 file is 342 kb. This format is compressed, since there is only two channels, but it's the filesize of the DXT5 with three channels, I'm assuming this is less compressed, nearer to 2:1 compressions.. I'm unsure of the specifics of this format, but I know it's for normal maps. I believe it's somehow only saving two channels worth of information, leaving the green channel with more information than the red (which is in the alpha). It uses an algorithm for reconstructing the missing blue channel at a very low cost (in terms of gpu power). I'm fairly sure than neither ATI, nor the PS3 is able to take advantage of this format.
PDF version here