The rainbow format Uniform Spectrum is definitely the most colorful around. Although the USPEC files doesn't contain any color value.

A USPEC file is basically just a list of positions. All source image pixels get sorted by color value and stored as position id, one integer value or 4 byte per pixel.

The image is usually build of as many colors as pixels, up to 16.777.216.