The present disclosure is a novel design of a polarization based stereoscopic display system that efficiently utilizes the optical energy from three primary color solid state light sources of random polarization, combines the three primary colors into a full color beam of a single polarization state to enable passive separation of the two image channels. The high optical energy efficiency is achieved by splitting each primary color light is into two orthogonal polarization states. The single polarization state of the combined full color image beam is achieved by employing a spectrally selective light beam combiner or X-cube. By making the optical configuration of sub-module basically identical and sharing a number of optical components among color and image channels, the size and cost is reduced. By compensating the depolarization effect that is introduced by folding mirror(s), the cross talk between the two displayed stereo images is minimized.