The invention discloses a two-stage sludge drying and energy recovering system and a drying process thereof. The two-stage sludge drying and energy recovering system comprises a sludge storage chamber. The two-stage sludge drying and energy recovering system is characterized in that an exhaust port of the sludge storage chamber is connected with a deodorization reactor through a first fan; a sludge outlet of the sludge storage chamber is connected with a sludge inlet end of a drier through a sludge pump; a sludge forming machine is arranged at the sludge outlet end of the drier; and a belt drier is arranged at the sludge discharge hole of the sludge forming machine. The system adopts a two-stage drying manner; and the sludge of which the moisture content is 70-85% is finally dried into granular sludge of which the solid content is 70-90%. The system is high in heat transfer efficiency, small in carrier gas quantity, and low in heat loss, and does not generate dust at all; the energy recovering system is efficient and energy-saving; the energy saving rate achieves about 30%, and the drying rate and the energy-saving rate are obviously improved from the aspect of actual use effect.