The invention relates to a fault diagnosis method and system of a planetary gear transmission system, belonging to the fields of a fault diagnosis technique and a signal processing and analyzing technique. The fault diagnosis method comprises the following main fault diagnosis steps: (1) processing acquired vibration signals of the planetary gear transmission system by utilizing a homomorphic filtering technical method, and separating signals comprising fault features and external excitation background noise signals to reduce the complexity of data; (2) carrying out frequency spectrum reconstruction on the signals comprising the fault features after homomorphic filtering processing by adopting complex cepstrum transform, and extracting the fault feature information in the vibration signals of the planetary gear transmission system; and (3) diagnosing the running state of the planetary gear transmission system to be the normal state or the fault state by combining a vibration mechanism of the planetary gear transmission system, utilizing a classifier of a support vector machine (SVM) and taking the time domain statistics at different states and peak value energy after frequency spectrum reconstruction as fault feature vectors.