The unbalance of the wind turbine rotor is unavoidable during its operation, and it will affect its reliability and reduce its lifetime for a long time. Aiming at the problem that the non-linearity and instability of unbalance of the wind turbine rotor, the traditional frequency domain transform spectrum analysis method has certain limitations. Therefore, this paper studies a method for detecting unbalance of the rotor based on variation mode decomposition. Combined with the actual operating data of the wind farm for comparative analysis, this method can decompose the complex multi-signal into several signals, and can effectively extract the fault characteristics. Compared with the traditional frequency domain variation method, it has a great advantage. The results indicate that the rotor aerodynamic unbalance will cause the 1-P vibration significantly increase, and the greater of the deviation of the pitch angle, the greater the axial vibration changes.