The quality of phase unwrapping is an important part of 3D reconstruction in grating projection. After the phase modulation of the deformed fringes, The folded phase needs to be unwrapped throughout the region ,which affects the result of 3D reconstruction of the measured object. Aiming at the problem that the phase unwrapping method based on discrete least square cannot get the correct expansion results in the case of interference, so put forward using ranks expansion method to improve the discrete cosine least square method, reducing the error in the process of phase unwrapping. The result of simulation and experimental show that the method improved can raise the quality of phase unwrapping, which is of practical significance to improve the accuracy of 3D reconstruction.