In this paper, a variety of color models for face detection system are examined, detailed analysis of the interfixed threshold segmentation method, Gaussian skin color model and Otsu, respectively, on the Caltech library of color face experiments. The experimental results show that under different lighting conditions and the background environment, three color segmentation model is not consistent, but the main factors is the lighting conditions. By analyzing large amounts of data, get the lighting conditions of the three models to achieve the best results when dividing. Combined with Y channel information, present a new face detection algorithm with three color models automatically switching based on the lighting conditions. Compared to the past face detection system with a signal skin color model, the new system is more stable and adapt to the environment.