Hand gesture recognition becomes a significant way of the future humancomputer interaction gradually. The method of finger number detection under complicated background has been studied in this paper. Firstly, through the analysis of the different hand area extraction methods under various color space, a new extraction method based on the HSCrCb color space is proposed in this paper, and a more clear and complete hand area is extracted use this method. Secondly, the convex defect, the hand’s shape and the angle between different fingers are used to build a hand model and detect the finger number. This method can recognize the according finger number under different gestures accurately. The algorithm is verified under the Qt and OpenCV platform, and achieves a fine result, which means the algorithm have a good reference and application value.