In order to optimize the extraction of pearl color features and improve the accuracy of pearl color detection, a shadow detection algorithm based on K-means clustering and local gradients is proposed. The results show that the algorithm can accurately detect the shadow of pearls in pearl images and eliminate the interference of shadows on the extraction of color features. In the Lab color space, a color feature extraction based on the area of pearl echo gallery effect is proposed, GA-SVM is used as the pearl color identification method, and a secondary color detection strategy is proposed to determine the pearl color category through two color detections. The comparison experimental results show that the accuracy rate of pearl body color detection is 100%, and the accuracy rate of pearl color detection is 98.7878%.