As the core component of fiber optic gyroscope (FOG), the winding quality of the fiber optic coils (FOC) is critical to the accuracy of the FOG. In order to ensure the accuracy and efficiency of the fiber winding system, a defect detection method based on the improved YOLO algorithm is proposed. The model uses the Shufflenetv2 network to replace the convolution layer and pooling layer in the YOLO backbone network, which improves the feature extraction ability of the network; the Focus module is added to improve the training speed; the K-means clustering algorithm is used to cluster the original anchor boxes, and obtain a prediction frame suitable for fiber winding defect detection, the accuracy of defect detection is improved; at the same time, the loss function is modified, the CIOU is used to calculate the positioning loss, and the Focal Loss is used as the confidence loss and classification loss function to speed up the network convergence; and data enhancement is carried out to enhance the generalization ability of the network. It is concluded from the experiments that our proposed method is able to detect FOC winding defects with an average accuracy of 99.63%, which is an improvement of 2.06% over the original YOLO algorithm, and a detection speed of 91 fps. This will guarantee the practical application of the FOC winding system.