This paper propose an improved yolov4 algorithm for detecting fatigue driving. First, under the transfer learning, the weight of VOC dataset has been trained as pre weight for training. Then in the training, convolution is added before and after the SPP structure of feature pyramid in the frame to improve the extraction of deep features. And introduce the dilated convolution to increase convolution output receptive field and the ability to obtain image location information. Experimental results show that the map value of the improved yolov4 algorithm during the test is 97.29%, 1.98% higher than raw yolo v4 algorithm, the detection of eye parts increased by 6%. Add the frame delay to the detection, for avoiding other behaviors affecting the results and reducing the probability of misjudgment.