In view of the difficulty in detecting hot spots of photovoltaic panels in power stations in China, combined with UAV inspection technology, a fast detection method of hot spots of photovoltaic panels based on deep convolutional neural network was proposed. Firstly, a photovoltaic panel recognition model was designed. The Yolov4 backbone feature extraction network was replaced by the lightweight MobileNetV2 network, and the standard 3×3 convolution in PANet was replaced by the deeply separable convolution, which could realize the rapid recognition of photovoltaic panels from infrared images. In order to quickly identify hot spots and solve the problem of reflective noise of photovoltaic panels, MobileNetV2 network is introduced into deeplabv3 + model, improve the target loss caused by sampling and the cross entropy loss function is modified to dice loss function to further improve the segmentation accuracy. The experimental results show that the method can accurately identify hot spots of photovoltaic panels, with an accuracy of 99. 56% and a detection speed of 22. 1 frames per second. The hot spot segmentation accuracy of photovoltaic panel recognition reaches 95. 99%, MIoU reaches 85. 58%, and the detection speed is 24. 5 frames per second. This method can meet the needs of photovoltaic panel fault detection.