Aiming at the problems of low accuracy and slow speed in the detection of surface defects of lithium batteries by traditional methods, an improved YOLOv4 algorithm is proposed. Firstly, a dilated convolution is used to replace the conventional convolution in the CSPDarknet-53 backbone network, which improves detection of defects of different scales. Secondly, an efficient channel attention is inserted into the neck network to adaptively select the size of the one-dimensional convolution kernel to reduce the complexity and computations of the model. Finally, a conditional convolution is fused in classification and bounding box regression to improve the network performance, and the data set is expanded to solve the problem of network training overfitting caused by too few defective samples. The experimental results show that the improved YOLOv4 algorithm can effectively detect the surface defects of lithium batteries and improve the ability to identify and locate surface defects of lithium batteries. The mean average precision of the improved algorithm is 93.46%, which is 3.03% higher than the original algorithm.