Bird nest encroachment on overhead transmission lines can cause safety hazards to the power equipment on the towers, which may indirectly affect the stable operation of the whole power system. Aiming at the current overhead transmission line bird's nest detection model in the complex scene as well as the small target scene detection accuracy is not high, the detection efficiency is low, the model is complex and other problems. This study proposes a lightweight overhead transmission line bird's nest detection network based on YOLOv5s framework. Firstly, the YOLOv5s feature extraction network is reconstructed using Fasternet in the backbone part to reduce the model complexity and improve the operation speed; then the ConvMixer layer is embedded in the feature fusion network part, and the structural design of ConvMixer helps to better capture the relationship between space and channel in the feature information and improve the model's detection ability for small targets; finally, the Finally, the ODConv module is introduced in the feature fusion network part, so that the feature map sent to the detection head contains more effective features to improve the model's detection performance for complex scenes and small targets. The experimental results show that compared with the YOLOv5s baseline model, the computational amount and model volume are reduced by 86% and 72%, the average accuracy rate reaches 96.4%, and the detection speed reaches 104.2 frames/s, which verifies the effectiveness and feasibility of the improved model in this paper.