Aiming at the issue of small target defect detection and multi-size defect detection in X-ray images, a weld defect detection algorithm based on improved Faster RCNN is proposed. Firstly, the algorithm utilizes ResNet50 and feature pyramid network as the backbone network of Faster RCNN for detecting defects of different sizes on multiple feature maps. Then, the background subtraction layer is added before the backbone network to reduce the interference of the image background on the small target defects. Then, the three-branch region proposal network layer refine the predictions of candidate boxes in the original region proposal network layer, thereby reducing the number of candidate boxes and optimizing the detection speed. Finally, the number of convolutional layers in the network is fine-tuned to enhance the network’s feature extraction ability. The experimental results show that the improved network has a mean average precision of 83.09% and a single image detection speed of 20.8 ms. Compared to the network before improvement, the preset anchor boxes are increased by 10 779, and the mean average precision is increased by 19.43%, while the detection speed is only decreased by 3.1 ms. The improved network effectively improves the detection effect of small target defects and multi-size defects while maintaining detection speed.