In response to low efficiency and high leakage rate caused by different sizes of transmission tower components and more defects in the current defect detection of power line towers, this paper proposes a Dynamic Position Query-Guided Multi-Scale Instance Segmentation method and a Graph Feature Memory-based Defect Detection method. The proposed instance segmentation method extracts multi-scale aerial image features, selects low-resolution pixels with the highest attention scores from the features, maps them to the corresponding positions in high-resolution features, and incorporates a bounding box detector to enhance the segmentation accuracy of power transmission towers. In the defect detection algorithm, a learnable graph feature descriptor is introduced, a memory bank is constructed to extract key elements for more accurate sample feature extraction, thereby improving defect detection efficiency. The power transmission tower defect detection method presented in this paper is compared with other state-of-the-art algorithms on two self-constructed defect detection datasets, the box_APand mask_AP of instance segmentation saw significant improvements of 7.6% and 0.5%, respectively, compared to Mask2Former. The AUROC indicator of defect algorithm was 7.3% and 1.6% higher than the second-best algorithm for the two datasets, and the F1-Score was improved by 6.7% and 6.9%, respectively. These results strongly demonstrate the outstanding performance of our algorithm in the transmission tower defect detection.