Regarding the perennial safety production problems that factories constantly encounter, such as the strict prohibition of smoke and fire in the workshop area, the need for constant attention to the behavioral safety of workers, and whether workers wear masks in adverse working condition scenarios, an improved worker behavior and fire detection algorithm FDH-DETR based on RT-DETR was proposed. Firstly, through the fusion of the Deep Faster feature depth fusion module and FasterNet, the number of parameters and the amount of computation of the algorithm were reduced. Secondly, through the DRBC3 module size convolution kernel conversion mechanism, the inference cost of the model was decreased. Finally, through the HiLo-AIFI high-low frequency scale withinfeature interaction module, the extraction ability of high-low frequency features was enhanced. Experimental results indicate that the improved algorithm achieved an average accuracy of 93.8%, a reduction of 31.6% in parameters, a reduction of 61.4% in computation, and an FPS of 150 frames per second. Inference experiments were conducted in real working condition scenarios, verifying the effectiveness of the algorithm.