Based on multi-factor fusion, this paper proposes a new driving early warning method. First, combining the existing fatigue factors, a risk assessment criterion is proposed based on multiple factors, which overcomes traditional single-factor methods’ limitations in application and shortcomings of being vulnerable to external interference. Secondly, a detection network is proposed based on SSD, in which the backbone network VGG is replaced with the advanced MobileNetV3, fast target detection is achieved with a modified NMS layer, and finally multi-task detection is realized with newly designed multi-task detectors and loss functions. After transfer-learning with the pre-training weights, the tested detection accuracy rate is 95.7%, and the speed is 41fps, which is accurate and real-time.