It is a challenging and meaningful task to achieve more accurate emotion recognition. Because of the complex diversity of emotions, it is difficult to measure emotions comprehensively and objectively with a single mode of EEG signal. Therefore, a multi-modal lightweight hybrid model PCA-MWReliefF-GAPSO-SVM is proposed in this paper. The hybrid model consists of a PCA-MWReliefF feature channel selector and a GAPSO-SVM classifier. Electroencephalogram (EEG), electromyographic signal (EMG) and temperature signal (TEM) were used for emotion recognition. Through many experiments on DEAP public data set, the classification accuracy of 97.500 0%, 95.833 3% and 95.833 3% in titer dimension, wake dimension and four categories were obtained, respectively. The experimental results show that the proposed mixed model can improve the emotion recognition accuracy and is significantly better than the single mode emotion recognition. Compared with the recent similar work, the hybrid model proposed in this paper has the advantages of higher accuracy, less computation and fewer channels, and is easier to be applied in practice.