Currently, the brain-computer interface based on steady-state visual evoked potential (SSVEP) has received wide attention in human-computer collaboration, and the existing deep learning classification methods oriented to the phase and frequency information of SSVEP signals still suffer from problems such as underutilization of information. To address the above problems, this paper proposes a deep neural network model for SSVEP signal classification based on the idea of transfer learning, which takes the complex vectors after the fast Fourier transform as inputs, and convolves the real and imaginary part vectors of each lead to learn the corresponding phase-frequency characteristics. The model is divided into two parts: the first part uses the statistical commonality among all subjects to obtain the global phase-frequency feature module for phase and frequency information; the second part uses the trained global phase-frequency feature module to initialize the local phase-frequency feature module, and fine-tunes the training parameters through further reinforcement learning of the local phase-frequency feature module in order to reduce the individual differences between each subject. The experiments are conducted using the BETA public dataset, and the results show that the proposed model in this paper has an average accuracy and an average information transfer rate of 89.98% and 71.80 bit/min, respectively, when the time window is 1.5s, which are both better than the existing methods. This study provides a brand new idea for deeply mining and utilizing the phase and frequency information in EEG signals, which is of great significance for promoting the development of brain-computer interface technology.