To address the challenge of effectively recognizing unknown modulation types in signal modulation recognition applications using deep learning models, this paper introduces a novel recognition model based on zero-shot learning and autoencoders for open set signal modulation recognition. Features of the modulation signals are extracted through an autoencoder, which incorporates cross-entropy loss, center loss, and reconstruction loss to ensure effective separation of features across different modulation types. Further, open set recognition of modulation signals is conducted based on the distribution of features in the feature space. Additionally, by incorporating the reconstructed signals back into training, the model′s recognition accuracy is significantly enhanced. Experimental results demonstrate that the proposed model not only distinguishes unknown classes effectively, achieving an unknown class recognition rate of 80%, but also maintains a stable known class recognition rate of approximately 95%, outperforming traditional open set recognition methods.