Aiming at the problem of high feature dimension and low recognition rate in speech emotion recognition, In this paper, we propose a genetic algorithm for feature dimension reduction and construct a multilayer SVM classifier based on binary tree structure for the recognition of speech emotion. First, the common emotional features are extracted after preprocessing the speech signal. As there are many features and redundant data, the genetic algorithm is used to optimize the extracted features. Then, the hierarchical SVM classification model of the binary tree structure is trained by using the most discriminative features. The experimental results demonstrate the effectiveness of the proposed speech emotion recognition scheme on the Berlin emotion corpus containing 7 emotions.