Abstract:The new generation of video compression standard (H.266/VVC) provides 67 prediction modes in intra-frame prediction, which greatly improves the coding efficiency, but also brings extremely high computational complexity. This paper proposes a fast algorithm for intra-mode decision-making based on deep learning. First, for the problem of the size and shape of the block after the size of the coding block is divided, the extracted brightness block is preprocessed, and the block size and quality are guaranteed through random cropping, resampling, and convolutional neural network (CNN) upsampling. . Then the CNN architecture is carefully designed to reduce the complexity of intra prediction, and it is proposed to use the current coding block, the adjacent reference block and the residual block as the input of the network to convert the rate-distortion decision-making process into a classification problem and reduce unnecessary Pattern traversal. In order to train the proposed deep learning network, this paper establishes a model decision data set based on the characteristics of H.266. Experimental results show that compared with VTM10.0, the algorithm proposed in the article reduces the coding time by 39.56%~43.45% on average, which effectively reduces the computational complexity of coding, while the rate-distortion performance remains basically unchanged, which is comparable to the latest references. The overall performance has also been improved.