An object-face based RGB-D image stitching optimization method is studied to solve the problem of low resolution, small range, and high noise of RGB-D depth images, which is not conducive to three-dimensional reconstruction. First, the RGB-D images are pre-processed and aligned; the feature points are extracted and roughly matched using the algorithm. Then, the mismatching is eliminated by the corresponding relationship of the same object face under different perspectives studied in this paper. Finally, the RGB-D images with wide viewing angles and three-dimensional models are obtained based on the homography matrix. Three algorithms, Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB), are used for comparison experiments. The experimental results show that 41%, 29%, and 52% erroneous matches are removed on distorted and revolved images, and the Root Mean Square Error is reduced by 5%, 27%, and 33% respectively. In the scaled image, 53%, 57%, and 51% erroneous matches are removed, and the Root Mean Square Error is reduced by 14%, 17%, and 28%, which improves the matching accuracy and verifies the feasibility of this method.