Aiming at the problem that the structure from motion algorithm needs to change the camera position in the process of camera photographing, the relative motion relationship between the target object and the camera is used to obtain image sequences containing multi angle views of target objects. A structure from relative motion algorithm is designed and implemented by locating the region of the target object in the image sequence and constraining the applicable range of feature point extraction. The experimental results show that SFRM algorithm is suitable for the photographing mode of fixed position camera. SFRM can increase the proportion of matching pairs located in the target object area from 32% to 84% of the total matching pairs. Compared with the traditional SFM algorithm, the time efficiency is improved by 38.29%. The available scenarios of SFRM algorithm is successfully expanded.