Aiming at the problems of low measuring efficiency and large error in the current process of measuring loader loading and unloading action time, a measuring system of loader loading and unloading action time based on machine vision is designed. Firstly, the video image sequence of loader loading and unloading motion is preprocessed, and then the shape template matching and motion prediction are used to find the position of the bucket in the image and obtain the center of mass pixel coordinates of the bucket. The motion track of loader bucket is established by using the pixel coordinates of the bucket′s center of mass. Then, the trajectory dividing points of the four loading and unloading motion stages are located, and the starting and ending frames and their frame numbers of the four motion stages are determined by using the trajectory dividing points. Finally, the frame number of the start frame and the end frame is converted into time by frame rate for calculation, so as to realize the automatic measurement of the loading and unloading action time of the loader. The experimental results show that the measured deviations of the measuring system for the loading and unloading action time of the loader is less than 05 s, the measuring results is stable and effective. On the other hand, because motion prediction is used to estimate the bucket position, the time consumption for shape template matching is reduced by 1645%.