In human computer interaction, the interaction between humanhands and machine is the most common way, so gesture interaction research is one of the focuses of humancomputer interaction research today. Based on Leap Motion, this paper studies the threedimensional space object detection and tracking, and proposes a new algorithmLBFGSEKF algorithm. The algorithm is based on the EKF method proposed by reducing noise, through the method of the LBFGS algorithm is used to replace the Hessian matrix in each iteration instead of the EKF algorithm, which consumes memory and reduces the computational rate, resulting in poor realtime, thus forming a new target tracking algorithm. The simulation results show that by this new algorithm LBFGSEKF for gesture recognition, the error can be reduced and the realtime performance of target tracking can be improved.