In order to improve the recognition accuracy and recognition speed of face recognition in embedded devices, a Ghostnet-based lightweight face recognition algorithm called Ghostfacenet is proposed. Firstly, a fixed number of intrinsic features are generated by pre-determined convolution. To address the problem of computationally intensive convolutional operations, linear operations with low computational cost are used instead of convolutional operations to generate a series of feature information associated with intrinsic features. Secondly, the Ghostfacenet-Bottleneck is designed based on the Ghost module in Ghostnet and the depthwise separable convolution. And the Ghostfacenet lightweight convolutional neural network is constructed from Ghostfacenet-Bottleneck. Finally, the Arcface loss function and the Airface loss function are combined to further increase the intra-class compactness of faces as well as inter-class differences. It also allows for better convergence and generalization capabilities of lightweight models. The experimental results show that Ghostfacenet is 11.08 times, 8.57 times, 2.75 times and 2.82 times faster than Resnet50, Efficientnet, MobilenetV2 and Mobilefacenet respectively in embedded devices. This is a significant improvement in operational efficiency without a significant reduction in recognition performance and is ideal for embedded devices with limited resources.