In order to address the leakage detection problem of traditional water supply pipeline, in this paper, we propose a pipeline leakage detection algorithm based on the sparse and lightweight convolutional neural network technology. First, the sound signal leaked from the pipeline is collected by the sound sensors. After preprocessing operations such as stereo conversion, resampling, and length alignment, it is converted to a mel spectrogram. Then, a sparse and lightweight convolutional neural network model is proposed to perform feature extraction and leak detection on the mel spectrogram. Due to the sparse and time-delayed characteristics of sound feature images, we introduce the Inception structure to improve the feature extraction ability. In addition, to deploy the proposed model to the edge side, a lightweight convolutional neural network based on SqueezeNet is designed to reduce model parameters and thus reduce the model complexity. Massive experimental results show that the proposed pipeline leakage detection algorithm has less computation complexity and better recognition accuracy.