The high-frequency measurement data of underground vibration signals can record more specific details about the dynamic response of drilling tools, which is helpful for analyzing and diagnosing abnormal vibrations underground. However, the high-frequency measurement generates a large amount of measurement data, resulting in significant storage pressure for underground vibration measurement equipment. The proposed method uses compressed sensing technology to selectively collect and store sparse underground vibration data and then recover high-frequency measurement results through a signal reconstruction algorithm. In the process of realizing this method, an innovative method of constructing a layered Fourier dictionary against spectrum leakage is proposed, and an improved OMP signal reconstruction algorithm based on layered tracking is researched and realized, which greatly reduces the time required for signal recovery. Simulation and experimental test results demonstrate the method′s effectiveness, achieving a system compression ratio of 18.9 and a reconstruction error of 52.1 dB. The proposed method may greatly reduce the data storage pressure of the measuring equipment in the underground, and provides a new way to obtain high-frequency measurement data of underground vibration.