The fastener plays a critical role in ensuring the stability and safety of modern engineering structures. However, factors such as vibration, shock, temperature changes, and others can cause fasteners to become loose, thereby compromising the stability and safety of the overall structure. Currently, fastener inspection is primarily done manually, and there is no widely adopted automatic method for real-time detection of fastener looseness. We proposed a fastener looseness inspection method based on digital shearing speckle pattern interferometry (DSSPI) and convolutional neural network (CNN). The key distinction between our proposed method and traditional vision-based methods is that our approach can identify loose fasteners without relying on direct visual observation of the fasteners themselves. Our approach involved setting up a DSSPI system to capture shearing speckle patterns, and the CNN model automatically extracted features from these images and classified them to detect loose fasteners. Principle-validation experiments were conducted to verify the feasibility of the methodology. The trained model achieved a classification accuracy of 92.57%, with a resolution of 0.04 mm and a completion time of only 2.03 ms for a single judgment. That is quite accurate, more sensitive, and much faster than the three-dimensional vision-based method. Furthermore, our proposed method has strong robustness to ambient light, while the incorporation of Random Erasing technology enhances robustness to part occlusion or damage of the measured surface. Most notably, our method can detect loose fasteners before deformation, a capability that conventional vision-based methods lack. These findings demonstrate the tremendous potential of our method for real-time inspection of fastener looseness.