In this paper a real-time signal process method is proposed for a new grating-based sensor LDGI (Liner Diffraction Grating Interferometer). The LDGI signal shows much higher frequency than conventional optical encoders. When the grating moves 416nm the LDGI system generates one wave cycle. The waveforms have some typical distortions: DC offsets, amplitude variation and phase error. For real-time measurement, in every millisecond the waveforms are normalized to eliminate DC offsets and amplitude variation. Then the phase error is corrected with an operation of coordinate rotation. After that, with zero-pass counting and phase subdivision the displacement can be worked out. If the displacement is too short to generate a whole wave cycle, which means there are not enough data to work out the signal distortions, an optimization method for sine curve fitting is used to calculate the displacement. If the displacement is shorter than 20nm, a group of empirical values are used in signal process. Experiments show that with the proposed method, the measurement repeatability of LDGI is within 5nm. Especially when this system is used for nanoscale measurement the uncertainty can hardly be detected with a laser interferometer. Besides, the proposed method helps to get higher resolution. Experiments show that the minimum displacement that the system can detect is 1nm.© (2009) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.