Atomic force microscopy (AFM) and interference microscopy are two methods often used to measure roughness, but the probe size is very different and they respond to different physical properties (hardness and reflectivity). In earlier work we have shown that the limited resolution of interference microscopy can be approximated by the longwave components of a Gaussian filtering of the AFM image with a cut-off wavelength λc a little larger than the wavelength of light. This description was valid for smooth and hard surfaces with good reflectivity such as polished metal surfaces (Rq < 10 nm). In this paper we extent the analysis to directly measure the effective cut-off wavelength λc = 2600 nm for a particular interference microscope based on the profiles of grooves with a period of 3000 nm, a depth of (104 ± 1) nm and vertical sidewalls. To validate the measured parameter λc, the same area on a polished hip joint prosthesis was measured by both an AFM and the particular interference microscope. Without a Gaussian filtering of the AFM image the appearance and calculated roughness of the images were significantly different (Ra = 1.7 nm, Rq = 2.2 nm versus Ra = 1.0 nm, Rq = 1.2 nm). However, using the measured cut-off wavelength the visual appearance of the longwave components of the AFM image and the interference microscope image are almost identical and the calculated roughness is equal. This strongly suggests that an effective cut-off wavelength can be measured and used to give consistency between the different methods in the range where they overlap.© (2003) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.