It is often convenient to represent a wavefront aberration by the superposition of several aberration modes, for example, using the set of Zernike polynomials. In many practical situations the total aberration can be accurately represented by a small number of such modes. It is therefore desirable to be able to measure directly the modal content of the wavefront. The modal wavefront sensor allows us to do just this. This sensor can be applied to closed-loop aberration correction in adaptive systems and to direct absolute measurement of modal aberration coefficients. One implementation offers the possibility of an adaptive optics system where we have disposed altogether of a separate wavefront sensor. We present here extension of the modal wavefront sensing theory and explore the options for optimization of the design. We investigate the linear measurement range of the sensor and the performance in closed-loop systems.© (2003) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.