Optical proximity correction (OPC) methods are resolution enhancement techniques (RET) used extensively in the semiconductor industry to improve the resolution and pattern fidelity of optical lithography. Recently, a set of generalized gradient-based OPC optimization methods have been developed to solve for the inverse lithography problem under coherent illumination. Most practical lithography systems, however, operate under partially coherent illumination due to non-zero width sources and off-axis illumination from spatially extended sources. OPC methods derived under the coherent illumination assumption fail to account for the nonlinearities of partially coherent illumination and thus perform poorly in the latter scenario. This paper focuses on developing gradient-based binary mask optimization methods which account for the inherent nonlinearities of partially coherent systems. Two nonlinear models are used in the optimization. The first relies on a Fourier representation of the nonlinear model which approximates the partially coherent system as a sum of coherent systems. The second model is based on an average coherent approximation which is computationally faster. In order to influence the solution patterns to have more desirable manufacturability properties, wavelet regularization is added to the optimization framework. The advantages and limitations of both models in the inverse lithography problem are discussed and several illustrative simulations are presented.© (2008) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.