Intuitive design of the lithographic process becomes increasingly complicated in the regime of off-axis illumination and optical proximity correction. Therefore, new optimization procedures have to be introduced to facilitate the search for ideal process settings. This paper proposes mutual optimization of illumination and mask geometries using an automatic optimization approach based on a genetic algorithm. As presented elsewhere, this optimization procedure has been applied to different mask representations. It has been found that a blend of a fully parameterized and a pixel-based representation, i.e., a rectangle representation, leads to highly innovative solutions, but can still maintain an acceptable convergence behavior. This representation is revisited and its main principles and limitations are shortly discussed. The main focus of this paper is on a refinement of the source geometry representation. In previous versions, the general illumination setup had to be prespecified. Merely its parameters (e.g, inner and outer radius of annular illumination, number, offset, and radius of poles for multipole illumination) were optimized. In this work, the source is represented by a sector/track definition, which allows different sections of the illumination to have different transmission values. The obtained illumination geometry is transferred into a pixel-based representation, processable by the utilized Fraunhofer IISB in-house lithography simulator. The illumination shapes achieved with the proposed approach can, for example, be produced by diffractive optics elements (DOEs). Various merit criteria determine the imaging performance of both the mask and the source settings. As the merit or fitness function plays one of the central roles in the proposed optimization scheme, individual fitness criteria and their transformation into an objective function are revisited and shortly explained. New results for both dense and chain contact hole layouts, and a comparison with former results validate the proposed approach and illustrate its further potentials.© (2005) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.