Progress in photomask technology, exposure system flexibility and lithographic computation has enabled the practical exploration of jointly optimized lithographic imaging variables. The interaction between certain components, such as mask and illuminator, allows co-optimized solutions to frequently achieve better results when compared with sequential, single factor optimizations. In this work, we focus on the automated co-optimization of embedded phase shift mask transmission factor and exposure system illumination source profile for improving image based merit functions. Algorithm descriptions are provided and the critical interaction of optimization parameters with mask layout is highlighted. Our co-optimization algorithm is exercised on the more challenging random logic case and the concept of manipulating or restricting layout conditions to improve the achieved merit function is studied. Finally, suggestions for the experimental prototyping of solutions are provided and an assessment on deviating from the industry accepted 6% transmission discussed.© (2005) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.