Fiber nonlinearity is the important determinant of applying orthogonal frequency division multiplexing (OFDM) in optical transmission systems. In this paper, the fiber nonlinearity impairments in 40Gb/s coherent optical (CO-OFDM) system are post-compensated by a new method of fiber nonlinearity post-compensation (FNPC). FNPC located before the CO-OFDM receiver includes spectrum inversion (SI) and a subsequent 80km compensation high nonlinear fiber (HNLF), in which SI is based on four wave mixing (FWM) in semiconductor optical amplifier (SOA), not necessary to be inserted into the middle of link. The fiber nonlinearity impairments in transmission link are post-compensated after SI by transmission through the HNLF with large nonlinearity coefficient. Simulating results show that the nonlinear threshold (NLT) (for Q >10 dB) can be increased by almost 2.5 dB and maximum Q factor can increase about 1.2 dB for single-channel 40Gb/s CO-OFDM system with optical dispersion compensation. For 50-GHz channel spacing WDM systems a benefit of about 1.1 dB is still possible in NLT, equating to a 0.7 dB increase in maximum Q.© (2010) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.