Nonlinear distortion is one of the major obstacles in DWDM systems with enhanced spectral efficiencies. In this paper several approaches to address the issue of nonlinear impairments by means of digital signal processing are discussed. Firstly, implementation-efficient and novel intra-channel nonlinear compensation schemes are proposed; one is based on digital pre-distortion at the transmitter end and the other is based on digital back-propagation at the receiver end. The virtues of the two approaches and implications to various applications are discussed; the pre-distortion technique is in particular advantageous with QPSK modulation format; on the other hand, the improved version of digital back-propagation is attractive in transceivers with adaptive or variable modulation/demodulation. Second, digital signal processing algorithms to counteract inter-channel nonlinearities, namely cross-phase modulation, are discussed; nonlinear polarization crosstalk canceller (NPCC) is proposed for mitigating the impact of nonlinear-induced fast polarization crosstalk in dual-polarization systems (in the speed beyond MHz), which is too fast to be tracked by ordinary polarization demultiplexing algorithms; improvement to the carrier phase recovery circuit and its combination with NPCC are even more useful for further performance improvement. Numerical and experimental data are introduced to support the above discussions.© (2012) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.