An innovative constellation-shaped three-dimensional 14-quadrature amplitude modulation QAM (3D-14QAM) mapping scheme based on regular tetrahedron is proposed. The non-uniform 14QAM signals are designed by Huffman code and mapped onto the designed 3D constellation based on regular tetrahedron. The simulation demonstration is carried out successfully by VPI software in a 4  ×  100  Gb  /  s wavelength division multiplexing (WDM) polarization division multiplexed coherent optical communication system with a transmission distance of 300 km. For WDM system, the simulation results show that the proposed scheme is capable of achieving about 1.8 dB OSNR gain over the traditional 3D-16QAM scheme at the hard-decision forward error correction (HD-FEC) limit of 3.8  ×  ^{10  −  3}. In addition, the proposed 3D-14QAM can obtain 1.1 dB OSNR gain compared with the traditional 3D-16QAM for single-channel scenario at the HD-FEC. The outstanding performance indicates that the proposed scheme can provide a performance improvement for both single carrier system and WDM system. And benefited from the ability of suppress nonlinearity, the proposed scheme can obtain a larger gain in the WDM system than a single carrier system.