The noise characteristics of high-power operated erbium- doped fiber amplifiers are evaluated experimentally and theoretically. Optical high-power post-amplifiers are indispensable for the transmitter in high-bit rate optical inter-orbit communication (OIC) links applications in which repeaters is not applicable. Thus, the effects of optical amplification noise on communication performance must be analyzed to establish reliable optical links. The dominant noise component and the influence on receiver sensitivity are discussed for various kinds of OIC links in our study. In addition, intensity modulation direct detection (IM-DD) scheme utilizing optical amplifiers are evaluated for OIC link applications. Experimental demonstrations of coherent detection are also performed with optical fiber amplifiers. The performance of IM-DD and coherent optical communication system are discussed in case of operation under the heavy amplified spontaneous emission (ASE) noise environment induced by the optical amplifiers. The requirements for optical band-pass filters to reduce ASE noise are clarified for IM-DD scheme and the superior feature of coherent systems are shown in aspects of tolerance for bandwidth of optical filter. High-sensitive receiving operation under 50 photons/bit at bit error rate of 10-9 is achieved in the system with a 500 mW optical booster-amplifier assuming GEO-GEO links (approximately 40,000 km).© (2000) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.