The fundamental performance limits and channel capacity of optical communications systems operating over the free space channel will be examined using quantum detection theory. The performance of the optimum quantum receiver for on-off keying (OOK) and optical binary phase shift keying (BPSK) is first examined as a pure state (no noise) problem. The classical capacity of the binary symmetric channel for these two modulation schemes is then evaluated for the optimum quantum receiver by making use of the concept of quantum measurement states. The performance of M-ary pulse position modulation, which requires a product state representation, is evaluated along with the performance of certain 'dense signal sets.' Performance comparisons with classical techniques shows over 5 dB improvement in some cases when quantum detection is employed. As a further application of the quantum detection theory, the capacity of the binary channel with on-off keyed modulation and quantum detection is evaluated, and shown to exceed the capacity obtained with classical photon counting.© (2002) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.