Constellation optimization for coherent optical channels distorted by nonlinear phase noise

We consider the design of amplitude phase-shift keying (APSK) constellations, targeting their application to coherent fiber-optical communications. Phase compensation is used at the receiver to combat nonlinear phase noise caused by the Kerreffect. We derive the probability density function of the post-compensated observation for multilevel constellations. Optimal APSK constellations in terms of symbol error probability (SEP) are found assuming a two-stage detector. Performance gains of 3:2 dB can be achieved compared to 16-QAM at a SEP of 10^-2. We optimize the number of rings, the number of points per ring, as well as the radius distribution of the constellation. For low to moderate nonlinearities, radius optimization only yields minor improvements over an equidistant spacing of rings. In the highly nonlinear regime, however, a smaller SEP can be achieved by “sacrificing” the outer ring of the constellation, in favor of achieving good SEP in the remaining rings.