Performance of MLSE-based receivers in lightwave systems with nonlinear dispersion and amplified spontaneous emission noise

Maximum likelihood sequence estimation (MLSE) has been proposed in earlier literature to combat the effects of nonlinear dispersion in intensity modulation/direct detection (IM/DD) optical channels. We develop a theory of the bit error rate (BER) of MLSE-based IM/DD receivers operating in the presence of nonlinear intersymbol interference (ISI) and amplified spontaneous emission (ASE) noise. Numerical results confirm the predictions of our theory. Based on the new analysis, we also investigate the log-likelihood ratio (LLR) of the received signal yielded by a soft-input/soft-output (SISO) front-end decoder. Our study shows that traditional channel codes designed for transmissions over AWGN channels, in combination with SISO front-end detectors, achieve an asymptotically optimal performance in transmissions over IM/DD optical channels.