Optimization of Branch Metric Exponent and Quantization Range in MLSE Receivers for Duobinary Systems

We analyze by simulation the performance of an optically amplified uncompensated duobinary system using a 64-state maximum-likelihood sequence estimation (MLSE) receiver (Rx) based on a Euclidean branch metric with variable postdetection nonlinear distortion exponent. We found that the optimum exponent depends on the accumulated dispersion, the Rx analog-to-digital converter (ADC) resolution, and signal clipping. On the other hand, we found performance to be weakly dependent on the exponent value. When using a finite number of ADC resolution bits, drastic signal clipping proved very beneficial in improving the performance of the MLSE processor. Assuming three resolution bits, with joint clipping and exponent optimization, uncompensated transmission at 10.7 Gb/s over 550 km of standard single-mode fiber could be achieved with essentially no penalty with respect to back-to-back.