Maximum Likelihood Sequence Estimation of Binary Sequences Transmitted Over Bandlimited Nonlinear Channels

The problem of designing and evaluating the performance of a maximum likelihood sequence receiver for binary PSK transmission over bandlimited nonlinear channels is considered in this paper. The effects of intersymbol interference followed by AM/AM and AM/PM conversions are taken into account while optimizing the performance in the presence of white Gaussian noise. A new representation for the output of a bandpass nonlinearity is given when the input consists of a carrier signal modulated by a sum of binary overlapping pulses. The structure of a maximum likelihood sequence receiver for a bandlimited nonlinear channel is derived using this representation. The receiver uses a modified Viterbi algorithm to determine the most likely sequence of data symbols transmitted. An upperbound on the probability of symbol error for this receiver is obtained. Numerical results illustrating the applicability of the present work to optimizing the performance of a digital satellite communications link are also presented.