Optimal Receiver Filters for BPSK Transmission over a Bandlimited Nonlinear Channel

In this paper we specify an optimal receiver filter for 2phase PSK transmission over a channel consisting of the tandem connection of a linear filter and a bandpass nonlinearity, with thermal noise added at the channel output. This filter minimizes the bit estimation error subject to the constraint that it be a linear filter. Our main aim is to consider satellite communication channels and as such the system nonlinearity is taken to have both AM/AM and AM/PM conversions. In these systems we assume the up-link signal-to-noise ratio is large and thus our work relates to large transmit terminal applications. In an example the probability of error performance of the optimal received filter is found to be 2 dB better in terms of the effective output SNR as compared with that of a standard choice for the receiver filter.