Optimum Receiver Filters in Digital Quadrature Phase-Shift-Keyed Systems with a Nonlinear Repeater

The effects of filter shaping and nonlinearity in a coherent quadrature phase-shift-keyed (QPSK) system are investigated. Optimization of the receiver filter with respect to the meansquare error (MSE) is considered. Optimization is performed under two different constraints, corresponding to implementation of the filter at intermediate frequency and baseband, respectively. As an example a channel with a typical traveling-wave tube (TWT) nonlinearity is studied. Numerical results are obtained which describe the effect of different transmitter filter characteristics on system performance.