Interference cancellation enhancement through generalized widely linear equalization in QAM systems

In this paper, we introduce a generalized widely linear (WL) equalizer for quadrature amplitude modulation (QAM) systems with single/multiple antennas. In our proposed implementation, the WL receiver first separates the in-phase (I) and quadrature (Q) parts of the complex-valued baseband received signal and jointly filters the two branches for signal detection. Infinite length WL minimum mean-square error (WL-MMSE) linear, and WL decision-feedback-equalizer (WL-DFE) settings are derived and performance is analyzed in co-channel interference limited channels. It is shown that, in frequency selective Rayleigh fading channels, the interference cancellation (IC) gain depends mainly on the rank (r) of the interference correlation matrix (ICM) which is defined as the covariance of the vector-valued signal which consists of the real and imaginary parts of the noise-plus-interference signal collected at multiple antenna branches. Assuming that the DFE feedback path is error free, we show that a WL QAM receiver with N antennas exhibits full IC capability (that is complete interference removal) when the ICM is rank deficient i.e., when: r < 2N. This condition implies that a WL-DFE receiver can reject any combination of M₁ pulse-amplitude-modulation (PAM) and M₂ QAM interferers satisfying the constraint: M₁ + 2M₂ < 2N. Simulation results show that, in the presence of PAM-type interference, the gain of WL-DFE is reduced by decision feedback errors while the IC benefit of WL-MMSE is limited by the noise enhancement problem. Nevertheless, the proposed receivers are shown to be useful in cellular systems that employ a combination of PAM and QAM schemes.