Decision feedback detection using modulation constraints for MIMO Rayleigh fading channels

We propose a new technique for designing decision feedback detectors (DFDs) wherein the per-symbol decision rules are obtained by exploiting modulation constraints. We present our results in the context of the multi-input, multi-output (MIMO) Rayleigh fading channel with N receive and K transmit antennas. Three cases are considered to illustrate our technique where all the transmitters employ real-valued pulse-amplitude modulation (PAM), or phase-shift keying (PSK), or complex quadrature amplitude modulation (QAM). In each case, the corresponding per-symbol decision rule is obtained by imposing the modulation constraint on a generalized likelihood function maximization problem. For the case when all transmitters employ PAM, we show that the resulting DFD (the PAM-DFD) is equivalent to the decorrelating decision feedback detector (D-DFD) when the latter is used on a modified received statistic. We derive the exact joint and symbol error probability (JEP, SEP) for the PAM-DFD and show that the (possibly fractional) diversity orders of both the JEP and SEP are equal to N-(K-1)/2. This greatly improves on the diversity order of N-K+1 of the conventional D-DFD which does not exploit the real modulation constraint. We also derive the PSK-DFD and the QAM-DFD for the cases when all transmitters employ PSK and QAM, respectively, and show that these detectors result in significant performance improvements over the conventional D-DFD as well.