Performance of joint trellis-coded quantisation/modulation with a Bayes receiver

The performance, based on the mean squared error criterion, of joint trellis-coded-quantisation (TCQ) and trellis-coded-modulation (TCM) systems operating in the additive-white-Gaussian-noise (AWGN) channel and the Rayleigh flat-fading channel is studied. Instead of a conventional Viterbi receiver, the authors propose an optimal nonlinear receiver based on Bayesian estimation theory. Since the Bayes receiver makes use of information from both the source and the channel coders, it can perform significantly better than the Viterbi receiver (which only makes use of information about the channel coder), especially at moderate to low channel signal-to-noise ratios (SNR). For example, in an AWGN channel with a SNR of 5 dB and a source rate R of 3 bit/sample, the simulation results show that the Bayes receiver is about 3 dB better than the Viterbi receiver in terms of the source SNR. The improvement is even higher for the Rayleigh fading channel. For high channel SNR, the performances of the two receivers are almost identical. In its original form, the Bayes receiver has a very high computational complexity. It is shown that, by exploiting the structure of the TCQ, it is possible to reduce the decoding complexity of this receiver