Decoding of trellis-encoded signals in the presence of intersymbol interference and noise

A novel receiver for data transmission systems using trellis-coded modulation (TCM) is investigated. It comprises a whitened matched filter and a trellis decoder which combines the previously separated functions of equalization and TCM decoding. TCM encoder, channel, and whitened matched filter are modeled by a single finite-state machine with combined ISI (intersymbol interference) and code states. Using ISI-state truncation and the self-partitioning principles inherent in TCM, a systematic method is developed for reducing the state complexity of the corresponding combined ISI and code trellis. A modified branch metric is used for canceling those ISI terms which are not represented by the trellis states. The approach leads to a family of Viterbi decoders which offer a tradeoff between complexity and performance. An adaptive version of the receiver is discussed, and simulation results are presented for channels with severe linear distortion. It is shown that the adaptive receiver achieves a significant gain in noise margin.<>