Generalized square and hexagonal constellations for intersymbol-interference channels with generalized Tomlinson-Harashima precoders

For channels with severe amplitude distortion such as high-speed digital subscriber loops, using a Tomlinson-Harashima (TH) precoder in the transmitter has the same intersymbol-interference-elimination and noise-suppression advantages as using a decision feedback equalizer in the receiver, but without its disadvantage of error propagation. TH precoders have a further advantage of interworking easily with coded modulation techniques. Square constellations are generally preferred in systems using conventional TH precoders. The choice for the size of square constellations is, however, limited. The authors widen that choice by designing new families of generalized square and hexagonal constellations together with their matched generalized TH precoders. The design of the constellation and TH precoder is closely related to the design. Using a generalized TH precoder also results in a smaller peak-to-average power ratio of the transmitted signal than that for conventional TH precoders. Simple modifications of Viterbi decoders required by the use of TH precoders are also discussed