Nonlinear quantization and the design of coded and uncoded signal constellations

The author analyzes the effects of nonlinear quantizers, used in T1 carrier systems, on the performance of QAM (quadrature amplitude modulation) and TCM (trellis coded modulation) voiceband data communication modems, and introduces a method to counteract the effects of these nonlinearities. The effect of nonlinear quantizers is modeled as a source of multiplicative noise whose variance is proportional to the amplitude of the quantized signal. The relation between variance of the multiplicative noise, characteristics of the nonlinear quantizer, and impulse response of the pulse shaping filters used in the modems is derived. Effects of multiplicative noise on the performance of QAM and TCM modems are analyzed. A method for the design of QAM and TCM signal constellations is introduced which counteracts the harmful effects of nonlinear quantizers. The new constellations provide 2-3 dB performance improvements in multiplicative noise channels while losing less than 0.5 dB in additive noise channels