Theoretical error rate analysis of trellis shaped M-QAM constellations

Trellis shaping (TS) is a promising technique for controlling signal transition and capable of making the shape of the signal into a desired form by exploiting the trellis structure of convolutional codes. TS was originally proposed by Forney for the average power reduction of quadrature amplitude modulation (QAM) signals, and subsequently, it has found its application in the field of the peak-to-average power ratio (PAPR) reduction of the single-carrier as well as orthogonal frequency-division multiplexing (OFDM) signals. So far, due to its rather complicated signal generation structure, the performance of TS has been studied mostly based on computer simulations and, to the best of the authors´ knowledge, its precise theoretical analysis has been left undeveloped even for uncoded cases. In this paper, we thus analyze the theoretical symbol error rate (SER) and bit error rate (BER) of TS over AWGN and frequency-selective Rayleigh fading channels and develop their closed form expressions that asymptotically match the exact results obtained by the simulations.