Performance analysis of square M-QAM with dual-hop relay link in Nakagami-m fading channel

In this paper, we analyze the symbol error probability (SEP) performance of square M-ary quadrature amplitude modulation (M-QAM) with dual-hop relay transmission considering independent and non-identical flat Nakagami-m fading channels. The relay is operating under amplify-and-forward (AF) mode with average power scaling (APS) and instantaneous power scaling (IPS) constraints. Simplified closed-form expressions for the SEP are presented following the cumulative distribution function (cdf) based approach. Moreover, using a series representation of complementary error function er fc(·), we derive exact SEP expression under sufficiently large signal-to-noise ratio (SNR) for the source-relay link and compared the error performance with the simplified SEP solution. Numerical and computer simulations results are presented to verify the accuracy of the proposed mathematical analysis.