Robust Multibranch Tomlinson–Harashima Precoding Design in Amplify-and-Forward MIMO Relay Systems

This paper proposes the design of robust transceivers with Tomlinson-Harashima precoding (THP) for multiple-input-multiple-output relay systems with amplify-and-forward protocols based on a multibranch (MB) strategy. The MB strategy employs successive interference cancellation on several parallel branches, which are equipped with different ordering patterns so that each branch produces transmit signals by exploiting a certain ordering pattern. For each parallel branch, the proposed robust nonlinear transceiver design consists of THP at the source along with a linear precoder at the relay and a linear minimum-mean-square-error receiver at the destination. By taking the channel uncertainties into account, the source and relay precoders are jointly optimized to minimize the mean square error. We then employ a diagonalization method along with some attributes of matrix-monotone functions to convert the optimization problem with matrix variables into an optimization problem with scalar variables. We resort to an iterative method to obtain the solution for the relay and the source precoders via Karush-Kuhn-Tucker conditions. An appropriate selection rule is developed to choose the nonlinear transceiver corresponding to the best branch for data transmission. Simulation results demonstrate that the proposed MB-THP scheme is capable of alleviating the effects of channel state information errors and improving the robustness of the system.