Joint Tomlinson–Harashima Source and Linear Relay Precoder Design in Amplify-and-Forward MIMO Relay Systems via MMSE Criterion

Existing minimum-mean-square-error (MMSE) transceiver designs in amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay systems all assume a linear precoder at the source. Nonlinear precoders in such a system have yet to be considered. In this paper, we study a nonlinear transceiver in AF MIMO relay systems in which a Tomlinson-Harashima (TH) precoder is used at the source, a linear precoder is used at the relay, and an MMSE receiver is used at the destination. Since two precoders and three links are involved, the transceiver design, which is formulated as an optimization problem, is difficult to solve. We first propose an iterative method to overcome the problem. In the method, the two precoders are separately optimized in an iterative step. To further improve the performance, we then propose a non-iterative method that can yield closed-form solutions for the precoders. This method uses the primal decomposition technique in which the original optimization can first be decomposed into a master and a subproblem optimization. In the subproblem, the optimum source precoder is solved as a function of the relay precoder. In the master problem, the optimization is then transferred to a relay-precoder-only problem. However, the optimization is not convex, and the primal decomposition cannot be directly applied. We then propose cascading a unitary precoder after the TH precoder so that the optimization in the subproblem and the master problem can be conducted. Furthermore, using a relay precoder structure, we can transfer the master problem to a convex optimization problem and obtain a closed-form solution by the Karuch-Kuhn-Tucker (KKT) conditions. Simulations show that the proposed transceivers can significantly outperform existing linear transceivers.