Nonlinear Transceiver Designs in MIMO Amplify-and-Forward Relay Systems

Various linear transceiver design methods have been developed in three-node amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay systems. Nonlinear designs in such systems, however, have yet to be investigated. In this paper, we propose nonlinear transceiver designs for a linear source and relay precoded system with the QR successive-interference-cancellation (SIC) receiver and another linear source and relay precoded system with the minimum-mean-squared-error (MMSE) SIC receiver. Our designs minimize the criterion of the block error rate, which is a complicated function of the source and relay precoders. Solving the two precoders simultaneously is not feasible. To overcome the difficulties, we first resort to the primal decomposition approach, i.e., transferring the original optimization to a subproblem and a master problem and solving the two precoders individually. However, since two power constraints are mutually coupled, the decomposition cannot actually be conducted. We then propose a unitary structure for the source precoder and show that the power constraints can be decoupled. As a result, the source precoder can be solved as a function of the relay precoder in the subproblem. With a proposed relay precoder structure, the master problem can further be transferred to a scalar-valued concave optimization problem. A closed-form solution can finally be derived by the Karuch-Kuhn-Tucker (KKT) conditions. Simulations show that the proposed transceivers can significantly outperform the existing linear transceivers.