Transceiver optimization for interference MIMO relay systems using the structure of relay matrix

In this paper, we study the transceiver design problem for amplify-and-forward interference multiple-input multiple-output (MIMO) relay communication systems, where multiple transmitter-receiver pairs communicate simultaneously with the aid of a relay node. We aim at minimizing the mean-squared error (MSE) of the signal waveform estimation at the receivers subjecting to transmission power constraints at the transmitters and the relay node. Since the transceiver optimization problem is nonconvex with matrix variables, the globally optimal solution is intractable to obtain. To overcome the challenge, we propose an iterative transceiver design algorithm where the transmitter, relay, and receiver matrices are optimized iteratively by exploiting the optimal structure of the relay precoding matrix. Simulation results show that the proposed algorithm performs better than the existing technique in terms of both MSE and bit-error-rate.