Tomlinson-Harashima Precoding Based Transceiver Design for MIMO Relay Systems With Channel Covariance Information

In this paper, we investigate the performance of the Tomlinson-Harashima (TH) precoder based nonlinear transceiver design for a nonregenerative multiple-input multiple-output (MIMO) relay system assuming that the full channel state information (CSI) of the source-relay link is known, while only the channel covariance information (CCI) of the relay-destination link is available at the relay node. We first derive the structure of the optimal TH precoding matrix and the source precoding matrix that minimize the mean-squared error (MSE) of the signal waveform estimation at the destination. Then we develop an iterative algorithm to optimize the relay precoding matrix. To reduce the computational complexity of the iterative algorithm, we propose a simplified precoding matrices design scheme. Numerical results show that the proposed precoding matrices design schemes have a better bit-error-rate performance than existing algorithms.