Linear Precoding Designs for Amplify-and-Forward Multiuser Two-Way Relay Systems

Two-way relaying can improve spectral efficiency in two-user cooperative communications. It also has great potential in multiuser systems. A major problem of designing a multiuser two-way relay system (MU-TWRS) is transceiver or precoding design to suppress co-channel interference. This paper aims to study linear precoding designs for a cellular MU-TWRS where a multi-antenna base station (BS) conducts bi-directional communications with multiple mobile stations (MSs) via a multi-antenna relay station (RS) with amplify-and-forward relay strategy. The design goal is to optimize uplink performance, including total mean-square error (Total-MSE) and sum rate, while maintaining individual signal-to-interference-plus-noise ratio (SINR) requirement for downlink signals. We show that the BS precoding design with the RS precoder fixed can be converted to a standard second order cone programming (SOCP) and the optimal solution is obtained efficiently. The RS precoding design with the BS precoder fixed, on the other hand, is non-convex and we present an iterative algorithm to find a local optimal solution. Then, the joint BS-RS precoding is obtained by solving the BS precoding and the RS precoding alternately. Comprehensive simulation is conducted to demonstrate the effectiveness of the proposed precoding designs.