Transceiver optimization for multi-user multi-antenna two-way relay channels

In this paper, we study a multi-user multi-antenna two-way relay system, where a multi-antenna base station (BS) exchanges uplink and downlink signals with multiple users via a multi-antenna amplify-and-forward relay station (RS). We jointly design the BS and RS transceivers, aiming to maximize the system bidirectional sum rate under inter-user interference free constraint. Since the optimization problem is non-convex, we employ alternating optimization algorithm to design the transmit and receive weighting matrices at the BS and RS. Simulation results show that the proposed solution offers higher bidirectional sum rate than existing schemes.