Tomlinson-Harashima precoding design in MIMO wiretap channels based on the MMSE criterion

This paper investigates the Tomlinson-Harashima precoding (THP) design for secure communications in broadcast multiple-input multiple-output (MIMO) systems in the presence of a passive eavesdropper. We focus on optimizing the nonlinear transceiver to guarantee a certain Quality-of-Service (QoS) level for the intended receiver in terms of mean-squared-error (MSE). The scheme allocates the transmit power in order to achieve the target MSE for the intended receiver, and then uses the remaining available power to transmit artificial noise (AN) to degrade the eavesdropper´s channel. With the geometric mean decomposition (GMD) based THP, the lower MSE bound can be achieved. We convert the nonlinear transmit power minimization problem to a standard convex optimization problem. The solution for the problem is obtained by the waterfilling method via the Karush-Kuhn-Tucker (KKT) conditions. Simulation results demonstrate that the proposed nonlinear scheme outperforms existing linear transceiver designs.