OSTBC Transmission in MIMO AF Relay Systems with Keyhole and Spatial Correlation Effects

In this paper, we investigate the degenerative effects of antenna correlation and keyhole on the performance of multiple-input multiple-output (MIMO) amplify-and-forward (AF) relay networks. In particular, we considered a downlink MIMO AF relay system consisting of an n_S-antenna base station S, an n_R-antenna relay station R, and an n_D-antenna mobile station D, in which the signal propagation originated from the mobile station suffers from keyhole and spatial correlation effects. We have derived an exact expression for the moment generation function of the instantaneous signal-to-noise ratio which enables us to analyze the symbol error probability and outage probability of the considered system. We have shown that although the mobile station is in a poor scattering environment, i.e., keyhole, the relay channel (S → R → D link) still achieves a cooperative diversity order of min(n_R, n_D) provided that the channel from the source (S → R link) is keyhole-free. This result is important to radio system designers since under such a severe scenario it is unnecessary to deploy a large number of antennas on the relay station (n_R) but only requires n_R = n_D to obtain the maximum achievable diversity gain.