Subspace Identification for DOA Estimation in Massive/Full-Dimension MIMO Systems: Bad Data Mitigation and Automatic Source Enumeration

In this paper, the direction-of-arrival (DOA) estimation problem for massive multiple-input multiple-output (MIMO) systems with a two dimensional (2D) array (also known as full-dimension MIMO) is investigated, assuming no knowledge of path number, noise power, path gain correlations and bad data statistics. Based on the variational Bayesian framework, a novel iterative algorithm for subspace identification operating on tensor represented data is proposed with integrated features of effective bad data mitigation and automatic source enumeration. The subspace recovered from the proposed algorithm not only enables existing 2D DOA estimators to be readily applied, if the number of signal paths is less than the number of horizontal antennas and vertical antennas, the subspaces in elevation and azimuth domains can be separately estimated, from which one dimensional (1D) DOA estimators can be utilized, thus further lowering the complexity. Simulation results are presented to illustrate the excellent performance of the proposed subspace recovery method and subsequent DOA estimation in terms of accuracy and robustness.