Flexible Detection Criterion for Source Enumeration in Array Processing

Two new procedures are proposed using random matrix theory to evaluate theoretically the performance of the Bayesian information criterion (BIC) for source enumeration in array processing. The proposed procedures can predict precisely the estimation results of the BIC. Also, they shed light on the behavior of the BIC. In order to remedy the limitation of the BIC, that is, the BIC tends to underestimate the number of sources by one in most cases, an extra parameter is introduced into the BIC formulation, which results in a new criterion for source enumeration, referred to as the flexible detection criterion (FDC). By carefully choosing this parameter, the FDC is capable of substantially reducing the probability of underestimation, which is validated by theoretical analysis and simulations. Note that the FDC is consistent and has low computational complexity, as the BIC.