On the use of order selection rules for accurate parameter estimation in threshold region

Finding the number of signals is crucial to parametric direction-of-arrival (DOA) estimation methods such as MUSIC and ESPRIT. In challenging scenarios such as low signal-to-noise ratio (SNR) and/or presence of closely-spaced sources, only part of the parameters can be accurately estimated while others cannot. The number of former estimates is termed as the effective model order (EMO). We first propose a procedure to determine the EMO via Monte Carlo simulation. Ideally an order selection rule should return a source number estimate equal to EMO, since using an overestimated signal number larger than the EMO in a parameter estimator introduces inaccurate parameter estimates, which is a waste of resources in some applications, while using an underestimate renders some strong signals being treated as noise, which causes an accuracy loss in their parameter estimates. We propose to combine an under-enumerator with an over-enumerator for accurate parameter estimation in the threshold region. Simulations results using the combination of the Baysian information criterion with Akaike information criterion in ESPRIT show that our proposal retains the benefit of the under-enumerators with only accurate estimates while remarkably improves the estimation accuracy.