Automatic rank estimation for practical STAP covariance estimation via an expected likelihood approach

We address the problem of estimation of structured covariance matrices for radar space-time adaptive processing (STAP)¹. The knowledge of the interference environment has been exploited in many previous works to accurately estimate a structured disturbance covariance matrix. In particular, it has been shown that employing the rank of clutter subspace, i.e. rank constrained maximum likelihood (RCML) estimation, leads to a practically powerful estimator as well as a closed form solution. While the rank is a very effective constraint, often practical non-idealities make it difficult to be known precisely using physical models. We propose an automatic rank estimation method in STAP via an expected likelihood (EL) approach. We formulate rank estimation as an optimization problem with the expected likelihood criterion and formally prove that the proposed optimization has a unique solution. Through experimental results from a simulation model and KASSPER dataset, we show the RCML estimator with the rank obtained via the EL approach outperforms RCML estimators with the other rank selection methods in the sense of a normalized signal-to-interference and noise ratio (SINR) and the probability of detection.