Extended GLRT detectors of correlation and sphericity: The undersampled regime

Detecting the presence of one or multiple signals with unknown spatial signature can be addressed by testing the structure of the observation covariance matrix. The problem can be typically formulated as a sphericity test, which checks whether the spatial covariance matrix is proportional to the identity (white noise), or as a correlation test, which checks whether this matrix has a diagonal structure. When the number of samples is higher than the number of antennas, one can address this problem by formulating the generalized likelihood ratio test (GLRT), which basically compares the arithmetic and geometric means of the eigenvalues of the sample covariance/coherence matrix. The GLRT can be trivially extended to the undersampled regime by selecting only the positive sample eigenvalues. This paper investigates the asymptotic behavior of these extended GLRTs by determining the asymptotic law of the associated statistics under both hypotheses. The analysis is asymptotic in both the sample size (number of snapshots) and the observation dimension (number of antennas).