An improved whitening transformation for snapshot-deficient scenarios

Many adaptive signal processing algorithms, such as the MUltiple SIgnal Classification (MUSIC) spectral estimator, require knowledge of the number of sources. Source enumeration algorithms typically use the eigenvalues of the sample covariance matrix to infer the rank of the signal subspace. These algorithms often assume a white noise background. In colored noise environments, the standard solution is to whiten the data prior to processing. Using results from random matrix theory, Nadakuditi and Silverstein predict the fundamental signal detection limit as a function of the number of snapshots used to define the whitening filter and the number of snapshots used for the detector. They also describe a whitening-based detection algorithm whose performance approaches their asymptotic predictions. Motivated by prior work on the Dominant Mode Rejection beamformer (DMR), this paper proposes a modified whitening transform that uses a structured estimate of the noise-only covariance matrix. Simulations demonstrate that the new whitening transform performs substantially better than Nadakuditi and Silverstein´s algorithm when the number of snapshots available to estimate the whitening filter is small.