Modified Rao test for adaptive subspace detection

The problem of detecting a subspace signal is studied in colored Gaussian noise with an unknown covariance matrix. In the subspace model, the target signal belongs to a known subspace, but with unknown coordinates. We first present a new derivation of the Rao test based on the subspace model, and then propose a modified Rao test (MRT) by introducing a tunable parameter. The MRT is more general, which includes the Rao test and the generalized likelihood ratio test as special cases. Numerical results demonstrate that the MRT can offers the flexibility of being adjustable in the mismatched case where the target signal deviates from the presumed signal subspace. In particular, the MRT provides better mismatch rejection capacities as the tunable parameter increases.