DOA outlier mitigation for generalised spatial smoothing

This paper considers the problem of DOA (direction-of-arrival) estimation for a small number of fully correlated sources. The standard spatial smoothing technique [1] may be applied to this single-snapshot model, but only for a uniformly-spaced linear antenna array (ULA). In [2], we introduced a special class of nonuniform array geometry with embedded partial arrays and a corresponding generalised spatial smoothing (GSS) algorithm. The initialisation stage of GSS (which is followed by a local maximum-likelihood refinement) involves spatial averaging over all suitable noncontiguous sub-arrays with identical inter-sensor separations. These partial arrays are themselves nonuniform in geometry, and have a small number of sensors. It is well known that MUSIC may fail to resolve poorly-separated sources when the SNR and number of spatial averagings are insufficient, due to abnormal DOA estimates ("outliers"). An additional outlier mechanism for partial-array MUSIC occurs because each partial array has some associated manifold ambiguity [3, 4]. Thus for spatial smoothing, the set of (initial) DOA estimates often contains outliers. This paper introduces an algorithm which aims to identify each outlier and to correct it, if possible.