Greedy dirty models: A new algorithm for multiple sparse regression

This paper considers the recovery of multiple sparse vectors, with partially shared supports, from a small number of noisy linear measurements of each. It has recently been shown that it is possible to lower the sample complexity of recovery, for all levels of support sharing, by using a “dirty model”: a super-position of sparse and group-sparse modeling approaches; this is based on convex optimization. In this paper, we provide a new forward-backward greedy procedure for the dirty model approach. Each forward step involves the addition of either a shared feature common to all vectors, or a unique feature to one of the vectors, chosen in a natural greedy fashion. Each backward step involves greedy removal, again of at most one feature of either type. Analytical and empirical evidence shows that this outperforms all convex approaches, in terms of both sample and computational complexity.