Sparse basis selection, ICA, and majorization: towards a unified perspective

Sparse solutions to the linear inverse problem Ax=Y and the determination of an environmentally adapted overcomplete dictionary (the columns of A) depend upon the choice of a “regularizing function” d(x) in several previously proposed procedures. We discuss the interpretation of d(x) within a Bayesian framework, and the desirable properties that “good” (i.e., sparsity ensuring) regularizing functions, d(x) might have. These properties are: Schur-concavity (d(x) is consistent with majorization); concavity (d(x) has sparse minima); parameterizability (d(x) is drawn from a large, parameterizable class); and factorizability of the gradient of d(x) in a certain manner. The last property (which naturally leads one to consider separable regularizing functions) allows d(x) to be efficiently minimized subject to Ax=Y using an affine scaling transformation (AST)-like algorithm “adapted” to the choice of d(x). A Bayesian framework allows the algorithm to be interpreted as an independent component analysis (ICA) procedure