Sparse feature selection by information theory

Learning sparse structures in high dimensions defines a combinatorial selection problem of e.g. informative feature dimensions and a subsequent estimation task to determine adequate model parameters. The arguably simplest sparse inference problem requires estimating a sparse mean in high dimensions when most dimensions should be discarded [6]. We reduce sparse mean estimation to a pure selection problem by restricting the source to binary values that are contaminated with various noise models. The model selection principle of Approximation Set Coding [2], [3] is rigorously applied, and generalization capacity is used to evaluate different algorithms. Simulation results demonstrate the effectiveness of generalization capacity compared to traditional model selection approaches. Sampling-based approximation yields insights into the behavior of algorithms in high dimensions at different noise levels.