Learning the solution sparsity of an ill-posed linear inverse problem with the Variational Garrote

The Variational Garrote is a promising new approach for sparse solutions of ill-posed linear inverse problems (Kappen and Gomez, 2012). We reformulate the prior of the Variational Garrote to follow a simple Binomial law and assign a Beta hyper-prior on the parameter. With the new prior the Variational Garrote, we show, has a wide range of parameter values for which it at the same time provides low test error and high retrieval of the true feature locations. Furthermore, the new form of the prior and associated hyper-prior leads to a simple update rule in a Bayesian variational inference scheme for its hyperparameter. As a second contribution we provide evidence that the new procedure can improve on cross-validation of the parameters and we find that the new formulation of the prior outperforms the original formulation when both are cross-validated to determine hyperparameters.