Sparse model fitting in nested families: Bayesian approach vs penalized likelihood

We study the problem of model fitting in the framework of nested probabilistic families. Our criteria are: (i) sparsity of the identified representation, (ii) its ability to fit the (finite length) data set available. As we show in this paper, current methodologies, often taking the form of penalized versions of the data likelihood, cannot simultaneously satisfy these requirements, as the examples presented clearly demonstrate. On the contrary, maximization of the Bayesian model posterior, even without assumption of a complexity penalizing prior, is able to select models with appropriate complexity, enabling sound determination of its parameters in a second step.