The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions

The spike-and-slab restricted Boltzmann machine (ssRBM) is defined to have both a real-valued “slab” variable and a binary “spike” variable associated with each unit in the hidden layer. The model uses its slab variables to model the conditional covariance of the observation-thought to be important in capturing the statistical properties of natural images. In this paper, we present the canonical ssRBM framework together with some extensions. These extensions highlight the flexibility of the spike-and-slab RBM as a platform for exploring more sophisticated probabilistic models of high dimensional data in general and natural image data in particular. Here, we introduce the subspace-ssRBM focused on the task of learning invariant features. We highlight the behaviour of the ssRBM and its extensions through experiments with the MNIST digit recognition task and the CIFAR-10 object classification task.