To Be Bernoulli or to Be Gaussian, for a Restricted Boltzmann Machine

We introduce a method that automatically selects appropriate RBM types according to the visible unit distribution. The distribution of a visible unit strongly depends on a dataset. For example, binary data can be considered as pseudo binary distribution with high peaks at 0 and 1. For real-value data, the distribution can be modeled by single Gaussian model or Gaussian mixture model. Our proposed method selects appropriate RBM according to the distribution of each unit. We employ the Gaussian mixture model to determine whether the visible unit distribution is the pseudo binary or the Gaussian mixture. According to this distribution, we can select a Bernoulli-Bernoulli RBM(BBRBM) or a Gaussian-Bernoulli RBM(GBRBM). Furthermore, we employ normalization process to obtain a smoothed Gaussian mixture distribution. This allowed us to reduce variations such as illumination changes in the input data. After experimentation with MNIST, CBCL and our own dataset, our proposed method obtained the best recognition performance and further shortened the convergence time of the learning process.