Using hierarchical dirichlet processes to regulate weight parameters of Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBM) have been widely applied to solve various problems in machine learning. Much research has been performed to study the structures of RBM, such as sparsity and probabilistic distributions of hidden units. However, little attention has been paid to investigating the features of weight components that connect visible and hidden layers. In this paper, we formulate a nonparametric Bayesian RBM model, in the sense that Hierarchical Dirichlet Process (HDP) is imposed as a prior of weights. Thus, the original RBM is decomposed as a group-structured machine, where the groups are revealed by HDP. The clustering effect of HDP is helpful to simplify the structure of RBM and the hierarchical structure of our model is advantageous to maintain the diversity of weight components within each group. The Monte Carlo EM (MCEM) algorithm is adopted to perform weight training and hyperparameter estimation. Various experiments verify the effectiveness of our proposed model.