Examination of effectiveness of higher-order mean field Boltzmann machine learning based on linear response theorem

Mean field approximation (MFA) is an effective method to reduce the amount of computation for Boltzmann machine (BM) learning, but at the expense of losing accuracy. To improve the accuracy, one uses linear response theorem (LRT) in MFA and/or one incorporates higher-order terms of the Taylor-expanded Gibbs free energy that is used to derive MFA. In this paper, we discuss the effectiveness of this incorporation of the higher-order terms for the MFA based on the LRT. We examine the effectiveness for the BM with hidden units. When the MFA based on the LRT is used, one can use one-shot algorithm in the case of BM without hidden units, for which the effectiveness has already be examined, but one has to iteratively estimate the expectations and update weights and biases in the case of BM with hidden units, for which the effectiveness has not be examined yet. By numerical experiments, we showed that the incorporation of the higher-order terms is more effective as far as the learning had converged