Hierarchical mixtures of experts and the EM algorithm

The problem of training a mixture of experts architecture can be treated as a maximum likelihood estimation problem. Both Jacobs et al. (1991) and Jordan and Jacobs (1992) derived learning algorithms by computing the gradient of the log likelihood for their respective architectures. Empirical tests revealed that although the gradient approach succeeded in finding reasonable parameter values in particular problems, the convergence rate was not significantly better than that obtained by using gradient methods in multi-layered neural network architectures. The gradient approach did not appear to take advantage of the modularity of the architecture. An alternative to the gradient approach was proposed by Jordan and Jacobs (1994), who introduced an expectation-maximization (EM) algorithm for mixture of experts architectures. EM is a general technique for maximum likelihood estimation that can often yield simple and elegant algorithms. For mixture of experts architectures, the EM algorithm decouples the estimation process in a manner that fits well with the modular structure of the architecture. Moreover, Jordan and Jacobs (1994) observed a significant speedup over gradient techniques. The author looks at the performance of the hierarchical mixture of experts (HME) architecture trained with EM and a matched backpropagation network on the problem of learning the forward dynamics of a four-joint robot arm. The HME algorithm runs two orders of magnitude faster than backpropagation on this large-scale robot dynamics problem