A deterministic, annealing-based approach for learning and model selection in finite mixture models

We address the longstanding problem of learning and model selection in finite mixtures. A common approach is to generate solutions of varying number of components - via the expectation-maximization (EM) algorithm - and then select the best model in the sense of a cost such as the Bayesian information criterion (BIC). A recent alternative uses component-wise EM (CEM) and, further, integrates model selection within CEM. Both approaches are susceptible to finding poor solutions, the first due to the initialization sensitivity of EM and the second due to the sequential (greedy) nature of CEM. Deterministic annealing for clustering (DA) and mixture modeling (DAEM) provide potential for avoiding local optima. However, these methods do not encompass model selection. We propose a new technique with positive attributes of all these methods: it integrates learning and model selection, performs batch optimization over components, and has the character of DA, with the optimization performed over a sequence of decreasing temperatures. Unlike standard DA, with the partition entropy reduced as the temperature is lowered, our approach reduces the entropy of binary random variables that express whether each component is active or inactive. At low temperature, the method achieves explicit model order selection. Experiments demonstrate the favorable performance of our method, compared with several alternatives. We also give an interesting stochastic generative model interpretation for our method.