Online maximum-likelihood estimation for latent factor models

The online EM algorithm is a fast variant of the EM algorithm suitable for processing large streams of data. However the online EM algorithm is restricted to models in which an analytical expectation can be computed for the E-step. In this paper, we show that a new algorithm called the simulated online EM algorithm may be applied to a broad class of models used in signal processing and machine learning. These models, which are characterized by the presence of latent (or unobserved) positive factors, include in particular probabilistic variants of Non-Negative Matrix Factorization (NMF). We provide the main convergence properties of the simulated online EM algorithm and detail its application to the Latent Dirichlet Allocation (LDA) model.