Online parts-based feature discovery using competitive activation neural networks

The family of competitive activation models has recently attracted some interest. These models are a variation upon competitive neural networks where a local feedback process drives the competitive interaction rather than some form of lateral inhibition. However, this process can be viewed in terms of a generative model that reduces the generalized Kullback-Leibler divergence between the input distribution and the reconstruction distribution. From this insight we construct an online training method based on a stochastic gradient descent that reduces this measure while retaining the constraint of non-negativity inherent in the competitive neural network. We compare our results to non-negative matrix factorization (NMF), and show how the method results in a highly orthogonal, localized and parts-based representation of the data set, even when NMF does not, without the use of any explicit orthogonality or localization regularizers. Additionally, we show how the method leads to a basis better suited for discriminative tasks.