Label Consistent Fisher Vectors for Supervised Feature Aggregation

In this paper, we present a simple and efficient way to add supervised information into Fisher vectors, which has become a popular image representation method for image classification and retrieval purposes in recent years. The basic idea of our approach is to improve the Fisher kernel in the training process by adding a discriminative label comparison matrix to it. The resulting new representations, which we call Label Consistent Fisher Vectors (LCFV), can be solved for both over determined and underdetermined cases. We show that LCFV has better classification performance than traditional Fisher vectors on three public datasets.