The neural-SIFT feature descriptor for visual vocabulary object recognition

Recognizing the semantic content of an image is a challenging problem in computer vision. Many researchers attempt to apply local image descriptors to extract features from an image, but choosing the best type of feature to use is still an open problem. Some of these systems are only trained once using a fixed descriptor, like the Scale Invariant Feature Transform (SIFT). In most cases these algorithms show good performance, but they do not learn from their mistakes once training is completed. In this paper a continuous deep neural network feedback system is proposed which consists of an adaptive neural network feature descriptor, the bag of visual words approach and a neural classifier. Two initialization methods for the neural network feature descriptor were compared, one where it was trained on SIFT descriptor output and one where it was randomly initialized. After initial training, the system propagates the classification error from the neural network classifier through the entire pipeline, updating not only the classifier itself, but also the type of features to extract. Results show that for both initialization methods the feedback system increased accuracy substantially when regular training was not able to increase it any further. The proposed neural-SIFT feature descriptor performs better than the SIFT descriptor itself even with a limited number of training instances. Initializing on an existing feature descriptor is beneficial when not a lot of training samples are available. However, when there are a lot of training samples the system is able to construct a well-performing descriptor, solely based on classifier feedback.