Boosting Stochastic Newton with Entropy Constraint for Large-Scale Image Classification

Large scale image classification requires efficient scalable learning methods with linear complexity in the number of samples. Although Stochastic Gradient Descent is an efficient alternative to classical Support Vector Machine, this method suffers from slow convergence. In this paper, our contribution is two folds. First we consider the minimization of specific calibrated losses, for which we show how to reliably estimate posteriors, binary entropy and margin. Secondly we propose a Boosting Stochastic Newton Descent (BSN) method for minimization in the primal space of these specific calibrated loss. BSN approximates the inverse Hessian by the best low-rank approximation. The originality of BSN relies on the fact that it does perform a boosting scheme without computing iterative weight update over the examples. We validate BSN by benchmarking it against several variants of the state-of-the-art SGD algorithm on the large scale Image Net dataset. The results on Image Net large scale image classification display that BSN improves significantly accuracy of the SGD baseline while being faster by orders of magnitude.