On practical use of stagewise second-order backpropagation for multi-stage neural-network learning

We analyze the Hessian matrix H of the sum-squared-error measure for multilayer-perceptron (MLP) learning, showing the following intriguing results: At an early stage of learning, H is indefinite. The indefiniteness is related to the MLP structure, which also determines rank of H (per datum). Exploiting negative curvature leads to efficient learning algorithms. H can be much less ill-conditioned than the Gauss-Newton Hessian J^TJ. All these new findings are obtained by our stagewise second-order backpropagation; the procedure exploits MLP\´s "layered symmetry" to evaluate H quickly, making exact Hessian evaluation feasible for fairly large practical problems. In fact, it works faster than rank-update methods that evaluate only J^TJ. It also serves to appraise other existing learning algorithms that perform implicit Hessian-vector multiply.