Training feedforward neural networks using orthogonal iteration of the Hessian eigenvectors

The paper describes a training algorithm for multilayer perceptrons. It has scalable memory requirements, which may range from O(W) to O(W²), although in practice the useful range is limited to lower complexity levels. The algorithm is based upon a novel iterative estimation of the principal eigensubspace of the Hessian, together with a quadratic step estimation procedure. It is shown that the new algorithm has convergence time comparable to conjugate gradient descent, and may be preferable if early stopping is used as it converges more quickly during the initial phases. Results of experiments to confirm the algorithm´s performance are presented