The learning rate in back-propagation systems: an application of Newton´s method

In backpropagation learning, the internode connection strengths, or weights, are adjusted by a method of gradient descent in weight space. The author shows how to apply a multidimensional version of Newton´s method for finding the roots of an equation to the question of determining how far to move down the gradient in each learning cycle in backpropagation. The results of a few simulations for a fully recurrent net are presented. The results show an appreciable improvement, by a factor of five to ten, in the convergence rate for these hard-to-learn tests