Analysis of gradient descent learning algorithms for multilayer feedforward neural networks

Certain dynamical properties of gradient-type learning algorithms as they apply to multilayer feedforward neural networks are investigated. These properties are more related to the multilayer structure of the net than to the particular threshold units at the nodes. The analysis explains the empirical observation that the weight sequence generated by backpropagation and related stochastic gradient algorithms exhibits a long-term dependence on the initial choice of weights, and also a continued growth and/or drift long after the outputs have converged. The analysis is carried out in two steps. First, a simplified deterministic algorithm is derived using a describing function-type approach. Next, an analysis of the simplified algorithm is performed by considering an associated ordinary differential equation (ODE). Some numerical examples are given to illustrate the analysis. The dynamical behavior of backpropagation and related algorithms for the training of multilayer nets is discussed