Parameter sensitivity in the backpropagation learning algorithm

The sensitivity of the backpropagation training algorithm to its learning rate and gain parameters is investigated. The authors report results from numerical experiments giving evidence for extreme sensitivity of training time, and even training success, to these parameters. A dynamic parameter update method that avoids the chaotic regime is derived. It is concluded that small changes of the gain parameter or the learning rate can greatly alter training time for a backpropagation network and even whether training is possible. Combined with evidence of similar sensitivity to initializations, these results show the impossibility of simple recipes for useful convergence criteria in backpropagation networks. Perhaps by optimizing all the variables that comprise the stepsize together, it will be easier to achieve a useful optimization strategy for the backpropagation algorithm