Nonmonotone learning rules for backpropagation networks

In this paper we study nonmonotone learning rules, based on an acceptability criterion for the calculated learning rate. More specifically, we impose that the error function value at each epoch must satisfy an Armijo-type criterion, with respect to the maximum error function value of a predetermined number of previous epochs. To test this approach, we propose two training algorithms with adaptive learning rates that employ the above-mentioned acceptability criterion. Experimental results show that the proposed algorithms have considerably improved convergence speed, success rate, and generalization, when compared with other classical neural network training methods