An adaptive training algorithm for back-propagation neural networks

To improve the convergence speed of the backpropagation training algorithm, the authors have chosen a dynamic learning rate which is a weighted average of direction cosines of successive incremental weight vectors ΔW at the current and several previous iterations. These adjacent direction cosines reflect the local curvature of the error surface, along which an `optimum´ search for the minimum error is determined for the weight adjustment of the next iteration. The authors have tested this on a real problem of training a three-layer feedforward artificial neural network for REM (rapid eye movement) sleep stage recognition. The training performance was significantly improved in terms of both faster convergence and smaller error when the last three direction cosines were included in determining the dynamic learning rate