Adaptation Learning Rate Algorithm of Feed-Forward Neural Networks

BP algorithm solves how to change hidden neurons weights of multilayer feed-forward neural networks, it uses mean square error criterion as the cost function, which takes gradient descent method to optimize the cost function to get the minimum and propagate the error signals to tune the weights. The gradient descent method uses fixed learning rate which denotes the weights changing extent. If the learning rate is larger, the learning speed is faster, but it may induce the oscillating, in contrast, if the learning rate is smaller, the learning process is more stable, but learning speed is slower. In this paper, we propose a new adaptation learning rate algorithm, it decreases the learning rate as the error value decreases, which can accelerate the learning speed in case of the steady leaning process, and the experiment results show the improved algorithm is very effective.