TR-BFGS Algorithm for Multi-layer Feed-Forward Neural Networks

The most popular algorithm training feed-forward neural networks is the back-propagation algorithm which minimizes the error function using the steepest descent direction. In practice, although, even with a small learning rate which slows down the training process, the BP algorithm can exhibit oscillatory behavior when it encounters steep valleys. Trust region method has advantages of global convergence and quick convergence rate and so on. Broyden-Fletcher-Goldfard-Shano (BFGS) has advantages of computing only an approximation to the inverse of Hessian matrix, that is no second derivatives calculated. Also, the BFGS Hessian matrix update is symmetric and positive definite, making the algorithm numerically stable. This article presents an efficient training algorithm, namely TR-BFGS algorithm, combining with both of their advantages. We evaluate the convergence speed of proposed algorithm by using an example of approximation of function. The experimental results show that the new TR-BFGS algorithm is effective.