Robust Training of Multilayer Neural Networks Using Parameterized Online Quasi-Newton Algorithm

This paper describes a novel robust training algorithm based on quasi-Newton process in which online and batch error functions are associated by a weighting coefficient parameter. The parameter is adjusted to ensure that the algorithm gradually changes from online to batch. That is, the transition from the online method to the batch one is parameterized in the proposed algorithm in the same concept as the improved online quasi-Newton algorithm introduced in [9][10]. Furthermore, an analogy between the proposed and Langevin algorithms is considered. Langevin algorithm is a gradient-based continuous optimization method incorporating Simulated Annealing concept. The proposed algorithm is employed for robust neural network training purpose. Neural network training for some benchmark problems with high-nonlinearity is presented to demonstrate the validity of proposed algorithm. The proposed algorithm achieves more accurate and robust training results than the other quasi-Newton based training algorithms.