Recursive Bayesian Levenberg-Marquardt Training of Recurrent Neural Networks

This paper develops a Bayesian approach to recursive second order training of recurrent neural networks. A general recursive Levenberg-Marquardt algorithm is elaborated using Bayesian regularization. Individual local regularization hyperparameters as well as an output noise hyper-parameter are reestimated in order to maximize the weight posterior distribution and to produce a well generalizing network model. The proposed algorithm performs a computationally stable sequential Hessian estimation with RTRL derivatives. Experimental investigations using benchmark and practical data sets show that the developed algorithm outperforms the standard RTRL and extended Kalman training algorithms for recurrent nets, as well as feed forward and finite impulse response neural filters, on time series modeling.