On-line learning in recurrent neural networks using nonlinear Kalman filters

The extended Kalman filter has been successfully applied to the feedforward and the recurrent neural network training. Recently introduced derivative-free filters (unscented Kalman filter and divided difference filter) outperform the extended Kalman filter in nonlinear state estimation. In the parameter estimation of the feedforward neural networks UKF and DDF are comparable or slightly better than EKF, with a significant advantage that they do not demand calculation of the neural network Jacobian. In this paper, we consider the application of EKF, UKF and DDF to the recurrent neural network training. The class of non-linear autoregressive recurrent neural networks with exogenous inputs is chosen as a basic architecture due to its powerful representational capabilities.