The square root Kalman filter training of recurrent neural networks

The conventional Kalman filter suffers from the problem that the covariance matrix may not remain positive definite. In using the filter to train neural networks, the consequence of this problem is both a loss of efficiency during training and an eventual blow up. This problem has long been recognized in the application of the filter in signal processing and several forms of a square root Kalman filter have been derived to overcome it. However, the existing square root Kalman filter algorithms cannot incorporate the learning rate used in neural network training. In many situations, the incorporation of the learning rate into the training algorithms is crucial to obtaining excellent results in training time lagged recurrent neural networks (TLRNN). In this paper, a new square root Kalman filter equation is derived to train TLRNN which allows a user to incorporate the learning rate into the neural network training. With this new square root Kalman filter algorithm, we are able to train a neural network to convergence on a large and complex problem related to misfire diagnostics over a very large number of cycles, to produce extremely high classification accuracy on the diagnostic task. This training proceeded without any training problems we often experienced when a standard Kalman filter training algorithm was used for identical initializations and structures of the network. Furthermore, the neural network trained by the new method outperforms the one trained with the conventional Kalman filter algorithm by almost a factor of two