Multi-stream extended Kalman filter training for static and dynamic neural networks

We discuss a powerful and enabling extension to the class of second order neural network training methods based on the extended Kalman filter (EKF). EKF training procedures are generally considered to be recursive or sequential in nature, where weight updates are performed on an instance-by-instance basis. On the other hand, other second order weight update procedures, such as conjugate gradient and quasi-Newton methods, are typically based on batch processing of training patterns. The multi-stream EKF weight update procedure combines the useful stochastic and sequential characteristics of the base EKF method with a mechanism that allows for multiple instances to be processed simultaneously in a manner that is consistent with the EKF framework, thereby resulting in an effective semi-batch, second order training method. We motivate the use of multi-stream EKF training by relating Kalman training of a single linear node to a batch least squares solution. The paper concludes with a simulation example of the application of multi-stream EKF training