Optimize state equation of leaky-integrator ESN based on extend kalman filtering

Recently, a novel recurrent neural network (RNN) called leaky-integrator echo state network (Leaky-ESN) was proposed. Standard Leaky-ESN has showed the good prediction ability for time series. In this article, we introduce a new method called Leaky-ESN based on Extend Kalman Filtering (LEKF). The LEKF method uses the merits of extend kalman filtering (EKF) to improve Leaky-ESN. The reservoir state equation of standard Leaky-ESN ignores output feedback during the recursive process. Thus, to a certain extent, the deficiency of standard Leaky-ESN affects its prediction accuracy and make it expend more computing time. We find that the state update equation of EKF not only is similar with the reservoir state equation but also has the output feedback participating in the state equation updating, which inspires a new thought into us, thus, we propose the LEKF method. In LEKF method, we regarded the reservoir state equation and output mapping of Leaky-ESN as the nonlinear system model of EKF, and conduct derivation of the nonlinear model to obtain the important parameters like Jacobian matrix, Kalman gain and others of EKF recurrent equations. The LEKF method not only obtains a more smaller test error but also makes training error has a faster convergence due to adding the output feedback during the state update process, and shows good prediction ability than standard Leaky-ESN. We use two different time series to further validate the effectiveness of the method.