Bayesian learning of Echo State Networks with tunable filters and delay&sum readouts

In this paper we investigate the problem of learning Echo State Networks (ESN) with adaptable filter neurons and delay&sum readouts. A brute-force solution to this learning problem is often impractical due to nonlinearity and high dimensionality of the resulting optimization problem. In this work we propose an approximate solution to the ESN learning by appealing to the variational Bayesian EM-type of estimation algorithm. We show that such approach allows to significantly reduce the dimensionality of the resulting objective functions. Furthermore, it allows to implement ESN learning and adapt filter neurons and delays jointly within the variational framework. Simulations are performed for learning randomly generated target ESNs, as well as other synthetic nonlinear dynamic systems. The results demonstrate that the proposed learning algorithm can improve ESN learning for a wide class of problems.