Efficient gradient computation for continuous and discrete time-dependent neural networks

The author provides calculus-of-variations techniques for the construction of backpropagation-through-time (BTT) algorithms for arbitrary time-dependent recurrent neural networks with both continuous and discrete dynamics. The backpropagated error signals are essentially Lagrange multipliers. The techniques are easy to handle because they can be embedded into the Hamiltonian formalism widely used in optimal control theory. Three examples of important extensions to the standard BTT-algorithm provide proof of the power of the method. An implementation of the BTT-algorithms which overcomes the storage drawbacks is suggested