Learning to predict a context-free language: analysis of dynamics in recurrent hidden units

Previous results regarding the language aⁿbⁿ suggest that while it is possible for a small recurrent neural network to process context-free languages, learning them is difficult. This paper considers the reasons underlying this difficulty by considering the relationship between the dynamics of the network and weight-space. We show that the dynamics required for the solution lie in a region of weight-space close to a bifurcation point where small changes in weights may result in radically different network behaviour. Furthermore, we show that the error gradient information in this region is highly irregular. We conclude that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a non-independent manner