Extraction of high level sequential structure using recurrent neural networks and radial basis functions

The authors examine the performance of a simple recurrent neural network when applied to a temporal sequence prediction problem. It is shown that when trained with a combination of optimization techniques, a simple recurrent neural network can provide the same performance as the cascade correlation architecture in fewer training epochs. Conclusions are that for this and other finite-state based problems, subject to the training algorithm making optimal use of the architecture, the performance is determined by the number of weights and number of recurrent nodes. It is shown that by using a network trained in this manner on this problem, a radial basis function network can be used to extract a higher-level representation from the recurrent nodes in the network. More importantly, this network is able to map the extracted representation onto the representation that was used to create the input sequence