Parallel implementation of partially connected recurrent networks

Recurrent neural networks are suitable for solving problems with temporal extent e.g. speech recognition, time series prediction, sequence generation. The biggest problem is, however, its computational complexity during the training process. Using the well-known real time recurrent learning rule of Zipser and William (1989), the training time of each epoch is of order O(n⁴) where n is the total number of hidden units and output units. To cope with this low efficiency, the authors have devised a new network model called the partially connected recurrent network. Using the modified learning rule, the computational complexity of training becomes only O(mn+np), where m is number of outputs and p is the number of external inputs. In addition this new recurrent model is very suitable for running on a regularly connected SIMD parallel computer since each neuron needs only to communicate with those connected to it directly. The authors have verified this idea by implementing both a ring-structured recurrent network and a grid-structured recurrent network on the DECmpp 12000Sx System with 8192 processors. It can be shown that the training time per epoch now increases only linearly, i.e. O(m+n+p), which is a big advantage for solving large scale problems