Training schedules for improved convergence

This paper explores the effect of various training strategies for backpropagation training using simple recurrent and temporal autoassociation neural networks. We demonstrate that training on increasingly more complex combined subsets reduces drastically the number of weight updates to reach a given RMS error criterion, suggesting that easier subtasks should be learned first and the complexity of the task to be learned gradually increased. Training in this way much reduces the number of attractor basins (possible solutions) thus leading to faster convergence. Convergence time is related to the complexity of the problem, therefore learning easier subproblems first constrains the solution by eliminating many possibilities, causing faster convergence. We demonstrate that increased complexity training also outperforms combined subset training and is therefore an essential engineering tool for large and complex tasks which have difficulty in converging using conventional training on a fixed set. Results indicate that a TA network, under suitable conditions, outperforms a simple recurrent network, even though the temporal autoassociation network requires more units. Further results illustrate the required reduction in RMS values for each subsequent training subset leading to a training schedule for much quicker learning.