Effect of curriculum on the consolidation of neural network task knowledge

Fundamental to the problem of lifelong machine learning is how to consolidate the knowledge of a learned task within a long-term memory structure (domain knowledge) without the loss of prior knowledge. We investigate the effect of curriculum, i.e., a selection of tasks and the order in which they are learned, on the consolidation of task knowledge. Relevant background material on knowledge transfer and consolidation using multiple task learning (MTL) neural networks is reviewed. A large MTL network is used as the long-term memory structure and task rehearsal overcomes the stability-plasticity problem and the loss of prior knowledge. Experimental results demonstrate that curriculum has a noticeable effect on the accuracy of consolidated knowledge particularly for the first few tasks that are learned. The results also suggest that, for given set of tasks and training examples, the mean accuracy of consolidated domain knowledge converges to the same level regardless of the curriculum. This is caused by the averaging effect of sequential consolidation