Learning in parallel distributed processing networks: Computational complexity and information content

A set of experiments that precisely identify the power and limitations of the method of back-propagation is reported. The experiment on learning to compute the exclusive-OR function suggests that the computational efficiency of learning by the method of back-propagation depends on the initial weights in the network. The experiment on learning to play tic-tac-toe suggests that the information content of what is learned by the back-propagation method is dependent on the initial abstractions in the network. It also suggests that these abstractions are a major source of power for learning in parallel distributed processing networks. In addition, it is shown that the learning task addressed by connectionist methods, including the back-propagation method, is computationally intractable. These experimental and theoretical results strongly indicate that current connectionist methods may be too limited for the complex task of learning they seek to solve. It is proposed that the power of neural networks may be enhanced by developing task-specific connectionist methods