Distributed learning and cooperative learning

The concept of modularization and coupling connectionist network modules is a promising way of building large-scale neural networks and improving the learning performance of these networks. On the other hand, the modularization scheme would be of little use if there does not exist such an appropriate learning procedure to train high-level modules separately and to integrate those functionally pre-specified modules efficiently. This paper describes a way of realizing such a learning procedure that supports the modularization and coupling connectionist network modules. We present a distributed learning procedure for networks composed of many separate modular networks, each of which is trained to handle a subset of the complete set of training examples. This paper also describes a way of coupling decomposed neural networks modules. It is shown that with the proper decomposition, the whole network is composed as the weighted summation of the decomposed network modules.