Generation of organized internal representation in recurrent neural networks

A method with which internal representations (hidden unit patterns) are organized so as to increase information-theoretical redundancy in recurrent neural networks is presented. The information-theoretical redundancy is supposed to reflect the degree of organization or structure in hidden unit patterns. The representation by this method is expected to make it possible to interpret a mechanism of networks easily and explicitly. One of the problems in recurrent neural networks is that connection weights are smaller as the number of units in networks is larger, while producing uniform or random activity values at hidden units. Thus, it is difficult to interpret the meaning of hidden units. To cope with this problem, a complexity term proposed by D.E. Rumelhart was used. By using a modified complexity term, connections of networks could be highly activated, meaning that the connections could take larger absolute values. After a brief formulation of recurrent backpropagation with the complexity term, three experimental results-the XOR problem, a negation problem, and a sentence well-formedness problem-are presented