Selective responses by entropy minimization

In this paper, we propose a method of entropy minimization for increasing selectivity and obtaining simple network architectures. An entropy function is defined with respect to the state of hidden units. By minimizing this entropy, the selectivity of hidden units can significantly be increased. Since a unit tends to respond to specific input patterns, the meaning or the function of the hidden units can easily be understood. In addition, we have observed that by minimizing the entropy, some units are forced to be inactive, responding to no input patterns. Thus, these inactive units can be deleted, and we can construct smaller network architectures. We applied the entropy method to standard and recurrent back-propagation. Experimental results confirmed that the number of units selectively responding to a specific pattern increased gradually, while units with low selectivity responding to multiple patterns decreased as entropy decreased. In addition, the number of units responding to no input patterns increased in proportion to the decrease of the entropy. These results show that the entropy minimization method can be used to improve the selectivity, and therefore, the interpretability of the networkʹs behaviors. Then, the method can be used to suppress unnecessary units and to produce simple internal representation or simple network architectures.