D-entropy minimization: integration of mutual information maximization and minimization

In this paper, we propose a D-entropy minimisation method, aiming to unify information maximization and minimization methods. The D-entropy is defined by difference between Renyi´s entropy and Shannon´s entropy. The D-entropy minimization corresponds to both mutual information maximization and minimization. Thus, the method can be used to interpret explicitly internal representations and to improve generalization. The D-entropy minimization was applied to two problems: six alphabet character recognition and the inference of well-formedness of artificial strings. Experimental results confirmed that by minimizing the D-entropy a small number of principal hidden units can be detected and generalization performance can be improved