Nonextensive entropy and regularization for adaptive learning

A challenging situation gradient-based learning algorithms encounter is an occasional converge to undesired local minima. To alleviate this situation, this paper builds on the theory of nonextensive statistical mechanics to develop a new adaptive gradient-based learning scheme that applies a sign-based weight adjustment, inspired from the Rprop algorithm, on a perturbed version of the original error function. The perturbations are characterized by the q entropic index of the nonextensive entropy, and their impact is controlled by means of regularization. This approach modifies the error landscape at each iteration allowing the algorithm to explore previously unavailable regions of the error surface, and possibly escape undesired local minima. The performance of the adaptive scheme is empirically evaluated using problems from the UCI repository of machine learning databases and other classic benchmarks.