The robust backpropagation training of MLP´s: an optimal rapid restart method

It is proved that MLP´s with the training algorithm of backpropagation is a universal mapper, which can, in theory, approximate any continuous decision region arbitrarily well. Yet the convergence of backpropagation algorithms is still an open problem. It is well known that the time cost of backpropagation training often exhibits a remarkable variability, which seems to be of the generic character of heavy-tailed probability distribution. It has been demonstrated that, in most cases, rapid restart (RR) method can prominently suppress the heavy-tailed nature of training instances and improve efficiency of computation. However, it is usually time-consuming to verify whether a training instance is heavy-tailed. Moreover, if the heavy-tailed distribution is confirmed and the RR method is relevant, an optimal RR threshold should be chosen to facilitate the RR mechanism. In this paper, an approximate approach is proposed to quickly check whether a training instance is heavy-tailed or not by calculating the maximal Lyapunov exponent of a few generic running trace. Then a statistical estimator is deduced to determine the optimal RR threshold. The practically experimental results are consistent with the theoretical consideration perfectly.