Accelerating parallel tangent learning for neural networks through dynamic self-adaptation

In gradient based learning algorithms, momentum usually has an improving effect on convergence rate and reduces zigzagging phenomena but sometimes it causes the convergence rate to decrease. The parallel tangent (partan) gradient is used as a deflecting method to improve the convergence. In this paper, we modify the gradient partan algorithm for learning the neural networks by using two different learning rates, one for gradient search and the other for accelerating through parallel tangent, respectively. Moreover, the dynamic self-adaptation of the learning rate is used to improve the performance. In dynamic self adaptation, each learning rate is adapted locally to the cost function landscape and the previous learning rate. Finally we test the proposed algorithm, called the accelerated partan on various problems such as xor and encoders. We compare the results with those of dynamic self adaptation of learning rate and momentum.