Survey of two selected superlinear learning techniques

The article surveys theoretical and practical aspects of two superlinear learning algorithms. The introduced algorithms feature novel solution to the line search subproblem simplified to a single step calculation of the appropriate values of step length and/or momentum term. It remarkably improves the computational complexity and implementation of the line search subproblem and yet does not harm the stability of the methods. The algorithms are theoretically proven to be convergent and universal within the proposed classification framework. They are capable of reaching superlinear convergence rates on an arbitrary task. Performance of the proposed algorithms is extensively evaluated on five data sets and compared to the relevant standard first order optimization techniques.