Distributed robust training of multilayer neural netwroks using normalized risk-averting error

This paper describes a novel distributed quasi-Newton-based robust training using the normalized risk-averting error (NRAE) with the gradual deconvexification (GDC) strategy. The main purpose of the computation is accomplished by optimizing the NRAE criterion parallely across different computing units, thereby two big advantages such as faster computation and global convergence can be obtained. The key idea is to replace the log partition function of the NRAE with a parallelizable upper-bound based on the concavity of the log-function. As a result, it is confirmed that the method is robust, and provides high quality training solutions regardless of initial values. Furthermore, the CPU time is drastically improved by the proposed distribution method without losing the quality of solutions.