The effect of initial weight, learning rate and regularization on generalization performance and efficiency

The goal of this paper is to study the factors that affect the generalization performance and efficiency for neural network learning. First, this paper investigates the effect of initial weight ranges, learning rate, and regularization coefficient on generalization performance and learning speed. Based on this, we propose a hybrid method that simultaneously considers these three factors, and dynamically tunes the learning rate and regularization coefficient. Then we present the results of some experimental comparisons among these kinds of methods in several different problems. Finally, we draw conclusions and make plans for future work.