Analysis of weight initialization methods for gradient descent with momentum

The back propagation algorithm using gradient descent with momentum is a commonly used training algorithm for the artificial neural networks. In this work, a set of experiments were conducted to obtain a detailed comparison of various known weight initialization methods. By doing so, the best suited weight initialization routines for the gradient descent approach with momentum was identified. Six problems of the functions approximation domain were selected for these experiments. Statistical metrics like one sided tailed t-test, the standard deviation of simulation error as well as its mean value were evaluated and used for the purpose of decision making. Results obtained from these experiments strongly advocate that the weight initialization method proposed by Nguyen and Widrow was the best suited technique while training the network by Gradient Descent with momentum approach.