Statistically controlled activation weight initialization (SCAWI)

Author

Drago, Gian Paolo ; Ridella, Sandro

Author_Institution

Istituto per i Circuiti Elettronici, CNR, Genova, Italy

Volume

3

Issue

4

fYear

1992

fDate

7/1/1992 12:00:00 AM

Firstpage

627

Lastpage

631

Abstract

An optimum weight initialization which strongly improves the performance of the back propagation (BP) algorithm is suggested. By statistical analysis, the scale factor, R (which is proportional to the maximum magnitude of the weights), is obtained as a function of the paralyzed neuron percentage (PNP). Also, by computer simulation, the performances on the convergence speed have been related to PNP. An optimum range for R is shown to exist in order to minimize the time needed to reach the minimum of the cost function. Normalization factors are properly defined, which leads to a distribution of the activations independent of the neurons, and to a single nondimensional quantity, R, the value of which can be quickly found by computer simulation

Keywords

convergence of numerical methods; minimisation; neural nets; statistical analysis; backpropagation; convergence; cost function; neural nets; optimisation; paralyzed neuron percentage; scale factor; statistically controlled activation weight initialisation; Application software; Computer simulation; Convergence; Cost function; Feedforward neural networks; Neural networks; Neurons; Statistical analysis; Testing; Weight control;

fLanguage

English

Journal_Title

Neural Networks, IEEE Transactions on

Publisher

ieee

ISSN

1045-9227

Type

jour

DOI

10.1109/72.143378

Filename

143378