Learning by α-divergence

In the present paper, we propose a new cost function, called α-divergence, which is a generalized version of the relative entropy or the Kullback´s divergence measure in neural network. The most fundamental characteristics of this α-divergence are summarized by the following three points: 1) by changing the parameter α for the α-divergence, multiple cost functions can be obtained to be used for different purposes or problems; 2) α-divergence is effective in direct proportion to the error between targets and outputs, eliminating the derivative of the sigmoidal function; and 3) the α-divergence has an effect on eliminating saturated units. We formulated an update rule to minimize α-divergence, and applied the method to the acquisition of the grammatical competence. Experimental results confirmed marked improvement in the generalization by using α-divergence. This improvement is due to the property of α-divergence whose derivative is effective especially for eliminating saturated units