Convergence Analysis of Node Fault Injection During Training

Improving fault tolerance of a neural network is an important issue that has been studied for more than two decades. Various algorithms have been proposed in sequel and many of them have succeeded in attaining a fault tolerant neural network. Amongst all, on-line node fault injection-based algorithms are one type of these algorithms. Despite its simple implementation, theoretical analyses on these algorithms are far from complete. In this paper, an on-line node fault injection training algorithm is studied. By node fault injection training, we assume that the hidden nodes are random neuron in which the output of these hidden nodes can be zeros in a random manner. So, in each step of update, we randomly set the hidden outputs to be zeros. The network output and the gradient vector are calculated with these zero-output hidden nodes, and thus apply the standard online weight algorithm to update the weight vector. The corresponding objective function is derived and the convergence of the algorithm is proved. By a theorem from H. White, we show that the weight vector obtained by this algorithm can converge with probability one. The weight vector converges to a local minimum of the objective function derived.