On the stability of symmetric adaptive decorrelation

Adaptive decorrelation was introduced as an effective way to speed up the training of feedforward neural networks by performing a data orthonormalization at each layer of multi-layer networks. The algorithm is implemented using a single linear layer of unsupervised neurons and can be used as a data pre-processing scheme in any situation where orthonormality is a desirable feature of the input data. However, due to the symmetric structure of the algorithm, the final weight matrix is dependent on the initial conditions and, moreover, it can slowly change in time, even in stationary conditions, due to numerical errors. A similar problem can be found on symmetric unsupervised algorithms which compute principal subspace projections. This paper outlines the main properties of adaptive decorrelation and introduces a closed form solution for the network weights. The stability of the algorithm is considered and it is shown how a minor modification of the weight update rule is able to assure stability in stationary conditions