An analysis of noisy recurrent neural networks

In this paper we examine the effect of noise on the typical recurrent neural network (RNN) model. We perform qualitative analysis which routinely derives the appropriate bounds on useful practical measures such as the mean and variance of the RNN outputs. As in the earlier deterministic work of Cohen and Grossberg (1983), it is clear that the boundedness of the often-used sigmoidal nonlinearity plays the key role in establishing these bounds. We then present design examples which illustrate that identically performing deterministic RNNs may exhibit significant performance variations in the presence of noise. Our analysis provides a way of evaluating competing designs for noise robustness