Robust neural network online learning in time-variant regression models

We consider robust online learning in time-variant neural network regression models. Using a state space representation for the neural network´s weight evolution in time we derive weight estimates by maximizing posterior modes via the Fisher scoring algorithm. By taking the family of densities as the output error cost function we get a robust error measure suitable for handling additive outliers. Fisher scoring was implemented using a forward backward pass of fixed length through the data set for every time step resulting in so-called online smoothing algorithms. Furthermore, we derive an EM-type algorithm for approximate maximum likelihood estimation of unknown hyperparameters. Our experiments show that online posterior mode weight smoothing outperforms standard online methods like online backpropagation and extended Kalman filtering, both for Gaussian measurements and non-Gaussian measurements with additive outliers