Training high-dimensional neural networks with cooperative particle swarm optimiser

This paper analyses the behaviour of particle swarm optimisation applied to training high-dimensional neural networks. Despite being an established neural network training algorithm, particle swarm optimisation falls short at training high-dimensional neural networks. Reasons for poor performance of PSO are investigated in this paper, and hidden unit saturation is hypothesised to be a cause of the failure of PSO in training high-dimensional neural networks. An analysis of various activation functions and search space boundaries leads to the conclusion that hidden unit saturation can be slowed down by combining activation function choice with appropriate search space boundaries. Bounded search is shown to significantly outperform unbounded search in high-dimensional neural network error search spaces.