Simulated annealing of neural networks: The `cooling´ strategy reconsidered

The simulated annealing (SA) algorithm has been widely used to addressed intractable global optimizations in many fields, including training of artificial neural networks. Implementations of annealing universally always use a monotone decreasing, or cooling, temperature schedule which is motivated by the algorithm´s proof of optimality as well as analogies with statistical thermodynamics. The authors challenge this motivation: the fact that cooling schedules are optimal in theory is not related to the practical performance of the algorithm. The finding is based on a new best-so-far criterion for measuring the quality of annealing schedules. Motivated by studies of optimal schedules for small problems, highly nonstandard annealing schedules are studied for training of feedforward perceptron networks on a real-world sensor classification benchmark. Clear evidence is found that optimal schedules do not necessarily decrease monotonically to zero