Hybrid tuning of activation functions in feedforward neural networks

Tuning procedures for activation functions significantly increases the flexibility and the nonlinear approximation capability of feedforward neural networks in supervised learning tasks. As a consequence, the learning process presents a better performance, with the final state of the neural network being kept away from undesired saturation regions. Based on a hybrid architecture combining a gradient strategy with a fuzzy decision model, an auto-tuning algorithm is derived to adjust additional parameters associated with the activation functions. The other conventional parameters, the connection weights between layers, are adjusted using a powerful second-order approach based on a conjugate gradient algorithm. To demonstrate the performance of the proposed method we compare this technique with the standard algorithm and with an auto-tuning strategy based solely on the gradient descent method. The three algorithm are applied to several artificial and real world benchmarks