Multi-sigmoidal neural networks and back-propagation

A new neural network architecture based on dynamically created nonmonotonic activation functions that are modeled by a set of sigmoidal functions is introduced. A modification of the backpropagation algorithm is presented that is capable to learn both the weights and the unit activation functions themselves. The new architecture reveals an existing tradeoff in capturing interactions of the inputs via hidden units or nonmonotonic activation functions. In the classification problems examined, the new architecture resulted in very shallow networks with fewer degrees of freedom than the corresponding backpropagation networks. Those networks converged very fast to a solution and resulted in optimal or nearly optimal sigmoidal configurations