A multilayer neural network with nonlinear inputs and trainable activation functions: structure and simultaneous learning algorithm

Network size of neural networks is highly dependent on activation functions. A trainable activation function is proposed, which consists of a linear combination of some basic functions. The activation functions and the connection weights are simultaneously trained. An 8-bit parity problem can be solved by using a single output unit and no hidden unit. In this paper, we expand this model to multilayer neural networks. Furthermore, nonlinear functions are used at the unit inputs in order to realize more flexible transfer functions. The previous activation functions and the new nonlinear functions are also simultaneously trained. More complex pattern classification problems can be solved with a small number of units and fast convergence