Bi-modal derivative activation function for sigmoidal feedforward networks

A new class of activation functions is proposed as the sum of shifted log-sigmoid activation functions. This has the effect of making the derivative of the activation function with respect to the net inputs, be bi-modal. That is, the derivative of the activation functions has two maxima of equal values for nonzero values of the parameter, that parametrises the proposed class of activation functions. On a set of ten function approximation tasks, the usage of the proposed activation function demonstrates that there exists network(s), using the proposed activation, and are able to achieve lower generalisation error, in equal epochs of training using the resilient backpropagation algorithm. On a set of four benchmark problems taken from UCI machine learning repository, for which the networks are trained using the resilient backpropagation algorithm, the scaled conjugate algorithm, the Levenberg-Marquardt algorithm and the quasi-Newton BFGS algorithm, we observe that the usage of the proposed algorithms leads to better generalisation results, similar to the results for the ten function approximation tasks wherein the networks were trained using the resilient backpropagation algorithm. (C) 2014 Elsevier B.V. All rights reserved.