Parametric rectified nonlinear unit (PRenu) for convolution neural networks

Activation function unit is an extremely important part of convolution neural networks; it is the nonlinear transformation that we do over the input data. Using hidden layer incorporating with a well-chosen activation function improves both the accuracy and the CNN convergence speed. This paper proposes a parametric rectified nonlinear function unit (PRenu). The proposed activation function is nearly similar to Relu. It returns x - alpha log(x+1) for positive values (alpha is between 0 and 1) and zero for negative parts. In contrast to Relu that returns the same received gradient for all positive values in its back-propagation, the PRenu multiplies it by values between 1-alpha and 1 depending on the value with which each neuron was involved. The PRenu has been tested on three datasets: CIFAR-10, CIFAR-100 and Oxflower17, and compared to the activation function Relu. The experimental results show that using the proposed activation function PRenu, the CNN convergence is faster and the accuracy is also improved.

Automated summary

The proposed parametric rectified nonlinear function unit (PRenu) is similar to Relu but differentiates by providing a non-linear transformation for positive values and improves CNN convergence speed and accuracy.