Logish: A new nonlinear nonmonotonic activation function for convolutional neural network

Activation function is an important component of the convolutional neural network. Recently, nonlinear nonmonotonic activation functions such as Swish and Mish have illustrated good performance in deep learning structures. In this paper, we propose a new nonlinear nonmonotonic activation function called Logish, which can be represented by f (x) = x . ln[1 + sigmoid (x)]. Firstly, we take the logarithmic operation to reduce the numerical range of sigmoid (x) + 1, then we employ variable x to make the negative output have a strong regularization effect. Furthermore, we evaluate the image classification performance of Logish and its variant f (x) = alpha x . ln[1 + sigmoid (beta x)] in simple and complex networks with top-1 accuracy. Experimental results demonstrate that Logish's variant (alpha = 1; beta = 10) can achieve 94.8% top-1 accuracy with ResNet-50 network on CIFAR 10 dataset, and can reach 99.24% top-1 accuracy with DenseNet on MNIST dataset and 88.52% top-1 accuracy with SE-Inception-v4 network on SVHN dataset respectively. It is higher than the Sigmoid, Tanh, ReLU, Swish and Mish activation functions in the corresponding dataset. It also verifies the performance and effectiveness of Logish. (C) 2021 Elsevier B.V. All rights reserved.

Automated summary

In this paper, a new nonlinear nonmonotonic activation function called Logish is proposed, demonstrating better performance in image classification tasks compared to other common activation functions.