4.7 Article

Enhancement of neural networks with an alternative activation function tanhLU

Journal

EXPERT SYSTEMS WITH APPLICATIONS
Volume 199, Issue -, Pages -

Publisher

PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.eswa.2022.117181

Keywords

Neural networks; Activation function; tanhLUs

Funding

  1. The Pearl River Talent Recruitment Program in 2019 [2019CX01G338]
  2. Guangdong Province
  3. Shantou University [NTF19024-2019]

Ask authors/readers for more resources

tanhLU is a novel activation function that combines hyperbolic tangent function and linear unit, inspired by the infinity of ReLU and symmetry of tanh. It can control activation values and gradients through three variable parameters and has shown promising performance across various neural networks.
A novel activation function (referred to as tanhLU) that integrates hyperbolic tangent function (tanh) with a linear unit is proposed as a promising alternative to tanh for neural networks. The tanhLU is inspired by the boundlessness of rectified linear unit (ReLU) and the symmetry of tanh. Three variable parameters in tanhLU controlling activation values and gradients could be preconfigured as constants or adaptively optimized during the training process. The capacity of tanhLU is first investigated by checking the weight gradients in error back propagation. Experiments are conducted to validate the improvement of tanhLUs on five types of neural networks, based on seven benchmark datasets in different domains. tanhLU is then applied to predict the highly nonlinear stress-strain relationship of soils by using the multiscale stress-strain (MSS) dataset. The experiment results indicate that using constant tanhLU leads to apparent improvement on FCNN and LSTM with lower loss and higher accuracy compared with tanh. Adaptive tanhLUs achieved the state-of-the-art performance for multiple deep neural networks in image classification and face recognition.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.7
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

No Data Available
No Data Available