Journal
EXPERT SYSTEMS WITH APPLICATIONS
Volume 199, Issue -, Pages -Publisher
PERGAMON-ELSEVIER SCIENCE LTD
DOI: 10.1016/j.eswa.2022.117181
Keywords
Neural networks; Activation function; tanhLUs
Categories
Funding
- The Pearl River Talent Recruitment Program in 2019 [2019CX01G338]
- Guangdong Province
- Shantou University [NTF19024-2019]
Ask authors/readers for more resources
tanhLU is a novel activation function that combines hyperbolic tangent function and linear unit, inspired by the infinity of ReLU and symmetry of tanh. It can control activation values and gradients through three variable parameters and has shown promising performance across various neural networks.
A novel activation function (referred to as tanhLU) that integrates hyperbolic tangent function (tanh) with a linear unit is proposed as a promising alternative to tanh for neural networks. The tanhLU is inspired by the boundlessness of rectified linear unit (ReLU) and the symmetry of tanh. Three variable parameters in tanhLU controlling activation values and gradients could be preconfigured as constants or adaptively optimized during the training process. The capacity of tanhLU is first investigated by checking the weight gradients in error back propagation. Experiments are conducted to validate the improvement of tanhLUs on five types of neural networks, based on seven benchmark datasets in different domains. tanhLU is then applied to predict the highly nonlinear stress-strain relationship of soils by using the multiscale stress-strain (MSS) dataset. The experiment results indicate that using constant tanhLU leads to apparent improvement on FCNN and LSTM with lower loss and higher accuracy compared with tanh. Adaptive tanhLUs achieved the state-of-the-art performance for multiple deep neural networks in image classification and face recognition.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available