DPReLU: Dynamic Parametric Rectified Linear Unit and Its Proper Weight Initialization Method

Activation functions are essential in deep learning, and the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, owing to the dying ReLU problem and bias shift effect, deep learning models using ReLU cannot exploit the potential benefits of negative values. Numerous ReLU variants have been proposed to address this issue. In this study, we propose Dynamic Parametric ReLU (DPReLU), which can dynamically control the overall functional shape of ReLU with four learnable parameters. The parameters of DPReLU are determined by training rather than by humans, thereby making the formulation more suitable and flexible for each model and dataset. Furthermore, we propose an appropriate and robust weight initialization method for DPReLU. To evaluate DPReLU and its weight initialization method, we performed two experiments on various image datasets: one using an autoencoder for image generation and the other using the ResNet50 for image classification. The results show that DPReLU and our weight initialization method provide faster convergence and better accuracy than the original ReLU and the previous ReLU variants.

Automated summary

Activation functions play a crucial role in deep learning, and the rectified linear unit (ReLU) has become the most widely used activation function due to its ability to address the vanishing gradient issue. However, ReLU suffers from the dying ReLU problem and bias shift effect, limiting its ability to utilize negative values effectively. To tackle this problem, numerous variants of ReLU have been proposed. In this study, Dynamic Parametric ReLU (DPReLU) is introduced, which allows for dynamic control of the overall shape of ReLU through four learnable parameters. The parameters of DPReLU are determined through training, making it more suitable and flexible for each model and dataset. Additionally, an appropriate and robust weight initialization method for DPReLU is proposed. Experimental results on various image datasets demonstrate that DPReLU and the weight initialization method lead to faster convergence and better accuracy compared to the original ReLU and previous variants.