딥러닝을 위한 비단조 활성화 함수

Abstract

The activation function significantly affects the performance of neural networks. Among the numerous functions, the Rectified Linear Unit(ReLU) is widely used in many deep learning applications owing to its simplicity and performance. This study proposes a new nonlinear activation function derived from logarithmic and hyperbolic tangent functions. It exhibits the following distinct characteristics: 1) If the input is greater than 0, then the output is the same as the input, 2) if the input is approximately 0, then the output exhibits non-linear characteristics, and 3) if the input is negative infinity, then the output has a value of approximately zero. Simulation results show that the proposed activation function surpasses the ReLU, Mish, and Power Function Linear Units in terms of classification accuracy. In particular, when applied to the CIFAR-10 classification using the VGG19 network, it increases the accuracy by approximately 1%.