SinLU: Sinu-Sigmoidal Linear Unit
- Authors
- Paul, A.; Bandyopadhyay, R.; Yoon, Jin Hee; Geem, Zong Woo; Sarkar, R.
- Issue Date
- Feb-2022
- Publisher
- MDPI
- Keywords
- Activation function; CNN; Deep learning; Sigmoid function; Sinusoidal curve; Trainable parameter
- Citation
- Mathematics, v.10, no.3
- Journal Title
- Mathematics
- Volume
- 10
- Number
- 3
- URI
- https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/83563
- DOI
- 10.3390/math10030337
- ISSN
- 2227-7390
- Abstract
- Non-linear activation functions are integral parts of deep neural architectures. Given the large and complex dataset of a neural network, its computational complexity and approximation capability can differ significantly based on what activation function is used. Parameterizing an activation function with the introduction of learnable parameters generally improves the performance. Herein, a novel activation function called Sinu-sigmoidal Linear Unit (or SinLU) is proposed. SinLU is formulated as SinLU(x) = (x + a sin bx) · σ(x), where σ(x) is the sigmoid function. The proposed function incorporates the sine wave, allowing new functionalities over traditional linear unit activa-tions. Two trainable parameters of this function control the participation of the sinusoidal nature in the function, and help to achieve an easily trainable, and fast converging function. The performance of the proposed SinLU is compared against widely used activation functions, such as ReLU, GELU and SiLU. We showed the robustness of the proposed activation function by conducting experiments in a wide array of domains, using multiple types of neural network-based models on some standard datasets. The use of sine wave with trainable parameters results in a better performance of SinLU than commonly used activation functions. © 2022 by the authors. Licensee MDPI, Basel, Switzerland.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - IT융합대학 > 에너지IT학과 > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.