Adaptive Regularization via Residual Smoothing in Deep Learning Optimizationopen access
- Authors
- Cho, Junghee; Kwon, Junseok; Hong, Byung-Woo
- Issue Date
- Aug-2019
- Publisher
- IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
- Keywords
- Adaptive regularization; deep learning optimization; residual smoothing
- Citation
- IEEE ACCESS, v.7, pp 122889 - 122899
- Pages
- 11
- Journal Title
- IEEE ACCESS
- Volume
- 7
- Start Page
- 122889
- End Page
- 122899
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/39072
- DOI
- 10.1109/ACCESS.2019.2938200
- ISSN
- 2169-3536
- Abstract
- We present an adaptive regularization algorithm that can be effectively applied to the optimization problem in deep learning framework. Our regularization algorithm aims to take into account the fitness of data to the current state of model in the determination of regularity to achieve better generalization. The degree of regularization at each element in the target space of the neural network architecture is determined based on the residual at each optimization iteration in an adaptive way. Our adaptive regularization algorithm is designed to apply a diffusion process driven by the heat equation with spatially varying diffusivity depending on the probability density function following a certain distribution of residual. Our data-driven regularity is imposed by adaptively smoothing a simplified objective function in which the explicit regularization term is omitted in an alternating manner between the evaluation of residual and the determination of the degree of its regularity. The effectiveness of our algorithm is empirically demonstrated by the numerical experiments in the application of image classification problems, indicating that our algorithm outperforms other commonly used optimization algorithms in terms of generalization using popular deep learning models and benchmark datasets.
- Files in This Item
-
- Appears in
Collections - College of Software > Department of Artificial Intelligence > 1. Journal Articles
- College of Software > School of Computer Science and Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.