Regularization in network optimization via trimmed stochastic gradient descent with noisy labelopen access
- Authors
- Nakamura, K.; Sohn, Bong Soo; Won, K.; Hong, Byung-Woo
- Issue Date
- May-2022
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- Data models; data trimming; label noise; Loss measurement; network optimization; Neural networks; Noise measurement; Optimization; regularization; Stochastic processes; Training
- Citation
- IEEE Access, v.10, pp 34706 - 34715
- Pages
- 10
- Journal Title
- IEEE Access
- Volume
- 10
- Start Page
- 34706
- End Page
- 34715
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/58095
- DOI
- 10.1109/ACCESS.2022.3171910
- ISSN
- 2169-3536
- Abstract
- Regularization is essential for avoiding over-fitting to training data in network optimization, leading to better generalization of the trained networks. The label noise provides a strong implicit regularization by replacing the target ground truth labels of training examples by uniform random labels. However, it can cause undesirable misleading gradients due to the large loss associated with incorrect labels. We propose a first-order optimization method (Label-Noised Trim-SGD) that uses the label noise with the example trimming in order to remove the outliers based on the loss. The proposed algorithm is simple yet enables us to impose a large label-noise and obtain a better regularization effect than the original methods. The quantitative analysis is performed by comparing the behavior of the label noise, the example trimming, and the proposed algorithm. We also present empirical results that demonstrate the effectiveness of our algorithm using the major benchmarks and the fundamental networks, where our method has successfully outperformed the state-of-the-art optimization methods. Author
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Software > Department of Artificial Intelligence > 1. Journal Articles
- College of Software > School of Computer Science and Engineering > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/58095)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.