Implicit Jacobian Regularization Weighted with Impurity of Probability Output
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Sungyoon | - |
dc.contributor.author | Park, Jinseong | - |
dc.contributor.author | Lee, Jaewook | - |
dc.date.accessioned | 2023-11-14T08:15:09Z | - |
dc.date.available | 2023-11-14T08:15:09Z | - |
dc.date.created | 2023-11-01 | - |
dc.date.issued | 2023-07 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/192170 | - |
dc.description.abstract | The success of deep learning is greatly attributed to stochastic gradient descent (SGD), yet it remains unclear how SGD finds well-generalized models. We demonstrate that SGD has an implicit regularization effect on the logit-weight Jacobian norm of neural networks. This regularization effect is weighted with the impurity of the probability output, and thus it is active in a certain phase of training. Moreover, based on these findings, we propose a novel optimization method that explicitly regularizes the Jacobian norm, which leads to similar performance as other state-of-the-art sharpness-aware optimization methods. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | ML Research Press | - |
dc.title | Implicit Jacobian Regularization Weighted with Impurity of Probability Output | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Lee, Sungyoon | - |
dc.identifier.scopusid | 2-s2.0-85174424115 | - |
dc.identifier.bibliographicCitation | Proceedings of Machine Learning Research, v.202, pp.19094 - 19140 | - |
dc.relation.isPartOf | Proceedings of Machine Learning Research | - |
dc.citation.title | Proceedings of Machine Learning Research | - |
dc.citation.volume | 202 | - |
dc.citation.startPage | 19094 | - |
dc.citation.endPage | 19140 | - |
dc.type.rims | ART | - |
dc.type.docType | Conference paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.subject.keywordPlus | Deep learning | - |
dc.subject.keywordPlus | Gradient methods | - |
dc.subject.keywordPlus | Optimization | - |
dc.subject.keywordPlus | Stochastic models | - |
dc.identifier.url | https://proceedings.mlr.press/v202/lee23q.html | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.