A Reparametrization-Invariant Sharpness Measure Based on Information Geometry
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Jang, Cheongjae | - |
dc.contributor.author | Lee, Sungyoon | - |
dc.contributor.author | Park, Frank C. | - |
dc.contributor.author | Noh, Yung Kyun | - |
dc.date.accessioned | 2023-08-07T07:43:11Z | - |
dc.date.available | 2023-08-07T07:43:11Z | - |
dc.date.created | 2023-07-21 | - |
dc.date.issued | 2022-12 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/188887 | - |
dc.description.abstract | It has been observed that the generalization performance of neural networks correlates with the sharpness of their loss landscape. Dinh et al. (2017) have observed that existing formulations of sharpness measures fail to be invariant with respect to scaling and reparametrization. While some scale-invariant measures have recently been proposed, reparametrization-invariant measures are still lacking. Moreover, they often do not provide any theoretical insights into generalization performance nor lead to practical use to improve the performance. Based on an information geometric analysis of the neural network parameter space, in this paper we propose a reparametrization-invariant sharpness measure that captures the change in loss with respect to changes in the probability distribution modeled by neural networks, rather than with respect to changes in the parameter values. We reveal some theoretical connections of our measure to generalization performance. In particular, experiments confirm that using our measure as a regularizer in neural network training significantly improves performance. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Neural Information Processing Systems | - |
dc.title | A Reparametrization-Invariant Sharpness Measure Based on Information Geometry | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Noh, Yung Kyun | - |
dc.identifier.bibliographicCitation | Neural Information Processing Systems, pp.1 - 13 | - |
dc.relation.isPartOf | Neural Information Processing Systems | - |
dc.citation.title | Neural Information Processing Systems | - |
dc.citation.startPage | 1 | - |
dc.citation.endPage | 13 | - |
dc.type.rims | ART | - |
dc.type.docType | Proceeding | - |
dc.description.journalClass | 3 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | other | - |
dc.identifier.url | https://proceedings.neurips.cc/paper_files/paper/2022/hash/b2ba568effcc3ab221912db2fb095ea9-Abstract-Conference.html | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.