Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Minsoo | - |
dc.contributor.author | Lee, Sihwa | - |
dc.contributor.author | Hong, Sukjin | - |
dc.contributor.author | Chang, Du-Seong | - |
dc.contributor.author | Choi, Jung wook | - |
dc.date.accessioned | 2023-05-03T09:39:33Z | - |
dc.date.available | 2023-05-03T09:39:33Z | - |
dc.date.created | 2023-04-06 | - |
dc.date.issued | 2022-12 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/184845 | - |
dc.description.abstract | Knowledge distillation (KD) has been a ubiquitous method for model compression to strengthen the capability of a lightweight model with the transferred knowledge from the teacher. In particular, KD has been employed in quantization-aware training (QAT) of Transformer encoders like BERT to improve the accuracy of the student model with the reduced-precision weight parameters. However, little is understood about which of the various KD approaches best fits the QAT of Transformers. In this work, we provide an in-depth analysis of the mechanism of KD on attention recovery of quantized large Transformers. In particular, we reveal that the previously adopted MSE loss on the attention score is insufficient for recovering the self-attention information. Therefore, we propose two KD methods; attention-map and attention-output losses. Furthermore, we explore the unification of both losses to address task-dependent preference between attention-map and output losses. The experimental results on various Transformer encoder models demonstrate that the proposed KD methods achieve state-of-the-art accuracy for QAT with sub-2-bit weight quantization. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Association for Computational Linguistics (ACL) | - |
dc.title | Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoders | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Choi, Jung wook | - |
dc.identifier.doi | 10.48550/arXiv.2211.11014 | - |
dc.identifier.scopusid | 2-s2.0-85149442817 | - |
dc.identifier.bibliographicCitation | Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pp.6713 - 6725 | - |
dc.relation.isPartOf | Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 | - |
dc.citation.title | Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 | - |
dc.citation.startPage | 6713 | - |
dc.citation.endPage | 6725 | - |
dc.type.rims | ART | - |
dc.type.docType | Conference Paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.subject.keywordPlus | Computational linguistics | - |
dc.subject.keywordPlus | Personnel training | - |
dc.subject.keywordPlus | Signal encoding | - |
dc.subject.keywordPlus | Distillation | - |
dc.subject.keywordPlus | Best fit | - |
dc.subject.keywordPlus | Distillation method | - |
dc.subject.keywordPlus | In-depth analysis | - |
dc.subject.keywordPlus | Model compression | - |
dc.subject.keywordPlus | Quantisation | - |
dc.subject.keywordPlus | Reduced precision | - |
dc.subject.keywordPlus | State of the art | - |
dc.subject.keywordPlus | Student Modeling | - |
dc.subject.keywordPlus | Teachers&apos | - |
dc.subject.keywordPlus | - | |
dc.subject.keywordPlus | Weight parameters | - |
dc.identifier.url | https://arxiv.org/abs/2211.11014 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.