Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Janghyeon | - |
dc.contributor.author | Lee, Janghwan | - |
dc.contributor.author | Choi, Jungwook | - |
dc.contributor.author | Han, Jeongho | - |
dc.contributor.author | Lee, Sangheon | - |
dc.date.accessioned | 2023-11-14T08:23:46Z | - |
dc.date.available | 2023-11-14T08:23:46Z | - |
dc.date.created | 2023-10-16 | - |
dc.date.issued | 2023-07 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/192211 | - |
dc.description.abstract | This paper proposes a range-invariant approximation of non-linear operations for training computations of Transformer-based large language models. The proposed method decomposes the approximation into the scaling and the range-invariant resolution for LUT approximation, covering diverse data ranges of non-linear operations with drastically reduced LUT entries during task-dependent BERT fine-tuning. We demonstrate that the proposed method robustly approximates all the non-linear operations of BERT without score degradation on challenging GLUE benchmarks using only a single-entry LUT, facilitating 52% area savings in hardware implementation. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Institute of Electrical and Electronics Engineers Inc. | - |
dc.title | Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Choi, Jungwook | - |
dc.identifier.doi | 10.1109/DAC56929.2023.10247958 | - |
dc.identifier.scopusid | 2-s2.0-85173097937 | - |
dc.identifier.wosid | 001073487300265 | - |
dc.identifier.bibliographicCitation | Proceedings - Design Automation Conference, v.2023-July, pp.1 - 6 | - |
dc.relation.isPartOf | Proceedings - Design Automation Conference | - |
dc.citation.title | Proceedings - Design Automation Conference | - |
dc.citation.volume | 2023-July | - |
dc.citation.startPage | 1 | - |
dc.citation.endPage | 6 | - |
dc.type.rims | ART | - |
dc.type.docType | Proceedings Paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Artificial Intelligence | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.subject.keywordPlus | BERT | - |
dc.subject.keywordPlus | Fine tuning | - |
dc.subject.keywordPlus | Invariant approximations | - |
dc.subject.keywordPlus | Language model | - |
dc.subject.keywordPlus | Linear operations | - |
dc.subject.keywordPlus | Look-up table approximation | - |
dc.subject.keywordPlus | Lookup tables (LUTs) | - |
dc.subject.keywordPlus | Non linear | - |
dc.subject.keywordPlus | Non-linear operation | - |
dc.subject.keywordPlus | Transformer | - |
dc.subject.keywordPlus | Table lookup | - |
dc.subject.keywordAuthor | BERT | - |
dc.subject.keywordAuthor | look-up table approximation | - |
dc.subject.keywordAuthor | non-linear operation | - |
dc.subject.keywordAuthor | training | - |
dc.subject.keywordAuthor | Transformer | - |
dc.identifier.url | https://ieeexplore.ieee.org/document/10247958 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.