Range-Invariant Approximation of Non-Linear Operations for Efficient BERT Fine-Tuning
- Authors
- Kim, Janghyeon; Lee, Janghwan; Choi, Jungwook; Han, Jeongho; Lee, Sangheon
- Issue Date
- Jul-2023
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- BERT; look-up table approximation; non-linear operation; training; Transformer
- Citation
- Proceedings - Design Automation Conference, v.2023-July, pp.1 - 6
- Indexed
- SCOPUS
- Journal Title
- Proceedings - Design Automation Conference
- Volume
- 2023-July
- Start Page
- 1
- End Page
- 6
- URI
- https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/192211
- DOI
- 10.1109/DAC56929.2023.10247958
- Abstract
- This paper proposes a range-invariant approximation of non-linear operations for training computations of Transformer-based large language models. The proposed method decomposes the approximation into the scaling and the range-invariant resolution for LUT approximation, covering diverse data ranges of non-linear operations with drastically reduced LUT entries during task-dependent BERT fine-tuning. We demonstrate that the proposed method robustly approximates all the non-linear operations of BERT without score degradation on challenging GLUE benchmarks using only a single-entry LUT, facilitating 52% area savings in hardware implementation.
- Files in This Item
-
Go to Link
- Appears in
Collections - 서울 공과대학 > 서울 융합전자공학부 > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/192211)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.