How Does a Transformer Learn Compression? An Attention Study on Huffman and LZ4
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Seo, Beomseok | - |
dc.contributor.author | No, Albert | - |
dc.date.accessioned | 2024-01-03T06:00:13Z | - |
dc.date.available | 2024-01-03T06:00:13Z | - |
dc.date.issued | 2023 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hongik/handle/2020.sw.hongik/32404 | - |
dc.description.abstract | Transformers have excelled in natural language processing and vision domains. This leads to the intriguing proposition: can Transformers be adapted to a more generalized framework, such as understanding general finite state machines? To explore this, we trained a Transformer network on compression algorithms such as Huffman and LZ4, viewing them as stepping stones towards mastering general finite state machines. Our analysis indicates that Transformers can adeptly internalize these methods and replicate essential states, echoing human-like interpretation via their attention mechanisms. This provides evidence of their capability to decipher practical finite state machines. Examining the attention maps offers insights into the Transformer's methodology when engaging with these compression techniques. In Huffman encodings, the encoder predominantly focuses on input statistics to define the present state, which the decoder subsequently leverages to produce output bits. Remarkably, with a 2nd-order Markov input, the encoder's attention is prominently directed at the previous two symbols, underscoring its proficiency in summarizing input statistics. The cross-attention maps further elucidate the exact association between input symbols and output bits. For the more complex LZ4 compression, the attention maps vividly portray the Transformer's processing of input sequences and the close linkage between the input and resulting output bit stream. This study underscores the Transformer's proficiency in comprehending compression algorithms and its keen ability to grasp input statistics, implying that its mechanisms, as illustrated by attention maps, provide profound interpretability. | - |
dc.format.extent | 10 | - |
dc.language | 영어 | - |
dc.language.iso | ENG | - |
dc.publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC | - |
dc.title | How Does a Transformer Learn Compression? An Attention Study on Huffman and LZ4 | - |
dc.type | Article | - |
dc.publisher.location | 미국 | - |
dc.identifier.doi | 10.1109/ACCESS.2023.3341512 | - |
dc.identifier.scopusid | 2-s2.0-85179829744 | - |
dc.identifier.wosid | 001129536100001 | - |
dc.identifier.bibliographicCitation | IEEE ACCESS, v.11, pp 140559 - 140568 | - |
dc.citation.title | IEEE ACCESS | - |
dc.citation.volume | 11 | - |
dc.citation.startPage | 140559 | - |
dc.citation.endPage | 140568 | - |
dc.type.docType | Article | - |
dc.description.isOpenAccess | Y | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.subject.keywordAuthor | Attention | - |
dc.subject.keywordAuthor | compression | - |
dc.subject.keywordAuthor | finite state machine | - |
dc.subject.keywordAuthor | Huffman coding | - |
dc.subject.keywordAuthor | LZ4 | - |
dc.subject.keywordAuthor | transformer | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
94, Wausan-ro, Mapo-gu, Seoul, 04066, Korea02-320-1314
COPYRIGHT 2020 HONGIK UNIVERSITY. ALL RIGHTS RESERVED.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.