CTRL: Continual Representation Learning to Transfer Information of Pre-trained for WAV2VEC 2.0
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Lee, Jae-Hong | - |
dc.contributor.author | Lee, Chae-Won | - |
dc.contributor.author | Choi, Jin-Seong | - |
dc.contributor.author | Chang, Joon-Hyuk | - |
dc.contributor.author | Seong, Woo Kyeong | - |
dc.contributor.author | Lee, Jeonghan | - |
dc.date.accessioned | 2022-12-20T06:25:10Z | - |
dc.date.available | 2022-12-20T06:25:10Z | - |
dc.date.created | 2022-11-02 | - |
dc.date.issued | 2022-09 | - |
dc.identifier.issn | 2308-457X | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/173090 | - |
dc.description.abstract | Representation models such as WAV2VEC 2.0 (W2V2) show remarkable speech recognition performance by pre-training only on unlabeled datasets and finetuning on a small amount of labeled dataset. It is crucial to train on datasets of multiple domains to obtain a richer representation of such a model. The conventional approach used for handling multiple domains is training a model on a merged dataset from scratch. However, representation learning requires excessive computation for pre-training, which becomes a severe problem as the size of the dataset increases. In this study, we present continual representation learning (CTRL), a framework that leverages continual learning methods to continually retrain the pre-trained representation model while transferring information of the previous model without the historical dataset. The framework conducts continual pre-training for pre-trained W2V2 using the redesigned continual learning method for self-supervised learning. To evaluate our framework, we continually pre-train W2V2 with CTRL in the following order: Librispeech, Wall Street Journal, and TED-LIUM V3. The results demonstrate that the proposed approach improves the speech recognition performance of all three datasets compared with that of baseline W2V2 pre-trained on Librispeech. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | International Speech Communication Association | - |
dc.title | CTRL: Continual Representation Learning to Transfer Information of Pre-trained for WAV2VEC 2.0 | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Chang, Joon-Hyuk | - |
dc.identifier.doi | 10.21437/Interspeech.2022-10063 | - |
dc.identifier.scopusid | 2-s2.0-85140099591 | - |
dc.identifier.wosid | 000900724503113 | - |
dc.identifier.bibliographicCitation | Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, v.2022-September, pp.3398 - 3402 | - |
dc.relation.isPartOf | Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH | - |
dc.citation.title | Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH | - |
dc.citation.volume | 2022-September | - |
dc.citation.startPage | 3398 | - |
dc.citation.endPage | 3402 | - |
dc.type.rims | ART | - |
dc.type.docType | Proceedings Paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Acoustics | - |
dc.relation.journalResearchArea | Audiology & Speech-Language Pathology | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalWebOfScienceCategory | Acoustics | - |
dc.relation.journalWebOfScienceCategory | Audiology & Speech-Language Pathology | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Artificial Intelligence | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.subject.keywordPlus | Learning systems | - |
dc.subject.keywordPlus | Speech communication | - |
dc.subject.keywordPlus | Supervised learning | - |
dc.subject.keywordPlus | Speech recognition | - |
dc.subject.keywordPlus | Continual learning | - |
dc.subject.keywordPlus | Domain adaptation | - |
dc.subject.keywordPlus | Learning methods | - |
dc.subject.keywordPlus | Multiple domains | - |
dc.subject.keywordPlus | Pre-training | - |
dc.subject.keywordPlus | Representation learning | - |
dc.subject.keywordPlus | Representation model | - |
dc.subject.keywordPlus | Semi-supervised learning | - |
dc.subject.keywordPlus | Speech recognition performance | - |
dc.subject.keywordPlus | Transfer information | - |
dc.subject.keywordAuthor | continual learning | - |
dc.subject.keywordAuthor | domain adaptation | - |
dc.subject.keywordAuthor | representation learning | - |
dc.subject.keywordAuthor | semi-supervised learning | - |
dc.subject.keywordAuthor | speech recognition | - |
dc.identifier.url | https://www.isca-speech.org/archive/interspeech_2022/lee22i_interspeech.html | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.