Efficient Federated Learning with Pre-Trained Large Language Model Using Several Adapter Mechanisms
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Gyunyeop | - |
dc.contributor.author | Yoo, Joon | - |
dc.contributor.author | Kang, Sangwoo | - |
dc.date.accessioned | 2023-12-15T15:08:18Z | - |
dc.date.available | 2023-12-15T15:08:18Z | - |
dc.date.issued | 2023-11 | - |
dc.identifier.issn | 2227-7390 | - |
dc.identifier.issn | 2227-7390 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/89516 | - |
dc.description.abstract | Recent advancements in deep learning have led to various challenges, one of which is the issue of data privacy in training data. To address this issue, federated learning, a technique that merges models trained by clients on servers, has emerged as an attractive solution. However, federated learning faces challenges related to data heterogeneity and system heterogeneity. Recent observations suggest that incorporating pre-trained models into federated learning can mitigate some of these challenges. Nonetheless, the main drawback of pre-trained models lies in their typically large model size, leading to excessive data transmission when clients send these models to the server. Additionally, federated learning involves multiple global steps, which means transmitting a large language model to multiple clients results in too much data exchange. In this paper, we propose a novel approach to address this challenge using adapters. Adapters demonstrate training efficiency by training a small capacity adapter layer alongside a large language model. This unique characteristic reduces the volume of data transmission, offering a practical solution to the problem. The evaluation results demonstrate that the proposed method achieves a reduction in training time of approximately 20-40% and a transmission speed improvement of over 98% compared to previous approaches. | - |
dc.language | 영어 | - |
dc.language.iso | ENG | - |
dc.publisher | MDPI | - |
dc.title | Efficient Federated Learning with Pre-Trained Large Language Model Using Several Adapter Mechanisms | - |
dc.type | Article | - |
dc.identifier.wosid | 001100441000001 | - |
dc.identifier.doi | 10.3390/math11214479 | - |
dc.identifier.bibliographicCitation | MATHEMATICS, v.11, no.21 | - |
dc.description.isOpenAccess | Y | - |
dc.identifier.scopusid | 2-s2.0-85176608751 | - |
dc.citation.title | MATHEMATICS | - |
dc.citation.volume | 11 | - |
dc.citation.number | 21 | - |
dc.type.docType | Article | - |
dc.publisher.location | 스위스 | - |
dc.subject.keywordAuthor | federated learning | - |
dc.subject.keywordAuthor | deep learning | - |
dc.subject.keywordAuthor | transfer learning | - |
dc.subject.keywordAuthor | adapter transformer | - |
dc.relation.journalResearchArea | Mathematics | - |
dc.relation.journalWebOfScienceCategory | Mathematics | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
1342, Seongnam-daero, Sujeong-gu, Seongnam-si, Gyeonggi-do, Republic of Korea(13120)031-750-5114
COPYRIGHT 2020 Gachon University All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.