SHAT: A Novel Asynchronous Training Algorithm That Provides Fast Model Convergence in Distributed Deep Learning
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Ko, Yunyong | - |
dc.contributor.author | Kim, Sang-Wook | - |
dc.date.accessioned | 2022-07-06T02:15:36Z | - |
dc.date.available | 2022-07-06T02:15:36Z | - |
dc.date.created | 2022-01-26 | - |
dc.date.issued | 2022-01 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/138459 | - |
dc.description.abstract | The recent unprecedented success of deep learning (DL) in various fields is underlied by its use of large-scale data and models. Training a large-scale deep neural network (DNN) model with large-scale data, however, is time-consuming. To speed up the training of massive DNN models, data-parallel distributed training based on the parameter server (PS) has been widely applied. In general, a synchronous PS-based training suffers from the synchronization overhead, especially in heterogeneous environments. To reduce the synchronization overhead, asynchronous PS-based training employs the asynchronous communication between PS and workers so that PS processes the request of each worker independently without waiting. Despite the performance improvement of asynchronous training, however, it inevitably incurs the difference among the local models of workers, where such a difference among workers may cause slower model convergence. Fro addressing this problem, in this work, we propose a novel asynchronous PS-based training algorithm, SHAT that considers (1) the scale of distributed training and (2) the heterogeneity among workers for successfully reducing the difference among the local models of workers. The extensive empirical evaluation demonstrates that (1) the model trained by SHAT converges to the higher accuracy up to 5.22% than state-of-the-art algorithms, and (2) the model convergence of SHAT is robust under various heterogeneous environments. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | MDPI | - |
dc.title | SHAT: A Novel Asynchronous Training Algorithm That Provides Fast Model Convergence in Distributed Deep Learning | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Kim, Sang-Wook | - |
dc.identifier.doi | 10.3390/app12010292 | - |
dc.identifier.scopusid | 2-s2.0-85121979484 | - |
dc.identifier.wosid | 000741122100001 | - |
dc.identifier.bibliographicCitation | APPLIED SCIENCES-BASEL, v.12, no.1, pp.1 - 14 | - |
dc.relation.isPartOf | APPLIED SCIENCES-BASEL | - |
dc.citation.title | APPLIED SCIENCES-BASEL | - |
dc.citation.volume | 12 | - |
dc.citation.number | 1 | - |
dc.citation.startPage | 1 | - |
dc.citation.endPage | 14 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | Y | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Chemistry | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Materials Science | - |
dc.relation.journalResearchArea | Physics | - |
dc.relation.journalWebOfScienceCategory | Chemistry, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Engineering, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Materials Science, Multidisciplinary | - |
dc.relation.journalWebOfScienceCategory | Physics, Applied | - |
dc.subject.keywordAuthor | distributed deep learning | - |
dc.subject.keywordAuthor | data parallelism | - |
dc.subject.keywordAuthor | PS-based distributed training | - |
dc.subject.keywordAuthor | heterogeneous environments | - |
dc.identifier.url | https://www.mdpi.com/2076-3417/12/1/292 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.