Out-Of-Order BackProp: An Effective Scheduling Technique for Deep Learning
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Oh, Hyungjun | - |
dc.contributor.author | Lee, Junyeol | - |
dc.contributor.author | Kim, Hyeongju | - |
dc.contributor.author | Seo, Jiwon | - |
dc.date.accessioned | 2022-07-06T06:24:11Z | - |
dc.date.available | 2022-07-06T06:24:11Z | - |
dc.date.created | 2022-05-04 | - |
dc.date.issued | 2022-04 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/138969 | - |
dc.description.abstract | Neural network training requires a large amount of computation and thus GPUs are often used for the acceleration. While they improve the performance, GPUs are underutilized during the training. This paper proposes out-of-order (ooo) back-prop, an effective scheduling technique for neural network training. By exploiting the dependencies of gradient computations, ooo backprop enables to reorder their executions to make the most of the GPU resources. We show that the GPU utilization in single-and multi-GPU training can be commonly improve by applying ooo backprop and prioritizing critical operations. We propose three scheduling algorithms based on ooo backprop. For single-GPU training, we schedule with multi-stream ooo computation to mask the kernel launch overhead. In data-parallel training, we reorder the gradient computations to maximize the overlapping of computation and parameter communication; in pipeline-parallel training, we prioritize critical gradient computations to reduce the pipeline stalls. We evaluate our optimizations with twelve neural networks and five public datasets. Compared to the respective state of the art training systems, our algorithms improve the training throughput by 1.03-1.58× for single-GPU training, by 1.10-1.27× for data-parallel training, and by 1.41-1.99× for pipeline-parallel training. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | Association for Computing Machinery, Inc | - |
dc.title | Out-Of-Order BackProp: An Effective Scheduling Technique for Deep Learning | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Seo, Jiwon | - |
dc.identifier.doi | 10.1145/3492321.3519563 | - |
dc.identifier.scopusid | 2-s2.0-85128082743 | - |
dc.identifier.wosid | 000926506800027 | - |
dc.identifier.bibliographicCitation | EuroSys 2022 - Proceedings of the 17th European Conference on Computer Systems, pp.435 - 452 | - |
dc.relation.isPartOf | EuroSys 2022 - Proceedings of the 17th European Conference on Computer Systems | - |
dc.citation.title | EuroSys 2022 - Proceedings of the 17th European Conference on Computer Systems | - |
dc.citation.startPage | 435 | - |
dc.citation.endPage | 452 | - |
dc.type.rims | ART | - |
dc.type.docType | Proceedings Paper | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | N | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Hardware & Architecture | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Software Engineering | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Theory & Methods | - |
dc.subject.keywordPlus | Convolutional neural networks | - |
dc.subject.keywordPlus | Deep learning | - |
dc.subject.keywordPlus | Pipelines | - |
dc.subject.keywordPlus | Program processors | - |
dc.subject.keywordPlus | Scheduling algorithms | - |
dc.subject.keywordPlus | Scheduling | - |
dc.subject.keywordPlus | Critical operations | - |
dc.subject.keywordPlus | Data parallel | - |
dc.subject.keywordPlus | Deep learning system | - |
dc.subject.keywordPlus | Gradients computation | - |
dc.subject.keywordPlus | Large amounts | - |
dc.subject.keywordPlus | Neural networks trainings | - |
dc.subject.keywordPlus | Out of order | - |
dc.subject.keywordPlus | Parallel training | - |
dc.subject.keywordPlus | Performance | - |
dc.subject.keywordPlus | Scheduling techniques | - |
dc.subject.keywordAuthor | Deep learning systems | - |
dc.identifier.url | https://dl.acm.org/doi/10.1145/3492321.3519563 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.