Design and implementation of dynamic I/O control scheme for large scale distributed file systemsopen access
- Authors
- Kim, Sunggon; Sim, Alex; Wu, Kesheng; Byna, Suren; Son, Yong Seok
- Issue Date
- Dec-2022
- Publisher
- SPRINGER
- Keywords
- High-performance computing; Distributed dynamic resource management; Autonomous control; Parallel and distributed file system; Cloud system
- Citation
- CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, v.25, no.6, pp 4423 - 4438
- Pages
- 16
- Journal Title
- CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS
- Volume
- 25
- Number
- 6
- Start Page
- 4423
- End Page
- 4438
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/60819
- DOI
- 10.1007/s10586-022-03640-0
- ISSN
- 1386-7857
1573-7543
- Abstract
- In this work, we have analyzed the input/output (I/O) activities of Cori, which is a high-performance computing system at the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory. Our analysis results indicate that most users do not adjust storage configurations but rather use the default settings. In addition, owing to the interference from many applications running simultaneously, the performance varies based on the system status. To configure file systems autonomously in complex environments, we developed DCA-IO, a dynamic distributed file system configuration adjustment algorithm that utilizes the system log information to adjust storage configurations automatically. Our scheme aims to improve the application performance and avoid interference from other applications without user intervention. Moreover, DCA-IO uses the existing system logs and does not require code modifications, an additional library, or user intervention. To demonstrate the effectiveness of DCA-IO, we performed experiments using I/O kernels of real applications in both an isolated small-sized Lustre environment and Cori. Our experimental results shows that our scheme can improve the performance of HPC applications by up to 263% with the default Lustre configuration.
- Files in This Item
-
- Appears in
Collections - College of Software > School of Computer Science and Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.