Towards HPC I/O Performance Prediction through Large-scale Log Analysis
- Authors
- Kim, S.; Sim, A.; Wu, K.; Byna, S.; Son, Yong Seok; Eom, H.
- Issue Date
- 2020
- Publisher
- Association for Computing Machinery, Inc
- Keywords
- distributed file system; high performance computing; I/O performance prediction; log analysis
- Citation
- HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, pp 77 - 88
- Pages
- 12
- Journal Title
- HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing
- Start Page
- 77
- End Page
- 88
- URI
- https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/67256
- DOI
- 10.1145/3369583.3392678
- ISSN
- 0000-0000
- Abstract
- Large-scale high performance computing (HPC) systems typically consist of many thousands of CPUs and storage units, while used by hundreds to thousands of users at the same time. Applications from these large numbers of users have diverse characteristics, such as varying compute, communication, memory, and I/O intensiveness. A good understanding of the performance characteristics of each user application is important for job scheduling and resource provisioning. Among these performance characteristics, the I/O performance is difficult to predict because the I/O system software is complex, the I/O system is shared among all users, and the I/O operations also heavily rely on networking systems. To improve the prediction of the I/O performance on HPC systems, we propose to integrate information from a number of different system logs and develop a regression-based approach that dynamically selects the most relevant features from the most recent log entries, and automatically select the best regression algorithm for the prediction task. Evaluation results show that our proposed scheme can predict the I/O performance with up to 84% prediction accuracy in the case of the I/O-intensive applications using the logs from CORI supercomputer at NERSC. © 2020 ACM.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Software > School of Computer Science and Engineering > 1. Journal Articles
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.