Resource Management for Processing Wide Area Data Streams on Supercomputers
- Authors
- Chung; J.; Adhikari; M.; Srirama; S.N.; Jung, Eun Sung; E.-S.; Kettimuthu; R.
- Issue Date
- 2020
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Keywords
- data streaming; online resource provisioning; wide area networks
- Citation
- Proceedings - International Conference on Computer Communications and Networks, ICCCN, v.2020-August
- Journal Title
- Proceedings - International Conference on Computer Communications and Networks, ICCCN
- Volume
- 2020-August
- URI
- https://scholarworks.bwise.kr/hongik/handle/2020.sw.hongik/12514
- DOI
- 10.1109/ICCCN49398.2020.9209669
- ISSN
- 1095-2055
- Abstract
- Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis. © 2020 IEEE.
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - Graduate School > Software and Communications Engineering > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/hongik/handle/2020.sw.hongik/12514)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.