Towards Access Pattern Prediction for Big Data Applications

Abstract

The importance of data is becoming more and more prominent as modern applications produce a large amount of data. It is becoming common for applications to produce and process gigabytes or even terabytes of data. To improve the performance of data-intensive applications, the underlying storage systems utilize the I/O characteristics of applications such as access patterns to improve the storage performance. For example, existing storage schemes store frequently accessed data in high performance storage devices such as NVMe SSDs for low latency and stores rarely access data in low performance but high capacity storage devices such as tape storage for cost-efficiency. Thus, as the importance of data arises, it is important to understand the I/O characteristics of applications. In this paper, we propose an access pattern prediction scheme to understand the I/O characteristics of applications and utilize the characteristics for fast I/O processing. Our scheme uses the application history and machine learning algorithm to accurately predict the pattern. To do this, we first utilize a system log to collect access pattern data of applications. Then, by using the logs, we set up a machine learning based prediction model using the long short-term memory (LSTM) algorithm. Finally, when the application is executed repeatedly, we use the prediction model to predict the I/O requests of the application which can be used to improve the storage performance. Evaluation result using a real big data application shows that the proposed scheme can accurately predict the access pattern.