System information-based Hadoop load balancing for heterogeneous clusters

Abstract

Hadoop is a distributed file system for processing big data. Users can modify and analyze data efficiently in the distributed process environment by using a MapReduce programming model, which is provided by Hadoop. However, its performance deteriorates when the Hadoop works on heterogeneous clusters consisting of servers of various performance levels. In this paper, we propose a load balancing technique using system information. The number of CPU cores and the available memory size are used for identifying the performance of each DataNode. To evaluate the proposed load balancing technique, we compare the original Hadoop with the modified Hadoop that uses the proposed algorithm in two different environments-one is an environment with sufficient memory, and the other is an environment with insufficient memory. © 2015 ACM.