Dynamic slot-based task scheduling based on node workload in a MapReduce computation model

MapReduce is becoming a leading large-scale data processing model providing a logical framework for cloud computing. Hadoop, an open-source implementation of MapReduce framework, is widely used for realize such kind of parallel computing model. Nodes in the current Hadoop environment are normally homogeneous. Efficient resource management in clouds is crucial for improving the performance of MapReduce applications and the utilization of resources. However, the original scheduling scheme in Hadoop assign tasks to each node based on the fixed and static number of slots, without considering the physical workload on each node, such as the CPU utilization. This paper aims at proposing a dynamic slot-based task scheduling scheme by considering the physical workload on each node so as to prevent resource underutilization. The evaluation results show the proposed scheme can raise the overall computation efficiency among the heterogeneous nodes in cloud.