Energy-Aware Scheduling of MapReduce Jobs

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a large fraction of the data center´s overall costs. Therefore minimizing the energy consumption when executing MapReduce jobs is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA). We first model the problem of energy-aware scheduling of MapReduce jobs as an Integer Program. We then propose a greedy algorithm, called Energy-aware MapReduce Scheduling Algorithm (EMRSA), that finds the assignments of map and reduce tasks to the machine slots in order to minimize the energy consumed when executing the application. We perform experiments on a large Hadoop cluster to determine the energy consumption of several MapReduce benchmark applications, and then use this data in an extensive simulation study to characterize the performance of the proposed algorithm. The results show that EMRSA is able to find job schedules consuming 40% less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.