Resource aware scheduler for heterogeneous workload based on estimated task processing time

Hadoop is a widely adopted open source implementation of the Map Reduce framework for large scale distributed data processing. Effective scheduling is critical for the efficiency of such platforms. The Motivation of the work is to explore a possibly simple and effective design that addresses the heterogeneity within a given Hadoop cluster using a resource-aware approach. Here the concept is to use a scheduler that accounts for resource utilization in a systematic manner while making scheduling decisions. The Resource Scheduler which generically adopts the Shortest Job First scheduling policy will make scheduling decisions based on the estimated map task processing time of jobs in queue. This scheduler accounts resource awareness by estimating the task processing time solely based on the resource demand of job and each computing nodes ability to supply. The resources that concerned includes CPU, disk I/O and network I/O. Through running I/O intensive and CPU intensive jobs on a Hadoop cluster, accuracy of our estimations where assessed and benchmarked the performance of resource scheduler against that of Hadoop´s default FIFO scheduler.