Hadoop Job Scheduling with Dynamic Task Splitting

Fairness and data locality are often in conflict in Hadoop job scheduling. During scheduling, it is not always possible for data locality to be achieved for all jobs or for fairness to be attained for all users. Achieving pure fairness may compromise the data locality of the jobs which will negatively affect performances, and vice-versa. For example, a scheduler may opt to sacrifice performance by scheduling tasks to non-data local nodes. Alternatively, a scheduler may choose to sacrifice fairness by giving up an available slot and wait for a data-local node. The Dynamic Task Splitting Scheduler (DTSS) is proposed to mitigate the tradeoffs between fairness and data locality during job scheduling. DTSS does so by dynamically splitting a task and executing the split task immediately, on a non-data-local node, to improve the fairness. Analysis and experiments results show that it is possible to improve both fairness and the performance by adjusting the proportion of the task split. DTSS is shown to improve the make span of different users in a cluster by 2% to 11% as compared to delay scheduling under the situation where it is difficult to obtain data-local nodes on a cluster. Lastly, experiments show that DTSS is not a suitable scheduler under conditions where jobs are able to obtain data-local nodes easily.