A Parameter Dynamic-Tuning Scheduling Algorithm Based on History in Heterogeneous Environments

In MapReduce model, the job execution time was prolonged by the straggler tasks in heterogeneity environments. The LATE scheduler has introduced the longest remaining time strategy, but it also has some drawbacks such as inaccurate estimated time and the wasting of system resources. In order to solve these problems, we propose two main algorithms : The parameter dynamic-tuning algorithm based history estimates progress of a task accurately since it dynamically tunes the weight of each phase of a map task and a reduce task according to the historical values of the weights, The evaluation-scheduling algorithm reduce the wasting of system resources by evaluating the free slot before launching a straggler task on this node. The two main algorithms are implemented in hadoop 0.20.1. The environment results are satisfaction to our expects and significantly reduce the wasting of system resources.