Dynamic Token Based Improving MapReduce Performance in Cloud Computing

In recent years, Hadoop, the open-source implementation of Google´s MapReduce, is widely used and has become the de facto standard of big data processing. A typical running environment of Hadoop is cloud computing in which resource heterogeneity is very common due to varied factors including the different hardware of nodes and the different workload on the nodes and etc. The slot-based scheduling in Hadoop causes the inefficient utilization of computing resources in cloud computing environment, which lead to the performance degradation. To solve the problem mentioned above, we propose a dynamic token based method which dynamically controls the number of tasks running on each node according to the available computing resources on a node and the resource requirement of a task. The results of evaluations show that the completion times of single jobs with the proposed method are approaching to the static optimum in the dedicated environment and better than the static optimums in the other two competitive environments. Moreover, the proposed method significantly improves the throughput of mixed workloads in all computing environments and performance in real cloud computing environment have been improved by 45.9% on average.