A two-phase execution engine of reduce tasks in Hadoop MapReduce

In Hadoop MapReduce, reduce tasks issue massive remote I/O operations to copy the intermediate results of map tasks. The operations cause massive remote data access delays which degrade the system performance. To Handle this problem, we propose an execution engine of reduce tasks. The engine partitions the execution of reduce tasks into two phases. In the first phase, the engine selects the nodes to run reduce tasks, and then order the nodes to prefetch intermediate results for the reduce tasks. In the second phase, the selected nodes allocate computing and memory resource to the reduce tasks, and execute these tasks. Due to the fact that intermediate results have been prefetched, reduce tasks can access these results from local nodes, and the remote access delay of the results can be hidden. We have implemented the engine in Hadoop-0.20.2. We evaluated the engine in a Linux cluster. The results showed that the engine optimized the performance of Hadoop in most cases.