In-Map/In-Reduce: Concurrent Job Execution in MapReduce

Hadoop based Map Reduce (MR) has emerged as big data processing mechanism in terms of its data intensive applications. In data intensive systems, analysis and visualizations as a result of various algorithms can lead to differentiable and comparable results. Current implementations of MR facilitates to reuse the results of MR jobs in other MR jobs and to distribute the cloud resources among jobs. However, very little work is done in terms of using same data for multiple algorithms at the same time in a single job using either shared resources or dynamic resource allocation based on the data and scheduling of Map Reduce jobs. In this paper we propose a method to execute multiple algorithms on same data in HDFS concurrently and to use the same available resources by dynamically managing the task assignment and results aggregation. Our proposed approach reduces the execution time and supports multiple algorithms execution in parallel. In-Map/In-Reduce shows 200% decrease in execution time.