EHadoop: Network I/O Aware Scheduler for Elastic MapReduce Cluster

Over the last few years the usage of cloud computing dramatically increased. Many data analytics platforms run on the cloud. Such systems characterized by large data transfer among VMs. The network isolation between cloud users in modern data centers is not as good as CPU and memory isolation [20]. The weak isolation leads to unpredictable performance of the inter data center network. Moreover, with the raise of popularity of cloud computing the competition between providers get tougher, which leads to prices decrease. Some users decide to perform data-analytics in a cross-cloud fashion [11], which requires data transfer over WAN. It is known that WAN provides lower than LAN performance. We show that saturated network can greatly impact MapReduce job´s task completion time. It results in higher costs for the user, because according to the pay-as-you-go model the user pays for the time resources being used. In this work we present EHadoop network I/O aware scheduler for elastic MapReduce cluster which performs online job profiling and schedules tasks based on available network bandwidth. The evaluation results show that EHadoop allows to avoid network contention and does not increase MapReduce task completion time with network bandwidth degradation.