MR-SPS: Scalable parallel scheduler for YARN/MapReduce platform

In the Big Data era, MapReduce has been the most utilized model in academia and industry. The main objective of MapReduce and its well-known implementation Hadoop is to run distributed applications to analyze huge amount of datasets, on very large clusters of commodity machines. This can be very time consuming. Also, the cluster that runs MapReduce applications has to be very scalable. In order to improve the performance of the MapReduce model, efficient scheduling algorithms have to be developed. Now day, Scalability is a major concern for applications that works on datasets measured by petabytes. Schedulers for such applications must allow a high scalability and an optimal utilization of resources in order to minimize the execution time. For this end we propose MR-SPS, a scalable parallel scheduling algorithm that takes care of scalability of the cluster and its performance by managing workload and data locality. Our scheduler works in a parallel manner, which allows a higher use of the capacities that the Resource-manager can eventually provide, this will increase the number of node that it can manage. For experiments, we have used CloudSim to simulate MR-SPS and other scheduling algorithms designed for MapReduce model. The results show the superiority of our implementation.