Millipedes: Distributed and Set-Based Sub-Task Scheduler of Computing Engines Running on Yarn Cluster

Hadoop YARN is evolving to become the de-facto standard that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform. And, there are lots of researches about efficiently managing cluster resources and scheduling parallel jobs over YARN clusters. However, the scheduling of sub-tasks derived from one application doesn´t receive enough attention, which still managed by independent data processing engines on YARN. In this paper, we analyze the limitation of sub-task scheduling algorithms of popular data processing engines running on YARN, including MapReduce 2.0, Spark Then, we propose Millipedes: a distributed & set-based sub-task scheduling framework. In Millipedes, YARN´s application master dispatches a set of sub tasks to data nodes, and then the local task scheduler of each data node schedules the execution of these sub tasks according to real-time dynamic resource usage.