DETS: A Dynamic and Elastic Task Scheduler Supporting Multiple Parallel Schemes

Task scheduling plays an important role in task parallel computing platform. In this paper, we present DETS, a dynamic and elastic task scheduler that can support multiple parallel schemes. DETS works in a master-worker manner and schedules tasks dynamically. In order to execute various types of applications elastically, it uses task pool from which workers pull tasks to execute. Workers are supervised to form a logical group which can scale up/down during runtime with available nodes and processors. DETS supports several types of parallel computing schemes, including embarrassingly parallel, MapReduce, Tree-based searching, DAG-based processing, etc. Exemplars are conducted and the results show DETS is efficient and practical.