An application-level priority scheduling for many-task computing in multi-user heterogeneous environment

Many-task computing (MTC) is a widely used computing paradigm for complex scientific applications, which is utilized in large distributed system. One of the goals of MTC is to complete large quantities of relatively small tasks within a short timeframe, which lead to huge management overhead. As tasks are competing for limited heterogeneous resources, the resource competition would be prominent. Moreover, in multi-user environment, tasks from different users will have different time constraint, which leads to priority concern. Tasks should be finished before deadline to avoid great loss from user perspective. Traditional scheduling heuristics, which are originally designed for High Throughput Computing (HTC) cannot achieve satisfactory performance in MTC scenario. In this paper, we propose an application-level priority scheduling algorithm. On one hand, we utilize the knowledge of applications to assign different jobs to heterogeneous resources, so as to minimize the Flowtime of the completed jobs. On the other hand, we dynamically adjust the priorities of jobs when scheduling to further improve system performance. Experimental results show that the application-level priority scheduling approach, when compared with other scheduling algorithms, not only attains better performance in both Makespan and Flowtime, but also achieve better turnaround time of jobs in multi-user MTC environment.