A data-aware workflow scheduling algorithm for heterogeneous distributed systems

The workflow scheduling problem in heterogeneous distributed systems is hard to solve due to both intermediate data transfer time and the computation time for each task being considered. The heterogeneity of the computing power of distributed computational sites and the band width between them makes the scheduling problem challenging. In this study, we improve a heuristic-based data-aware algorithm to find the optimal scheduling so that the turnaround time of the workflow is minimized. Our improved algorithm outperforms the existing algorithms in both performance and time efficiency in most cases. We also extend our algorithm to solve the co-scheduling problem. In this problem, each task of the workflow can request data from a remote data site before its execution; and also store important intermediate data to a remote data site after the execution. The results show that the turnaround time of the workflow can be shortened significantly using our data-aware algorithm compared to the existing optimal algorithms.