Using SchedFlow for performance evaluation of workflow applications

Computational science increasingly relies on the execution of workflows in distributed networks to solve complex applications. However, the heterogeneity of resources in these environments complicates resource management and the scheduling of such applications. Sophisticated scheduling policies are being developed for workflows, but they have had little impact in practice because their integration into existing workflow engines is complex and time consuming as each policy has to be individually ported to a particular workflow engine. In addition, choosing a particular scheduling policy is difficult, as factors like machine availability, workload, and communication volume between tasks are difficult to predict. In this paper, we describe SchedFlow, a tool that integrates scheduling policies into workflow engines such as Taverna, DAGMan or Karajan. We show how SchedFlow was used to take advantage of different scheduling policies at different times, depending on the dynamic workload of the workflows. Our experiments included two real workflow applications and four different scheduling policies. We show that no single scheduling policy is the best for all scenarios, so tools like SchedFlow can improve performance by providing flexibility when scheduling workflows.