A Task Pipelining Framework for e-Science Workflow Management Systems

Workflow manager is a useful tool that brings the power of computational grid resources to the desktop, and allow them to conveniently put together and run their own scientific workflows. In existing workflow systems, individual tasks wait for input to be available perform computation, and produce output. Behind this, workflow manager automates the data movement from the data generating task to the data consumption task. This process is referred as file staging. Generally, stage-in, process, and stage-out are serially executed and staging is treated by traditional work- flow systems as a trivial step. However, as the data size is exponentially increasing and more and more scientific workflows require multiple processing steps to obtain the desired output, we argue that the data movement will possess high portion of overall running time and staging will become a challenging step of scientific workflow systems. In this paper, we propose a task pipelining framework for various e-Science workflow systems. Our system is a flexible and efficient tool to help the workflow systems to overlap the execution of adjacent tasks by enabling the pipelining of the intermediate data transfer between the interconnected tasks.