Fine-Grained Workflow in Heterogeneous Environments

This work details the design and analysis of a fully decentralised scientific workflow management system for the execution of real applications in heterogeneous and unpredictable computing environments. Our model is based on a novel approach to the overlay metacomputer concept, founded on a graph-oriented peer-to-peer task distribution framework. This platform provides a basis for the development of a distributed virtual machine that can be dynamically deployed over a heterogeneous resource collection to provide a useful layer of abstraction, harnessing remote independent resources and making them appear dedicated to the execution of a particular collection of tasks. This paper demonstrates the utility of our model using synthetic and real workflow applications executed in a real and heterogeneous environment, identifying the potential benefit of fine-grained workflow expressions as a method for achieving predictable task execution behaviour in unpredictable environments.