Handling Failures in Parallel Scientific Workflows Using Clouds

Failures are common in High Performance Computing (HPC) environments and can significantly impact the performance of scientific workflows executing on top of these large scale computing environments. Computing clouds are being used as promising HPC environments. Although clouds offer several advantages such as elasticity and availability, failures are very frequent in this type of environment, where virtualization, instabilities and providers´ actions directly impact on workflow execution. In this way, activity failures are almost inevitable in clouds where virtual machine failures are a reality rather than a possibility. In this paper we present a set of failure handling heuristics based on cloud characteristics, which are implemented within SciMultaneous, a service-oriented architecture that manages re-executions of failed scientific workflow activities using runtime provenance data. Experimental results on clouds showed that SciMultaneous and its heuristics considerably increase workflow completion and reduce the total execution time (TET) of the workflow (even considering executions or reexecutions) up to 45%, when compared to a posteriori reexecution approaches. We analyze SciMultaneous´ behavior under a series of activity failures types and concluded that even a single activity failure could have a large detrimental effect on scientific workflow TET.