Evaluating Workflow Tools with SDAG

Workflow management systems (WMS) are typically comprised of or make use of multiple independent software components. The design and development of those components is typically drawn from functional requirements of scientific applications that utilize the corresponding WMS. Consequently, the WMS design reflects those core functional requirements in applications that it supports in the future. Whereas most design criteria are engineered to be as generic as possible, some design trade-offs may prove sub-optimal for certain new workflow applications. We argue that WMS design tradeoffs that emerge from a limited set of real-world applications can be minimized by the use of larger, more varied synthetic application datasets. We present SDAG, a tool for generating synthetic well formed workflows (WFWs) that span a varied space of synthetic WFWs around any reference workflow. These synthetic WFWs enable developers to test and evaluate WMS or their constituent software components on a broad range of workflows and enable more generic design criteria for WMS.