A Bayesian runtime load manager on a shared cluster

The efficient execution of irregular data parallel applications, on dynamically shared computing clusters, requires novel approaches to manage the runtime load distribution. Such environments have an unpredictable dynamic behaviour both due to the application requirements and to the available system´s resources. This uncertainty was the main motivation to propose and evaluate an application level scheduler where decisions are efficiently taken with improved accurate predictions on the environment´s current and near future state, based on available incomplete and aged measured data. Bayesian decision networks are used as the scheduler´s decision making mechanism, its effectiveness to manage the load distribution of a parallel ray tracer is assessed and compared with alternative strategies. The evaluation results, with complex scenes on a 7 shared nodes cluster with dynamically, variable workloads, show considerable performance improvements over blind strategies, and stress the benefits over a sensor based deterministic approach of identical complexity