A Hybrid Cloud Framework for Scientific Computing

Cloud services are transforming many computing tasks, but the unique requirements of scientific computing have caused it to lag behind in cloud adoption because of the performance variation of cloud resources. Based on our experience with the Organic Grid, we propose a framework for a hybrid cloud that will intelligently distribute work to appropriate computing resources to mitigate the impact of performance variation. We describe a cloud framework that integrates with specialized hardware and distributes work intelligently among heterogeneous computing resources. Our approach is to organize a set of computing nodes in an overlay network, to allow each node as an individual agent to position itself within the network to maximize its productivity. An application finds the resources and decides which task to run on which cloud nodes. Our simulations demonstrate that our methods can significantly reduce communication burdens of the most overworked nodes, especially on networks with the highest task to node ratios.