Dynamic Job Scheduling in the Cloud Using Slowdown Optimization and Sandpile Cellular Automata Model

We present in this paper a general framework to study issues of effective load balancing and scheduling in highly parallel and distributed environments such as currently built Cloud computing systems. We propose a novel approach based on the concept of the Sandpile cellular automaton: a decentralized multi-agent system working in a critical state at the edge of chaos. Our goal is providing fairness between concurrent job submissions by minimizing slowdown of individual applications and dynamically rescheduling them to the best suited resources. The algorithm design is experimentally validated by a number of numerical experiments showing the effectiveness and scalability of the scheme in the presence of a large number of jobs and resources and its ability to react to dynamic changes in real time.