Scheduling Framework and Algorithms for Large-Scale Divisible Load Processing with Multi-Dimensional QoS Constraints

In this paper, we propose a scheduling framework and related algorithms for processing large-scale, computation-intensive divisible loads. The framework is organized into a two-level tree architecture. Based on this framework, admission test and load partitioning and distribution algorithms are designed to ensure that the multi-dimensional QoS requirements, i.e. processing deadline, security and reliability, of admitted loads can be satisfied. We take a novel approach to incorporate resource reservation and time step-size adaptive scheduling schemes into the optimal solution that makes computation nodes finish computing at the same time instant. We provide an implementation of the framework a top of a distributed communication middleware extended with QoS-aware resource management facilities. Prototype implementation and preliminary experimental results demonstrate the engineering feasibility and good performance of the proposed framework and algorithms.