An Evaluation of the Benefits of Fine-Grained Value-Based Scheduling on General Purpose Clusters

General purpose compute clusters fulfill a prominent role in a wide range of organizations to deliver the necessary computational power for their processes. In order to manage the shared use of such clusters, scheduling policies are installed to determine if and when the jobs submitted to the cluster are executed. Value-based scheduling policies differ from other policies in that they allow users to communicate the value of their computation to the scheduling mechanism. The design of market mechanisms whereby users are able to bid for resources in a fine-grained manner has proven to be an attractive means to implement such policies. In the clearing phase of the mechanism, supply and demand for resources are matched in pursuit of a value-maximizing job schedule and resource prices are dynamically adjusted to the level of excess demand in the system. Despite their success in simulations and research literature, such fine-grained value-based scheduling policies have been rarely used in practice as they are often considered too fragile, too onerous for end-users to work with, and difficult to implement. A coarse-grained form of value-based scheduling that mitigates aformentioned disadvantages involves the installation of a priority queueing system with fixed costs per queue. At present, it is however unclear to which extent a fine-grained form of value-based scheduling through auctions can outperform such a priority queueing system. Using workload traces of a general purpose cluster, we indicate under which conditions this is the case and quantify the resulting efficiency gains.