Resource Provisioning for Staging Components

To deal with the inordinate output data volumes of current and future high end simulations, researchers are turning to online methods in which multiple software components that implement desired data analytics and visualization are run on ´staging resources´ of the petascale machine, concurrently and coupled with the simulations producing these outputs. Efficient online execution of data analytics ´in the output stream´, however, requires careful provisioning of staging resources, to obtain delays for analytics processing that prevent applications from blocking on stalled output, while also bounding total required staging resources. This paper addresses the ´staging provisioning´ problem, assuming sets of components arranged as potentially multiple analytics/output pipelines that differ in runtime behavior and resource requirements. For such configurations, it then meets the throughput constraint of online analytics while also minimizing end-to-end pipeline latency, all based on runtime observations and predictions of component performance. Experimental evaluations demonstrate the algorithm´s utility. Its complexity for minimizing latency without violating throughput constraints is O(M), where M is the number of components in the staging area.