Towards A Better Understanding of Workload Dynamics on Data-Intensive Clusters and Grids

This paper presents a comprehensive statistical analysis of workloads collected on data-intensive clusters and grids. The analysis is conducted at different levels, including virtual organization (VO) and user behavior. The aggregation procedure and scaling analysis are applied to job arrival processes, leading to the identification of several basic patterns, namely, pseudo-periodicity, long range dependence (LRD), and (multi)fractals. It is shown that statistical measures based on interarrivals are of limited usefulness and count based measures should be trusted instead when it comes to correlations. We also study workload characteristics like job run time, memory consumption, and cross correlations between these characteristics. A "bag-of-tasks" behavior is empirically proved, strongly indicating temporal locality. We argue that pseudo-periodicity, LRD, and "bag-of-tasks" behavior are important workload properties on data-intensive clusters and grids, which are not present in traditional parallel workloads. This study has important implications on workload modeling and performance predictions in data-intensive grid environments.