Data Velocity Scaling via Dynamic Monitoring Frequency on Ultrascale Infrastructures

Monitoring ultrascale systems such as Clouds requires collecting enormous amount of data by periodically reading metric values from a system. Current approaches tend to select a static frequency for sampling monitoring data. On one hand, over-sampling the data by collecting it at high frequencies results in data redundancy during steady runs of the system. On the other hand, under-sampling with low monitoring frequencies results in information loss during volatile behaviour of the system as data is significantly diluted. Therefore, choosing an optimal monitoring frequency represents a challenging research issue. In this paper, we propose a dynamic monitoring frequency algorithm for collecting monitoring data from ultrascale systems such as Clouds. The algorithm deterministically reduces data velocity by self-adapting the monitoring frequency to the volatility of data being collected. Consequently, it collects less data due to fewer readings, while keeping the same data value as the equivalent static monitoring frequency. The proposed approach is evaluated using Google traces where it is able to reduce the velocity of monitoring data by up to 85% without diluting information quality.