Real-time anomaly detection over VMware performance data using storm

Anomaly detection is the identification of items or observations which deviate from an expected pattern in a dataset. This paper proposes a novel real time anomaly detection framework for dynamic resource scheduling of a VMware-based cloud data center. The framework monitors VMware performance stream data (e.g. CPU load, memory usage, etc.). Hence, the framework continuously needs to collect data and make decision without any delay. We have used Apache Storm, distributed framework for handling performance stream data and making prediction without any delay. Storm is chosen over a traditional distributed framework (e.g., Hadoop and MapReduce, Mahout) that is good for batch processing. An incremental clustering algorithm to model benign characteristics is incorporated in our storm-based framework. During continuous incoming test stream, if the model finds data deviated from its benign behavior, it considers that as an anomaly. We have shown effectiveness of our framework by providing real-time complex analytic functionality over stream data.