Anomaly Detection Algorithms on IBM InfoSphere Streams: Anomaly Detection for Data in Motion

This paper presents and shares excerpts from our implementation of near real-time anomaly detection algorithms on the IBM InfoSphere Streams platform. The purpose of this article is to: 1) Describe how to design and implement known anomaly detection algorithms on IBM InfoSphere Streams. 2) Present some performance optimization capabilities of IBM InfoSphere Streams platform and propose a method to use them in anomaly detection applications. 3) Present some IBM InfoSphere Streams best practices and describe how their adoption in the context of anomaly detection application. The document describes the architecture and design of anomaly detection algorithms developed on IBM InfoSphere Streams. Although the solution was designed to be used for cyber security, the implemented algorithms are agnostic regarding the data type that they monitor and therefore can detect anomalies in data from various industries such as healthcare, finance and retail. The document describes the implementation of two anomaly detection algorithms: KOAD and PCA. The KOAD algorithm performs online anomaly detection with incremental learning and the PCA algorithm in performs offline anomaly detection. The solution was designed to provide near real-time insight into low latency on large data volume observation.