Machine learning and windowed subsecond event detection on PMU data via Hadoop and the openPDC

The high rate of data samples reported by devices that support PMU functionality forces the use of non-traditional methods in order to attempt realtime anomaly detection. Two methods discussed are offline machine learning and a realtime sliding window procedure. In using machine learning techniques it is possible to assert a classifier algorithm, which to a certain degree of accuracy can flag incoming data for further operation when applied in realtime. The open source project Hadoop provides the storage architecture for large datasets (petabyte scale) as well as the MapReduce computational framework for distributed computing to produce these classifiers. Additionally, a sliding window of realtime data can be used to present a longer data sample window than the device report rate allowing for a heuristic hysteresis approach. The open source openPDC promotes the implementation of the classifier and sliding window in a realtime environment operating on new measurements thirty times a second.