Toward Real Time Data Analysis for Smart Grids

This paper describes the architecture and design of a novel system for supporting large-scale real-time data analysis for future power grid systems. The widespread deployment of renewable generation, smart grid controls, energy storage, plug-in hybrids, and new conducting materials will require fundamental changes in the operational concepts and principal components of the grid. As a result, the whole system becomes highly dynamic and requires constant adjusting based on real time data. Even though millions of sensors such as phase measurement units (PMU) and smart meters are being widely deployed, a data layer that can analyze this amount of data in real time is needed. Unlike the data fabric in other cloud services, the data layer for smart grids has some unique design requirements. First, this layer must provide real time guarantees. Second, this layer must be scalable to allow a large number of applications to access the data from millions of sensors in real time. Third, reliability is critical and this layer must be able to continue to provide service in face of failures. Fourth, this layer must be secure. We address these challenges though a scalable system architecture that integrates the I/O and data processing capability in a devise set of devices. Data process operations can be placed anywhere from sensors, data storage devices, to control centers. We further employ compression to improve performance. We design a lightweight compression customized for power grid data. Our system can reduce end-to-end response time by reduce I/O overhead through compression and overlap compression operations with I/O. The initial prototype of our system was demonstrated with several use cases from PNNL´s FPGI and show that our system can provide real time guarantees to a diverse set of applications.