Cyber situational awareness through operational streaming analysis

As the scope and scale of Internet traffic continue to increase the task of maintaining cyber situational awareness about this traffic becomes ever more difficult. There is strong need for real-time on-line algorithms that characterize high-speed / high-volume data to support relevant situational awareness. Recently, much work has been done to create and improve analysis algorithms that operate in a streaming fashion (minimal CPU and memory utilization) in order to calculate important summary statistics (moments) of this network data for the purpose of characterization. While the research literature contains improvements to streaming algorithms in terms of efficiency and accuracy (i.e. approximation with error bounds), the literature lacks research results that demonstrate streaming algorithms in operational situations. The focus of our work is the development of a live network situational awareness system that relies upon streaming algorithms for the determination of important stream characterizations and also for the detection of anomalous behavior. We present our system and discuss its applicability to situational awareness of high-speed networks. We present refinements and enhancements that we have made to a well-known streaming algorithm and improve its performance as applied within our system. We also present performance and detection results of the system when it is applied to a live high-speed mid-scale enterprise network.