Exploiting causality and communication patterns in network data analysis

Detecting the root-cause of failures in modern, complex networks is tedious. Understanding the problem fully requires good instrumentation and thorough understanding of the information flows in the network. In this paper, we describe two techniques for understanding the information flows and pinpointing the problems in them: causal relationship extraction and communication pattern detection. We instrumented a network with probes. The probes collect all the data from the network into a ringbuffer and index it, making it possible to either quickly retrieve flows and packets associated to them for further analysis. Our software then extracts and visualizes causal relationships between the events. The causal relationship extraction and communication pattern detection proved to be an effective method for pinpointing the cause of network system failures, understanding security risks and managing complexity. Our research prototype demonstrates a method for making problem solving faster and more systematic. The methods can also be used to detect emerging problems proactively.