Statistical Traffic Anomaly Detection in Time-Varying Communication Networks

We propose two methods for traffic anomaly detection in communication networks where properties of normal traffic evolve dynamically. We formulate the anomaly detection problem as a binary composite hypothesis testing problem and develop a model-free and a model-based method, leveraging techniques from the theory of large deviations. Both methods first extract a family of probability laws (PLs) that represent normal traffic patterns during different time-periods, and then detect anomalies by assessing deviations of traffic from these laws. We establish the asymptotic Newman-Pearson optimality of both methods and develop an optimization-based approach for selecting the family of PLs from past traffic data. We validate our methods on networks with two representative time-varying traffic patterns and one common anomaly related to data exfiltration. Simulation results show that our methods perform better than their vanilla counterparts, which assume that normal traffic is stationary.