On Threshold Selection for Principal Component Based Network Anomaly Detection

Principal component based anomaly detection has emerged as an important statistical tool for network anomaly detection. It works by projecting summary network information onto a signal and noise sub-spaces and detecting anomalies in the noise sub-space. Recently some major problems where detected with this network anomaly approach. The chief among the problems is the difficulty in selecting a threshold used to declare that the energy in the noise sub-space contains a network anomaly. We show that the reason for this problem is that some of the assumption previously used to select the threshold, namely that the traffic follows a Normal distribution, do not fit the reality of the available network traces. Then, we show that the energy in the noise sub-space can be modeled with the long-tailed Cauchy distribution and use this approximation to calculate reliable thresholds. Our analysis of network traces indicates that the Cauchy distribution approximation of the energy distribution should significantly lower the false alarm rate.