PCA-subspace method — Is it good enough for network-wide anomaly detection

PCA-subspace method has been proposed for network-wide anomaly detection. Normal subspace contamination is still a great challenge for PCA although some methods are proposed to reduce the contamination. In this paper, we apply PCA-subspace method to six-month Origin-Destination (OD) flow data from the Abilene. The result shows that normal subspace contamination is mainly caused by anomalies from a few strongest OD flows, and seems unavoidable for subspace method. Further comparison of anomalies detected by subspace method and manually tagged anomalies from each OD flows, we find that anomalies detected by subspace method are mainly caused by anomalies from medium and a few large OD flows, and most anomalies of minor OD flows are buried in abnormal subspace and hard to be detected by PCA-subspace method. We analyze the reason for those anomalies undetected by subspace method and suggest to use normal subspace to detect anomalies caused by a few strongest OD flows, and to further divide abnormal subspace to detect more anomalies from minor OD flows. The goal of this paper is to address limitations neglected by prior works and further improve the subspace method on one hand, also call for novel detection methods for network-wide traffic on another hand.