K-sparse approximation for traffic histogram dimensionality reduction

Traffic histograms play a crucial role in various network management applications such as network traffic anomaly detection. However, traffic histogram-based analysis suffers from the curse of dimensionality. To tackle this problem, we propose a novel approach called K-sparse approximation. This approach can drastically reduce the dimensionality of a histogram, while keeping the approximation error low. K-sparse approximation reorders the traffic histogram and uses the top-K coefficients of the reordered histogram to approximate the original histogram. We find that after reordering the widely-used histograms of source port and destination port exhibit a power-law distribution, based on which we establish a relationship between K and the resulting approximation error. Using a set of traces collected from a European network and a regional network, we evaluate our K-sparse approximation and compare it with a well-known entropy-based approach. We find that the power-law property holds for different traces and time intervals. In addition, our results show that K-sparse approximation has a unique property that is lacking in the entropy-based approach. Specifically, K-sparse approximation explicitly exposes a tradeoff between compression level and approximation accuracy, enabling to easily select a desired settlement point between the two objectives.