Statistical estimation for Single Linkage Hierarchical Clustering

Clustering is a key component of most detectors of cyber-attacks, and increasingly, for both theoretical and practical reasons, methods that produce hierarchical clusterings (dendrograms) are being deployed in this context. In particular Single Linkage Hierarchical Clustering (SLHC) is attracting considerable interest. Existing clustering algorithms take no account of uncertainties in the data. In this paper, we derive a statistical model for the estimation of dendrograms, taking into account the uncertainty (through noise or corruption) in the distances among data points. We focus on just the estimation of the hierarchy of partitions afforded by the dendrogram, rather than the heights in the latter. The concept of estimating this "dendrogram structure" under SLHC is introduced, and an approximate maximum likelihood estimator (MLE) for the dendrogram structure is described. The proposed concept is demonstrated by a simple Monte Carlo simulation which demonstrates that the proposed MLE method performs better than SLHC in obtaining the correct structure.