Locality-sensitive hashing optimizations for fast malware clustering

Large datasets, including malware collections are difficult to cluster. Although we are mainly dealing with polynomial algorithms, the long running times make them difficult to use in practice. The main issue consists in the fact that the classical hierarchical algorithms need to compute the distance between each pair of items. This paper will show a faster approach for clustering large collections of malware samples using a technique called locality-sensitive hashing. This approach performs single-linkage clustering faster than the state of the art methods, while producing clusters of a similar quality. Although our proposed algorithm is still quadratic in theory, the coefficient for the quadratic term is several orders of magnitude smaller. Our experiments show that we can reduce this coefficient to under 0.02% and still produce clusters 99.9% similar with the ones produced by the single linkage algorithm.