Measuring similarity of malware behavior

Malicious software (malware) represents a major threat for computer systems of almost all types. In the past few years the number of prevalent malware samples has increased dramatically due to the fact that malware authors started to deploy morphing (aka obfuscation) techniques in order to hinder detection of such polymorphic malware by anti-malware products. Using these techniques numerous variants of a malware can be generated. All these variants have a different syntactic representation while providing almost the same functionality and showing similar behavior. In order to effectively detect polymorphic malware it is advantageous (if not required) to know which malware samples are variants of a particular malware. Respective approaches for determining this relation between malware samples automatically are currently investigated by a number of researchers. A prerequisite for assessing this relation based on particular features of malware samples is an appropriate similarity or distance measure. In particular a number of approaches for clustering malware samples have been recently published. Thereby different similarity measures are used but without thoroughly discussing their choice. So it is an unanswered question which similarity measures are appropriate for determining respective relations between malware samples. To answer this question we study different distance measures in detail and discuss desirable properties of a distance measure for this particular purpose. We focus on behavioral features of malware and compare and experimentally evaluate different distance measures for malware behavior. Based on our results we identify a most appropriate distance measure for grouping malware samples based on similar behavior.