Divergence Detector: A Fine-Grained Approach to Detecting VM-Awareness Malware

Virtualized execution has become an effective mechanism to analyze malware in a dynamic way. To conceal its malicious behaviors, VM-aware malware probes the execution environment for analysis-resistance. These malware programs hide their malicious behaviors if they are launched in a virtual machine (VM). VM awareness becomes a barrier for malware analysis due to the concealment of malicious behaviors. In this paper, we discover that uncertain factors have significant influence on the effectiveness of malware detection. To cope with the problems, a new VM-aware detection scheme, namely Divergence Detector, is proposed to address the swindle of the evolved malware. Unlike conventional schemes, the Divergence Detector reduces the uncertain factors at instruction level, and can detect the divergence of multiexecution traces across heterogeneous virtual machines. The proposed Divergence Detector is implemented across the three commonly used VM platforms, that is, QEMU, Bochs and Xen. It compares the code coverage of the execution traces on various VM platforms to discover the deviation of behavior, thereby precisely detecting the VM-awareness. We will formally predict the effectiveness of Divergence Detector by constructing a mathematic model, which shows the maximum false positive rate is exponentially decreased with respect to the number of multi-executions. Representative samples utilizing seven types of commonly used VM-aware techniques were also employed for evaluation. The evaluation results indicate that the maximum false positive rate complies with our prediction. The uncertain factors play the major role in the VM-awareness detection. To reduce uncertain factors causing false positives, a method is proposed for VM-aware detection. The Divergence Detector can also enable the identification of new types of malware since the benign programs do not need to be aware of execution environment.