Detecting encrypted metamorphic viruses by hidden Markov Models

Virus writers make their viruses undetectable by using obfuscation methods, which ends in metamorphic viruses. We propose a method named detection circle which is based on the hidden Markov Model theory. We have used three elements to characterize a family of viruses: string occurrence probability, specifically-located character occurrence probability, and the amount of virus similarities. For the evaluation, we have created viruses and tested them by our method and four anti-virus software packages. The experimental results show that our detection rate was much higher in the first stage without obfuscation. Then we have encrypted the detected viruses and tested the proposed algorithm again. At this stage none of the four anti-viruses software packages detected viruses while our method found 70% of them.