An Efficient Dynamic Detection Method for Various x86 Shellcodes

Network-based dynamic shellcode detection, in which network traffic is examined by being executed on an emulator for detecting essential behavior of shellcode, has been studied intensively in recent years. The main issues of dynamic shellcode detection are (1) the computational cost is high and (2) it can detect only shellcodes whose behaviors match predefined detection rules. In this paper, we propose a novel dynamic shellcode detection method which is much faster and detects more variety of x86 shellcodes than existing methods. Our method utilizes a combination of static detection and emulation-based dynamic detection. Namely, it first performs a static binary string search over the to-be-examined traffic for particular x86 instructions to spot candidates of shellcodes. Then, it performs the dynamic detection on the candidates. Moreover, we add a new detection rule for our dynamic detection, which allows us to detect shellcodes for Windows systems or Linux systems. An evaluation with honeypot traffic shows an impressive improvement of the proposed method in terms of computational cost. Also, an evaluation using a penetration testing tool shows that the proposed method can detect more variety of shellcodes than the best existing method.