Using instruction sequence abstraction for shellcode detection and attribution

Although several research teams have focused on binary code injection, it is still an unsolved problem. Misuse-based detection lacks the flexibility to tackle unseen malicious code samples and anomaly-based detection on byte patterns is highly vulnerable to byte cramming and blending attacks. In addition, it is desperately needed to correlate newly-detected code injection instances with known samples for better understanding the attack events and tactically mitigating future threats. In this paper, we propose a technique for modeling shellcode detection and attribution through a novel feature extraction method, called instruction sequence abstraction, that extracts coarse-grained features from an instruction sequence. Our technique facilitates a Markov-chain-based model for shellcode detection and support vector machines for encoded shellcode attribution. We also describe our experimental results on shellcode samples to demonstrate the effectiveness of our approach.