Detection of Worm Propagation Engines in the System Call Domain using Colored Petri Nets

While network worms carry various payloads and may utilize any available exploits, they all have one common component - the propagation engine. Moreover, it is important to note that the number of conceptually distinct propagation engines employed by existing network worms is quite limited. This paper presents a novel signature-based approach for detecting attacks perpetrated by network worms as a manifestation of a semantic functionality performed by one of the few known propagation engines. We propose a novel methodology to recognize any semantic functionality in the system call domain through utilizing colored Petri Nets. In this application, Petri Nets embody behavior-based signatures of the propagation engine functionalities. These signatures are indicative of the shell code activity in the first stage of the worm proliferation. We developed, tested and evaluated a propagation engine detector (PED) system that detects activity of the worm shell code executed by a process during an attack. Moreover, PED is able to recognize the type of propagation engine employed by the attacking worm.