Scalable network-based buffer overflow attack detection

Buffer overflow attack is the main attack method that most if not all existing malicious worms use to propagate themselves from machine to machine. Although a great deal of research has been invested in defense mechanisms against buffer overflow attack, most of them require modifications to the network applications and/or the platforms that host them. Being an extension work of CTCP, this paper presents a network-based low performance overhead buffer overflow attack detection system called Nebula, which can detect both known and zero-day buffer overflow attacks based solely on the packets observed without requiring any modifications to the end hosts. Moreover, instead of deriving a specific signature for each individual buffer overflow attack instance, Nebula uses a generalized signature that can capture all known variants of buffer overflow attacks while reducing the number of false positives to a negligible level. In addition, Nebula is built on a centralized TCP/IP architecture that effectively defeats all existing NIDS evasion techniques. Finally, Nebula incorporates a payload type identification mechanism that reduces further the false positive rate and scales the proposed buffer overflow attack detection scheme to gigabit network links.