Fingerprinting custom botnet protocol stacks

This paper explores the use of TCP fingerprints for identifying and blocking spammers. Evidence has shown that some bots use custom protocol stacks for tasks such as sending spam. If a receiver could effectively identify the bot TCP fingerprint, connection requests from spam bots could be dropped immediately, thus reducing the amount of spam received and processed by a mail server. Starting from a list of known spammers flagged by a commercial reputation list, we fingerprinted each spammer and found the roughly 90% have only a single known fingerprint typically associated with well known operating system stacks. For the spammers with multiple fingerprints, a particular combination of native/custom protocol stack fingerprints becomes very prominent. This allows us to extract the fingerprint of the custom stack and then use it to detect more bots that were not flagged by the commercial service. We applied our methodology to a trace captured at our regional ISP, and clearly detected bots belonging to the Srizbi botnet.