A Method of Multiple Encryption and Sectional Encryption Protocol Reverse Engineering

Research on unknown network protocol reverse engineering is of great significance in many network security applications. Currently most of methods are limited in analyzing plain-text protocols, and a few of method can partly analyze the encryption protocol which is powerless for multiple encryption protocol or sectional encryption protocol. This paper proposes a method of encrypted protocol reverse engineering based on dynamic taint analysis. The method uses Pin to record executed instructions, and then conducts off-line analysis of the data dependencies to build two taint propagation graphs on instruction and function level, then recover the decryption process. The decrypted plaintext can be located due to the decryption process feature. And then, the format of protocol can be parsed. Experiments show that the method can accurately locate the decrypted protocol data of the multiple encryption and sectional encryption protocol, and restore the original format.