Generic unpacking techniques

Traditional signature-based malware detection techniques rely on byte sequences, called signatures, in executable for signature-matching. Modern malware authors can bypass signature-based scanning by employing the recently emerged technology of code obfuscation for information hiding. Obfuscation alters the byte sequence of the code without effectively changing the execution behavior. A commonly used obfuscation technique is packing. Packing compresses and/or encrypts the program code. Actual code stays hidden till runtime (when the executable is unpacked) making it immune to static analysis. Since every packer has its associated unpacker to undo packing, a successful generic unpacker is difficult to come by. A few automated unpacking techniques have been published so far that attempt to unpack packed binaries without any specific knowledge of the packing technique used. In this paper, we aim to provide a comprehensive summary of the currently published prevalent generic unpacking techniques and weigh their effectiveness at dealing with the spreading nuisance of packed malware. Dynamic analysis is a promising solution to the packing problem as every packed binary has to inevitably unpack itself for execution. Emulation (running code in a virtual environment) is an effective and powerful technique for generic unpacking. We will be reviewing various unpacking techniques based on emulation and a few other hybrid and alternative approaches.