Optimization of an Instrumentation Tool for Stripped Win32/X86 Binaries

Many software security, instruction set architecture virtualization and performance enhancement techniques require instrumentation of application program binaries either to add run-time checks or to perform dynamic analysis and transformation. Unfortunately, commercially distributed application binaries on the Win32 platform are often stripped of their symbol table, and therefore cannot be easily disassembled, let alone correctly instrumented. BIRD is an instrumentation tool that applies an IA-32 disassembler both statically and dynamically, and successfully guarantees that no instruction in an input binary can be executed without being examined first. Unfortunately, the first version of BIRD has several performance problems. This paper describes our experiences of optimizing the first BIRD prototype to remove these problems. In particular, we develop a novel speculative disassembly technique that successfully reaps most of the performance benefits of static disassembly while ensuring the same level of correctness as dynamic disassembly, a bitmap-based target address check algorithm that reduces the fixed performance overhead associated with every instrumentation, and a comprehensive in-place instrumentation technique that relies mostly on instruction substitution and drastically cuts down the number of debug exceptions (int 3) invoked at run time. Together these performance optimizations reduce the average performance overhead of a set of batch Win32 programs from 23.6% to 8.8%.