Automatically Generating External OS Kernel Integrity Checkers for Detecting Hidden Rootkits

The integrity checker validates the data structures in a target OS kernel from outside to enhance system security. Because of a huge number of kernel data structures, all possible invariants cannot be generated automatically, as we encounter a combinatorial explosion. In this paper, we propose a framework to generate a practical integrity checker automatically without examining all data structures in an OS kernel. Hidden rootkits infect the pointer variables of kernel data structures, a filter proposed in the framework reduces the number of target kernel data structures without decreasing the detection accuracy. In our experiments, the proposed system generates an integrity checker for three Linux kernels in a practical time, and a generated integrity checker can detect all of the hidden root kits infecting the kernel data structures.