Matryoshka: Strengthening Software Protection via Nested Virtual Machines

The use of virtual machine technology has become a popular approach for defending software applications from attacks by adversaries that wish to compromise the integrity and confidentiality of an application. In addition to providing some inherent obfuscation of the execution of the software application, the use of virtual machine technology can make both static and dynamic analysis more difficult for the adversary. However, a major point of concern is the protection of the virtual machine itself. The major weakness is that the virtual machine presents a inviting target for the adversary. If an adversary can render the virtual machine ineffective, they can focus their energy and attention on the software application. One possible approach is to protect the virtual machine by composing or nesting virtualization layers to impart virtual machine protection techniques to the inner virtual machines "closest" to the software application. This paper explores the concept and feasibility of nested virtualization for software protection using a high-performance software dynamic translation system. Using two metrics for measuring the strength of protection, the preliminary results show that nesting virtual machines can strengthen protection of the software application. While the nesting of virtual machines does increase run-time overhead, initial results indicate that with careful application of the technique, run-time overhead could be reduced to reasonable levels.