Automated Conformance Testing of Java Virtual Machines

We present a technique to fully automate the conformance testing of a Java virtual machine (JVM) implementation to the structural constraints it must satisfy to enforce type safety of program execution. The approach is based on formal modeling of the JVM as a finite state machine, ruled by the Java standard specifications. The model is used to derive a test suite and corresponding oracle that systematically explores the space of illegal states reachable by a JVM implementation under test. Also, a degree of conformance to the JVM specifications (i.e., too strict or too coarse) can be assessed by counting the number of false positives. Despite the huge test space, the entire proposed process need not human supervision. The technique is black box, fully automated, and can be applied for validating final products or during development i.e. for debugging purposes.