Systematic Testing of Database Engines Using a Relational Constraint Solver

We describe an automated approach for systematic black-box testing of database management systems (DBMS) using a relational constraint solver. We reduce the problem of automated database testing into generating three artifacts: (1) SQL queries for testing, (2) meaningful input data to populate test databases, and (3) expected results of executing the queries on the generated data. We leverage our previous work on ADUSA and the Automated SQL Query Generator to form high-quality test suites for testing DBMS engines. This paper presents a detailed description of our framework for Automated SQL Query Generation using the Alloy tool-set, and experimental results of testing database engines using our framework. We show how the main SQL grammar constraints can be solved by translating them to Alloy constraints to generate semantically and syntactically correct SQL queries. We also present experimental results of combining ADUSA and the Automated SQL Query Generator, and applying our framework to test the Oracle 11g database. Our framework generated 5 new queries, which reveal erroneous behavior of Oracle 11g.