Test-Based Specifications of Components and Systems

Several program-analysis schemes now make unusual use of information derived from execution sampling. From finite test sets these techniques derive measures with wider meaning, which can then be exploited in novel ways. We call test information used beyond its actual limitations a test-based specification. The idea is quite different from the usual software specification, which is an a priori statement of what a program should do. Rather, a test-based specification is an empirical approximation to what a program actually does do. The great virtue in analysis using test-based specifications is that (in contrast to the usual software specifications) test-based analysis is decidable and automatic. A test-based specification system for component-based software development (CBSD) has been implemented over the past five years, along with an extension of fundamental testing theory to precisely describe its properties. The CBSD tools provide an ideal context for experiments to study test-based specification, providing insights into sub- domain testing, composition of test results, and especially the role that program persistent state plays in testing and analysis. This paper describes the CBSD theory and tools, lists insights gained, and suggests new ways to think about and practice testing using test-based specifications.