Using observation and refinement to improve distributed systems test

Testing a distributed system is difficult. Good testing depends on both skill and understanding the system under test. We have developed a method to observe the system at the CORBA remote-procedure-call level and then use dynamic-query-based visualization to refine and improve the test cases. The method and accompanying tools have been tested and refined by using them as part of the software support effort for two distributed application, each having about 500 K lines of code. During this time the tools have been adapted to support testing by adding a scripting mechanism that permits the visualization tool to specify test reports. We also added parameter value observation and reporting. Finally, we added an active probing mechanism to induce faults and delays in order to stress the system under test. Our efforts have led to a substantial improvement in system test quality.