How the testing techniques for a decision support system changed over nine years

A decision support system has been under development since 1985 to help speech clinicians diagnose small children who have begun to stutter. This paper describes how testing of the system evolved during these nine years. Testing included: (1) having an expert use and evaluate it, (2) running test cases, (3) developing a program to detect redundant rules, (4) using the Analytic Hierarchy Process, (5) running a program that checks a knowledge base for consistency and completeness, (6) having five experts independently critique the system, (7) obtaining diagnoses of stuttering from these five experts derived from reports of children who had been evaluated for possible stuttering problems, (8) using the system to expose missing and ambiguous information in 30 clinical reports, and (9) analyzing the dispersion and bias of six experts and the decision support system in diagnosing stuttering. When using the final system, three clinicians with widely differing backgrounds produced diagnostic opinions that evidence little variability and were indistinguishable from those of a panel of five experienced clinicians.