Reliability estimation during prototyping of knowledge-based systems

Many knowledge based systems are designed and built with little attention paid to the reliability of the output. In this paper, we present an approach, using partitioning of both the knowledge base and the input space, that allows for the measurement of the reliability during any program increment in a rapid prototyping development cycle. Before presenting the approach, we formalize the problem using concepts from general systems theory and then describe our three objectives: 1) measurement of the reliability of the knowledge-based system at the current program increment, 2) prediction of the reliability of the future system, and 3) support for design decisions. Finally, we apply our approach to a design-aiding knowledge-based system for the selection of materials under various climatic conditions. The design-aiding knowledge-based system is used by U.S. Army personnel in the development of equipment to be used by the U.S. Army in various regions of the world. We find that the current system, containing 40 rules, has a reliability of approximately 0.85. However, more importantly, we have discovered the rules that led to many of the failures