Unreplicated experimental designs in reliability growth programs

Reliability-growth programs are usually implemented through repairs and modifications aimed at lengthening the system times-between-failures. The many factors that affect system performance suggest that a rational alternative to the common tweaking approach to reliability growth is experimental design techniques. However, economic and time constraints generally impose unreplicated experiments where the response is measured by the times between system failures. We show that statistical analysis of these experiments is more difficult because of this condition. Fortunately, several techniques to detect statistically significant effects in unreplicated experiments are available and can be used to identify which factors and interactions have the strongest influence in lengthening the system up-time. These techniques use a variety of principles and, expectedly, perform somewhat differently under various conditions. We present the results of the most extensive Monte Carlo benchmark ever undertaken to test the performance of these techniques. A new figure of merit is introduced, allowing a 1-quantity summary of the statistical behavior of the tested techniques. A numerical example illustrates the problem and its solution