Assessing the sensitivity to usage profile changes in test planning

Software reliability is an important characteristic for most systems, but due to its dynamic properties it is hard to assess until very late in the development. Nevertheless, the testing must be planned to meet the reliability requirements. In test planning, the notion of usage coverage may be used as an indicator of reliability as they are correlated with each other. When the testing is planned, the test cases to be run and the usage profile are derived. The usage profile is estimated using the available information of the expected usage. There are therefore some uncertainties in the estimated usage profile. The paper presents and evaluates a method for analysing the impact of uncertainties in the usage profile on the usage coverage. The method is applied during test planning to evaluate how sensitive the usage coverage is to usage profile uncertainties. Different usage profiles are simulated using the expected usage profile and an uncertainty expressed as a percentage value or a range within which the changed usage profile may vary. Test cases are derived from the expected usage profile and the resulting usage coverage is estimated for each simulated usage profile. Thus the impact in the usage coverage can be analysed. The presented analysis method is illustrated with an example system. One of the conclusions from this first study of the method is that the sparseness of the usage profile, which is defined by limitations in the usage pattern, reduces the impact of the usage profile uncertainties. Further, the number of test cases is an important factor in the sensitivity to uncertainties in the usage profile