Stochastic Bayes measures to compare forecast accuracy of software-reliability models

ARE (absolute relative error) and SqRE (squared relative error), are random variables that are suggested as measurements of forecast accuracy of the total number of estimated software failures at the end of a mission time. The purpose is to compare the predictive merit of competing software reliability models, an important concern to software reliability analysts. This technique calculates the Bayes probability of how much better the prediction accuracy is for one method relative to a competitor. This novel approach is more realistic, in the assessment of predictive merit, than (a) comparing merely the average values of ARE and SqRE as conventionally done; and (b) conducting statistical hypothesis tests of pair-wise means of ARE and SqRE, an approach somewhat more sensible than (a), because (b) incorporates variability of predicted values, which (a) does not. To implement this technique, first noninformative (across the border) are used and then informative (specified) priors. For the informative case, half-normal priors are placed on the mean of the ARE or SqRE random variables, because these means are hypothesized to remain peaked around zero relative-error (ideal error percentage). This problem is related to the general problem of ranking usual means discussed in the literature by Berger and Deely (1988), and is a follow-up to an invited research paper presented at ISI-97 by Sahinoglu and Capar (1997)