Experiences from an Initial Study on Risk Probability Estimation Based on Expert Opinion

Background: Determining the factor probability in risk estimation requires detailed knowledge about the software product and the development process. Basing estimates on expert opinion may be a viable approach if no other data is available. Objective: In this paper we analyze initial results from estimating the risk probability based on expert opinion to answer the questions (1) Are expert opinions consistent? (2) Do expert opinions reflect the actual situation? (3) How can the results be improved? Approach: An industry project serves as case for our study. In this project six members provided initial risk estimates for the components of a software system. The resulting estimates are compared to each other to reveal the agreement between experts and they are compared to the actual risk probabilities derived in an ex-post analysis from the released version. Results: We found a moderate agreement between the rations of the individual experts. We found a significant accuracy when compared to the risk probabilities computed from the actual defects. We identified a number of lessons learned useful for improving the simple initial estimation approach applied in the studied project. Conclusions: Risk estimates have successfully been derived from subjective expert opinions. However, additional measures should be applied to triangulate and improve expert estimates.