Eliciting probabilities from experts in complex technical problems

A formal elicitation process is summarized to obtain probabilities from experts in a large-scale study involving nuclear safety. A first expert elicitation, using primarily in-house experts and staff members of the project team was strongly criticized in peer review. Subsequently, many changes in the process and procedures were made including training experts in making probability judgments, providing assistance in decomposing judgments, formally eliciting probabilities in individual sessions, and documenting the results. Approximately 1000 probability distributions were assessed from about 40 experts from universities, consulting firms, and national laboratories. This second elicitation was much better received by peer review. Based on this experience, a comprehensive process to elicit probability judgments is outlined in detail