A method to estimate the reliability of expert judgement

One of the difficulties encountered in the development of an expert system is the fact that the available information/evidences are not known with certainty and very often they are not consistent, and conflict with one another to some extent. The authors present a technique to estimate the variability of results obtained from inconsistent information, available in the form of ratio judgements. These variabilities can then be used as a measure of reliability or creditability of these results. The inconsistency in the expert judgements is accounted for by associating them with error factors. The variability of the priorities computed from these expert judgements is obtained by modeling the error factors as log-normally distributed, chosen to satisfy some basic assumptions. The appropriateness of this distribution is further justified by comparing its performance with poor performances exhibited by other probable distributions. The validity of the model results are verified by numerical simulation. This is carried out by constructing a large number of matrices of judgement ratios conforming to the chosen error distribution. The resulting priorities are analyzed and compared with the model results. For the case considered the model and observed priorities are found to be within 2%, and their standard deviations within 5% of each other