Comparing diagnostic tests without a gold standard: How well does the Hui-Walter model really work?

Purpose Hui and Walter (1980) proposed a model to estimate the sensitivities (Se) and specificities (Sp) of two diagnostic tests in the absence of a “gold standard.” Many methodological advances in this area have used their model as a starting point. Its performance has not been empirically investigated, however. This study examines the Hui-Walter (H-W) modelʹs performance both when its assumptions are met and when they are not. Methods Following the model, this study considered samples drawn from two populations of low (<50%) and high (>50%) disease prevalences (n = 250 x 2 groups). Test parameters (Se, Sp) ranged from 50% to 100%. All parameters were randomly determined at each iteration of the simulation. The two tests were assumed either to be independent (r = 0) or correlated (r = 0.5) given an individualʹs disease status. A covariate that reduced the specificity of one of the tests was considered either to be absent or to be present at random low levels (≤20%) in the two populations. Data were generated in SAS, and parameters and credible intervals were estimated with WinBUGS. Results When model assumptions are true, the H-W model provides unbiased (bias: −0.01–0.002) estimates of the testsʹ performances and accurately describes the testsʹ relative performances 96% of the time. When the tests are correlated and when test performance is modified by the presence of a covariate, the model does not perform well. Average bias of test parameter estimates exceeds 0.10, with the 95% credible intervals covering the true values only 10%–58% of the time. The relative performances of the two tests, however, were accurately described 96% of the time. Conclusion The H-W model provides a solid foundation for methods to compare test performance without a gold standard. The model is sensitive, however, to violations of its assumptions. Methods are needed that can incorporate covariates that modify test performance.