Better than you think: Interval estimators of the difference of binomial proportions

The paper studies explicitly defined interval estimation of the difference in proportions arising from independent binomial distributions for small to moderate sample sizes. In particular, the interval proposed by Agresti and Caffo is compared with the Newcombe interval, the KMS interval of Kulinskaya, Morgenthaler and Staudte, the Wald interval and the ‘Jeffreys’ interval proposed by Brown and Li. Our comparative contour plot summaries empirical studies help to identify where each of the methods performs best in terms of coverage and width. For example, for very unbalanced designs we recommend the Newcombe intervals. For obtaining the nominal coverage, the KMS intervals are recommended, providing coverages nearly always between 95% and 97%. Two new summary scores for interval coverage are introduced. In addition to comprehensive empirical findings, this paper also connects the mean value of the KMS variance stabilized statistic to the Kullback–Leibler symmetrized divergence, which helps to explain the good coverage properties of the interval based on it.