Sample Size Analysis for Confidence Interval Estimation of Performance Metrics in ATR Evaluation

In this paper, we address the problem of sample size requirement for confidence interval estimation of performance metrics in ATR evaluation. The sample size of test data in ATR evaluation application has been reviewed firstly. We choose the Bayesian method to analyze the problem and select "minimum length criterion" to obtain the shortest confidence interval (CI) with the same estimation accuracy. The binomial probability density function is regarded as the likelihood function to calculate the posterior distribution about the metrics. Then CI accuracy requirement in ATR evaluation is discussed. Because the posterior distribution of the metrics depends on the test result, criteria to eliminate the uncertainty from the test result must be considered. The worst outcome (WOC) criterion is chosen to calculate sample sizes for various CI accuracy requirements and the corresponding sample sizes are listed in Table I. Table I shows that the sample size is very large when the CI accuracy requirement is high. Two approaches (specifications and prior information) to reduce the sample size are proposed and discussed. The absolute number of the sample size reduction is great when using the specifications in ATR evaluation applications. Table II contains those minimum sample sizes with various specifications, and its comparison with table I is shown. Whereas the sample size reduction is not large when using prior information because precise prior information about the metrics is always absent. The approximate sample size reductions can be got from table III and IV when considering beta distribution as the prior distribution.