Confidence Interval for Measure of Algorithm Performance Based on B

In studies on the application of machine learning such as Information Retrieval (IR), the focus is typically on the estimation of the F₁ measure of algorithm performance. Approximate symmetrical confidence intervals constructed by the F₁ value based on cross-validated L distribution are commonly used in the literature. However, theoretical analysis on the distribution of F₁ values shows that such distribution is actually non-symmetrical. Thus, simply using symmetrical distribution to approximate non-symmetrical distribution may be inappropriate and may result in a low degree of confidence and long interval length for the confidence interval. In the present study, a non-symmetrical confidence interval of the F₁ measure based on Beta prime distribution is constructed by using the F₁ value computed based on the average confusion matrix of a blocked 3 x 2 cross-validation. Experimental results show that in most cases, our method has high degrees of confidence. With an acceptable degree of confidence, our method has a shorter interval length than the approximate symmetrical confidence intervals based on the blocked 3 x 2 and 5 x 2 cross-validated L distributions. The approximate symmetrical confidence interval based on the 10-fold cross-validated L distribution has the shortest interval length of the four confidence intervals but with low degrees of confidence in all cases. Taking these two factors into consideration, our method is recommended.