The effect of finite sample size on the holdout error probability estimator of homoscedastic multi-class Gaussian classification problems

Consider a homoscedastic multi-class Gaussian classification problem where the class mean vectors and the common covariance matrix are not known to the practitioner. Rather, they are estimated from given sample vectors available for each class. In this paper, an empirical procedure for approximating the bias of the holdout estimator of the Bayesian error probability (BEP) is presented. Synthetic experiments demonstrate the accuracy of the proposed procedure and how it can be used for guiding the practitioner about the necessary amount of data vectors required to achieve a certain level of accuracy in the BEP estimation. When applied to real world classification problems from the UCI machine learning repository, the proposed procedure was successfully used to estimate the test error probability based on the training data only. Moreover, with a reasonable degree of accuracy, the proposed procedure predicted the test BEP when the amount of the training data in increased.