Optimistic bias in the assessment of high dimensional classifiers with a limited dataset

It is commonly recognized that using the same dataset for training and testing the classifier introduces optimistic bias in estimating classifier performance. However, bias of the same kind may still exist even when independent datasets are used for training and testing a classifier. This problem is especially important in the setting of high dimensional feature space and limited data. Bioinformatics data is typically characterized by a tremendous amount of data per patient but from a limited number of patients. Often the entire data set is utilized in a “pre-training” stage during which the feature set is winnowed to a manageable number, and the parameters of the training algorithm are established. Subsequently the data is bifurcated into training and test sets; however, bias has already been introduced into the classifier development process. We investigate the significance of this bias by performing simulated gene expression experiments. We find that, for data with moderate intrinsic separability and modest sample size, any observed separation is due to selection bias introduced in the aforementioned pre-training process. For greater intrinsic separability, correct data hygiene, i.e., complete separation of development and validation data yields a positive result, but one far less impressive than that mistakenly obtained using incomplete data separation.