Managing stochastic algorithms cross-validation variability using an interval valued multiple comparison procedure

The usual procedure to compare metaheuristics or evolutionary algorithms using cross validation is repeating the training stage several times for each train/test pair. In general, the results of the different repetitions are not independent and this practice is questionable. In this work, it is suggested to represent the results of each train/test set pair by an interval or by a fuzzy set and use the proper statistical tests to these kind of data. In this way, the paired comparison of these sets leads to an interval valued p-value or to a fuzzy p-value that holds information about the mean differences but also about the differences in dispersion between the compared algorithms.