Methodology for experimental computer vision

A key criticism in the experimental aspect in computer vision is that there are many reported experiments which illustrate results on only a few images and many experiments done all under essentially the same conditions. Such experiments are not sufficient. The author argues that the most informative kinds of experiments should state the set of controlled conditions under which a vision algorithm can be utilized and under which the vision algorithm performance exceeds some given specification. The planning document which describes the design for these experiments is called the experimental protocol. The specific elements of such a protocol are reviewed, stressing that the experimental data analysis plan must state how the hypothesis that the algorithm meets the specified requirement will be tested. The plan must be supported by theoretically developed statistical analysis which shows that an experiment carried out according to the experimental design and analyzed according to the data analysis plan will produce a statistical test itself having a given accuracy