What size test set gives good error rate estimates?

Author

Guyon, Isabelle ; Makhoul, John ; Schwartz, Richard ; Vapnik, Vladimir

Author_Institution

955 Creston Rd., Berkeley, CA, USA

Volume

20

Issue

1

fYear

1998

fDate

1/1/1998 12:00:00 AM

Firstpage

52

Lastpage

64

Abstract

We address the problem of determining what size test set guarantees statistically significant results in a character recognition task, as a function of the expected error rate. We provide a statistical analysis showing that if, for example, the expected character error rate is around 1 percent, then, with a test set of at least 10,000 statistically independent handwritten characters (which could be obtained by taking 100 characters from each of 100 different writers), we guarantee, with 95 percent confidence, that: (1) the expected value of the character error rate is not worse than 1.25 E, where E is the empirical character error rate of the best recognizer, calculated on the test set; and (2) a difference of 0.3 E between the error rates of two recognizers is significant. We developed this framework with character recognition applications in mind, but it applies as well to speech recognition and to other pattern recognition problems

Keywords

character recognition; maximum likelihood estimation; normal distribution; statistical analysis; character recognition task; error rate estimates; expected error rate; pattern recognition; speech recognition; statistical analysis; statistically independent handwritten characters; statistically significant results; test set; Benchmark testing; Character recognition; Error analysis; Handwriting recognition; Helium; Pattern recognition; Speech recognition; State estimation; Statistical analysis; System testing;

fLanguage

English

Journal_Title

Pattern Analysis and Machine Intelligence, IEEE Transactions on

Publisher

ieee

ISSN

0162-8828

Type

jour

DOI

10.1109/34.655649

Filename

655649