Bootstrap estimates for confidence intervals in ASR performance evaluation

Author

Bisani, M. ; Ney, H.

Author_Institution

Dept. of Comput. Sci., Technische Hochschule Aachen, Germany

Volume

1

fYear

2004

fDate

17-21 May 2004

Abstract

The field of speech recognition has clearly benefited from precisely defined testing conditions and objective performance measures such as word error rate. In the development and evaluation of new methods, the question arises whether the empirically observed difference in performance is due to a genuine advantage of one system over the other, or just an effect of chance. However, many publications still do not concern themselves with the statistical significance of the results reported. We present a bootstrap method for significance analysis which is, at the same time, intuitive, precise and and easy to use. Unlike some methods, we make no (possibly ill-founded) approximations and the results are immediately interpretable in terms of word error rate.

Keywords

error statistics; estimation theory; speech recognition; statistical analysis; ASR performance evaluation; bootstrap method; confidence interval estimation; speech recognition; statistical estimates; statistical significance analysis; word error rate; Automatic speech recognition; Computer science; Dynamic programming; Error analysis; Heuristic algorithms; Probability; Speech recognition; Statistical analysis; Testing; Vocabulary;

fLanguage

English

Publisher

ieee

Conference_Titel

Acoustics, Speech, and Signal Processing, 2004. Proceedings. (ICASSP '04). IEEE International Conference on

ISSN

1520-6149

Print_ISBN

0-7803-8484-9

Type

conf

DOI

10.1109/ICASSP.2004.1326009

Filename

1326009