Towards understanding spontaneous speech: word accuracy vs. concept accuracy

Author

Boros, M. ; Eckert, W. ; Gallwitz, F. ; Görz, G. ; Hanrieder, G. ; Niemann, H.

Author_Institution

Bayerisches Forschungszentrum fur Wissensbasierte Syst., Erlangen, Germany

Volume

2

fYear

1996

fDate

3-6 Oct 1996

Firstpage

1009

Abstract

Describes an approach to the automatic evaluation of both the speech recognition and understanding capabilities of a spoken dialogue system for train timetable information. For performance judgement, we use word accuracy for recognition and concept accuracy for understanding. Both measures are calculated by comparing these modules´ outputs with a correct reference answer. We report evaluation results for a spontaneous speech corpus with about 10,000 utterances. We observed a nearly linear relationship between word accuracy and concept accuracy

Keywords

natural language interfaces; public information systems; speech recognition; travel industry; automatic evaluation; concept accuracy; correct reference answer; module outputs; nearly linear relationship; performance judgement; speech recognition; spoken dialogue system; spontaneous speech understanding; train timetable information; utterances; word accuracy; Automatic testing; Monitoring; Natural language processing; Noise measurement; Speech analysis; Speech processing; Speech recognition; Standards development; System performance; System testing;

fLanguage

English

Publisher

ieee

Conference_Titel

Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on

Conference_Location

Philadelphia, PA

Print_ISBN

0-7803-3555-4

Type

conf

DOI

10.1109/ICSLP.1996.607774

Filename

607774