Evaluating the performance of connected-word speech recognition systems

Outputs of connected-word recognizers may contain substitution, deletion and insertion errors, and their interpretation is not trivial. Simulations show that the commonly used dynamic programming word-sequence matching algorithm has serious shortcomings as an evaluation method at low performance levels, though it is generally reliable at high performance levels. The strategy of comparing input and output words in strict sequence is found to have little to recommend it. A method using word end-point information, which provides precise, detailed performance analyses, is described. Tests with real data confirm the reliability of the end-point method and the presence of positive bias in performance estimates form the word-sequence matching method