The influence of automatic speech recognition accuracy on the performance of an automated speech assessment system

The effectiveness of automated scoring systems for evaluating spoken language proficiency depends greatly on the quality of the automatic speech recognition (ASR) output that is used to calculate the features for the scoring model. In this paper, we examine the effects of ASR word error rate (WER) on the scores produced by a system for automated scoring of non-native English speaking proficiency, as well as on the scoring model features (especially content features) in order to demonstrate the impact of ASR improvements on the performance of the automated speech assessment system. Five different sets of transcriptions with varying degrees of WER ranging from 0% to 52% (including four sets of ASR hypotheses and manual transcriptions) were obtained for a dataset of spoken responses from a pilot administration of an assessment of non-native English speaking proficiency. The experimental results show that higher performing ASR leads to better performance in the automated assessment system; furthermore, the correlation between human and automated scores drops substantially with an increase in WER from 10.7% to 28.9%, whereas the correlation changes little within the following two ranges of WERs: 0% to 10.7% and 28.9% to 52%. A detailed analysis of the features used in the scoring model shows that the ASR errors have a bigger impact on the content features than the delivery and language use features.