Exploiting a ‘gaze-Lombard effect’ to improve ASR performance in acoustically noisy settings

Author

Cooke, Neil ; Ao Shen ; Russell, Matthew

Author_Institution

Sch. of Electron. & Electr. & Comput. Eng., Univ. of Birmingham, Birmingham, UK

fYear

2014

fDate

4-9 May 2014

Firstpage

1754

Lastpage

1758

Abstract

Previous use of gaze (eye movement) to improve ASR performance involves shifting language model probability mass towards the subset of the vocabulary whose words are related to a person´s visual attention. Motivated to improve Automatic Speech Recognition (ASR) performance in acoustically noisy settings by using information from gaze selectively, we propose a `Selective Gaze-contingent ASR´ (SGC-ASR). In modelling the relationship between gaze and speech conditioned on noise level - a `gaze-Lombard effect´-simultaneous dynamic adaptation of acoustic models and the language model is achieved. Evaluation on a matched set of gaze and speech data recorded under a varying speech babble noise condition yields WER performance improvements. The work highlights the use of gaze information in dynamic model-based adaptation methods for noise robust ASR.

Keywords

acoustic noise; gaze tracking; probability; speech recognition; ASR performance; SGC-ASR; WER performance improvements; acoustic models; acoustically noisy settings; automatic speech recognition performance; dynamic model-based adaptation methods; eye movement; gaze information; gaze-Lombard effect; language model probability mass; noise level; selective gaze-contingent ASR; speech babble noise condition; speech data; visual attention; vocabulary; Acoustic noise; Adaptation models; Noise; Noise measurement; Speech; Visualization; ASR; Acoustic Model adaptation; Language Model adaptation; Mutual information; acoustic noise; gaze; noise robust ASR. eye movement; speech; visual attention;

fLanguage

English

Publisher

ieee

Conference_Titel

Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on

Conference_Location

Florence

Type

conf

DOI

10.1109/ICASSP.2014.6853899

Filename

6853899