Tri-modal speech recognition for noisy and variable lighting conditions

Automatic speech recognition (ASR) has found widespread applications in consumer products. Often, ASR performance can be compromised in noisy environments. Previous research has shown that adding visual cues can improve the performance of ASR, particularly in noisy environments. However, audiovisual (AV) ASR is not robust against changing lighting conditions, which are often encountered by end users of consumer products. Since thermal imaging is highly invariant to changing lighting conditions, we propose a tri-modal ASR involving thermal imaging and audiovisual (TAV) data for consumer applications. Experimental results demonstrate the applicability of this approach over a range of signal-to-noise ratios: Tri-modal TAV recognition rates were +39.2% over audio-only and +11.8% over AV recognition rates.