Automated lip-sync animation as a telecommunications aid for the hearing impaired

Vocal communication is most effective when the listener is able to observe the mouth of the speaker. This is especially true for the hearing impaired, and dramatically true for the deaf, who rely on lip-reading for comprehending speech. Communication over telephone lines is particularly onerous for the hearing impaired as visual information is unavailable. Our research addresses that problem by providing a computational means of taking speech as input and producing an animated mouth as output that moves precisely as if it were articulating the speech. In this paper we continue reporting on our progress in using moments of spectra-a measure of spectral shapes-to provide a direct mapping from the speech signal to parameters controlling the shape of the lips and position of the jaw during the articulation of the speech. The method requires no text nor does it rely on any form of speech recognition. We report in particular on the progress we have made in distinguishing the visemes-the visible phonemes-corresponding to /m/ and /n/