Cross-speaker viseme mapping using hidden Markov models

In this paper, a method of mapping visual speech between different speakers is proposed. This approach adopts hidden Markov model (HMM) to model the basic visual speech element - viseme. Some mapping terms are applied to associate the state chains decoded for the visemes produced by different speakers. The HMMs configured in this way are trained using the Baum-Welch estimation, and are used to generate new visemes. Experiments are conducted to map the visemes produced by several speakers to a destination speaker. The experimental results show that the proposed approach provides good accuracy and continuity for mapping the visemes.