Constrained Optimization for Audio-to-Visual Conversion

We have developed a new audio-to-visual conversion algorithm that uses a constrained optimization approach to take advantage of dynamics of mouth movements. Based on facial muscle analysis, the dynamics of mouth movements is modeled, and constraints are obtained from it. The obtained constraints are used to estimate visual parameters from speech in a framework of hidden Markov model (HMM)-based visual parameter estimation. To solve the constrained optimization problem, the Lagrangian approach is used to transform the constrained problem into an unconstrained problem in our implementation. The proposed method is tested on various noisy environments to show its robustness and correctness. Our proposed algorithm is favorably compared with the mixture-based HMM method, which also uses audio-visual HMMs and finds optimal estimates based on a joint audio-visual probability distribution. Our proposed algorithm can estimate optimal visual parameters while satisfying the constraints and avoiding performance degradation in noisy environments.