Co-channel audiovisual speech separation using spectral matching constraints

In this paper, the problem of co-channel speech separation for convolutive mixtures is considered where visual cues from one of the speakers is available as side information. The visual cues from the one speaker in the two speaker speech separation are used to estimate the spectral content of the speech and this spectral estimate is in turn used to constrain the solution of the coupling reconstruction filters in the convolutive mixture. The preliminary experimental results show that good performance in speech separation is obtained for our limited case study of visual cues obtained from the spoken numbers of "one" thru "four".