Recognizing Conversational Expressions Using Latent Dynamic Conditional Random Fields

Facial expressions are one of the most important elements for our social interaction. Automatic processing and recognition of facial expressions is hence one of the core areas in computer vision, computer graphics, and social signal processing. Conditional Random Fields (CRFs) and their extensions are widely used for recognizing facial expressions. Most research in this area, however, is done either with a limited set of emotional expressions (such as the six universal expressions), or it concentrates on extracting facial action units (individual muscle movements) from video sequences. Little research has been conducted to analyze the complex facial movements that occur in conversational contexts. Conversational expressions such as "agree", "disagree", "thinking", "looking confused", however, form an integral part of non-verbal communication and systems that can automatically parse and understand such expressions are a key ingredient for the development of efficient human-computer interaction systems. Since conversational expressions may consists of several sub-expressions and contain complex dynamics, however, standard CRF approaches are not suited for the task. In this paper, we conduct a detailed comparison of CRFs and Latent Dynamic Conditional Random Fields (LDCRFs) for recognizing complex conversational expressions. We show the importance of modeling sub-expression dynamics and discuss challenges for applying LDCRFs to recognize a large set of conversational expressions.