Exploring temporal patterns in classifying frustrated and delighted smiles (Extended abstract)

We created two experimental situations to elicit two affective states: frustration and delight. In the first experiment, participants were asked to recall situations while expressing either frustration or delight. The second experiment tried to elicit these states naturally with a frustrating experience and a delightful video. There were two significant differences between the acted and natural occurrences of the expressions. First, the acted instances were much easier for the computer to classify. Second, in 90 percent of the acted cases, participants did not smile when frustrated. In 90 percent of the natural cases, participants smiled during the frustrating interaction, despite self-reporting significant frustration with the experience. As a follow up study, we develop an automated system to distinguish between naturally occurring spontaneous smiles under frustrating and delightful stimuli by exploring their temporal patterns, given video of both. We extracted local and global features related to human smile dynamics. Next, we evaluated and compared two variants of Support Vector Machines (SVM), Hidden Markov Models (HMM), and Hidden-state Conditional Random Fields (HCRF) for binary classification. While human classification of the smile videos under frustrating stimuli was below chance, a dynamic SVM classifier obtained an accuracy of 92 percent in distinguishing smiles under frustrating and delighted stimuli.