Implicit video multi-emotion tagging by exploiting multi-expression relations

In this paper, a novel implicit video multi-emotion tagging method is proposed, which considers the relations between the users´ outer facial expressions and inner emotions as well as the relations among multiple expressions. First, the audiences´ expressions are inferred through a multi-expression recognition model, which consists of an image driven expression measurement recognition and a Bayesian network representing the co-existence and mutual exclusion relations among multi-expressions. Second, the videos´ multi-emotion tags are obtained from the recognized expressions by another Bayesian Network, capturing the relations between outer expressions and inner emotions. Results of the experiments conducted on the NVIE database demonstrate that multi-expression recognition considering the relations among expressions improves the recognition performance. Furthermore, the performance of our proposed emotion tagging method considering the relations between outer expressions and inner emotions outperforms the traditional expression-based or image-based implicit video emotion tagging methods.