Clauselets: Leveraging Temporally Related Actions for Video Event Analysis

We propose clause lets, sets of concurrent actions and their temporal relationships, and explore their application to video event analysis. We train clause lets in two stages. We initially train first level clause let detectors that find a limited set of actions in particular qualitative temporal configurations based on Allen\´s interval relations. In the second stage, we apply the first level detectors to training videos, and discriminatively learn temporal patterns between activations that involve more actions over longer durations and lead to improved second level clause let models. We demonstrate the utility of clause lets by applying them to the task of "in-the-wild" video event recognition on the TRECVID MED 11 dataset. Not only do clause lets achieve state-of-the-art results on this task, but qualitative results suggest that they may also lead to semantically meaningful descriptions of videos in terms of detected actions and their temporal relationships.