Human behavior segmentation and recognition using Continuous Linear Dynamic System

Recognizing continuous action composition in human behavior is an important and yet challenging problem. In this paper we tackle the task by developing both reliable image features and classification algorithms. For image features, we introduce the Embedded Optical Flow (EOF) feature based on embedding optical flow using Locality-constrained Linear Coding with weighted average pooling. The EOF feature is histogram-like but presents excellent linear separability. For classification, we propose the Continuous Linear Dynamic System (CLDS) framework that consists of two sets of Linear Dynamic System (LDS) models, one to model the dynamics of individual actions and the other to model the transition between actions. The inference process estimates the best decomposition of the whole sequence into continuous alternating between human actions and action transitions. In this way, both action type and action boundary can be accurately recognized. Extensive experiments demonstrate the effectiveness and efficiency of the proposed EOF feature and CLDS algorithm.