Frequencygrams and multi-feature joint sparse representation for action and gesture recognition

Features play a vital role in human action recognition (HAR), as they encapsulate the underlying dynamics of the action. We propose the features (frequencygrams) based on frequency domain analysis of histograms of the motion and its spatiotemporal gradient (rate of change in motion flow). Feature extraction is quite simple and can be performed in real time using sparse or interest point motion flow. They are resilient to delayed initiated actions, scale variation, moving background, sudden illumination changes (high frequency noise) and avoid the overload of person detection and tracking. Being robust to camera motions, they also provide a natural, compact and discriminative representation for reciprocating motions by preserving comprehensive temporal information of the action sequences. As other global features also bear some action semantics, we fuse all these features together in a systematic way to improve the overall HAR performance, by employing the joint sparse representation with group sparsity regularization. The extensive experimental results, on three benchmark action datasets and one gesture recognition dataset, show the effectiveness and generality of the proposed method.