Integration of spatial and temporal contexts for action recognition by self organizing neural networks

We present a neural architecture which learns to recognize object-directed actions by visually observing examples. Our architecture learns to extract spatial (e.g. object relationships) and movement contexts, self-organizes symbolic representations of them, and integrates them in a temporal context, producing self-organized symbolic action classes. Each of the above functions is realized by a self organizing neural network module. A preprocessing module takes a video input and feeds object and movement features to the modules. The System can learn to recognize simple grasp-transfer-place actions performed by a human hand in 2D scenes by simply observing example performances. Intermediate and top level categorical representations are self organized without explicit external supervisory signals.