P-Net: A Representation for Partially-Sequenced, Multi-stream Activity

In this paper, we devise a Propagation Net (P-Net) as a new mechanism for the representation and recognition of multi-stream activity. Most of daily activities can be represented by temporally partial ordered intervals where each interval has not only temporal constraint, i.e., before/after/duration, but also a logical relationship such as a and b both must happen. P-Net associates a node for each interval that is probabilistically triggered function dependent upon the state of its parent nodes. Each node is also associated with an observation distribution function that associates perceptual evidence. This evidence, generated by lower level vision modules, is a positive indicator of the elemental action. Using this architecture, we devise an iterative temporal sequencing algorithm that interprets a multi-dimensional observation sequence of visual evidence as a multi-stream propagation through the P-Net. Simple vision and motion-capture data experiments demonstrate the capabilities of our algorithm.