Goal-directed visual inference for multi-modal analysis and fusion

This paper presents an extensible architecture for the interpretation of visual data and fusion of different sources of information. It is based on a joint inference approach which relies on a novel active knowledge representation methodology consisting of an ontological language. This language allows one to express task-specific knowledge of the syntactic and semantic structure of entities, relationships, and events of interest in a given domain. Current research on video analysis and multi-modal fusion for a sentient computing system are used to illustrate the advantages of this approach.