Sensor Fusion and Environmental Modelling for Multimodal Sentient Computing

Sentient computing uses networks of sensors to capture and maintain an internal representation ("world model") of an indoor environment, thereby allowing applications to have greater awareness of users and their requirements. This chapter shows how computer vision information obtained by several cameras can be used to enhance the capabilities of a sentient computing system which previously relied on ultrasound to track people and devices. Integration is achieved at the system level through the metaphor of shared perceptions in the sense that the different modalities are guided by and provide updates for a shared internal model. This world model incorporates aspects of both the static (e.g. positions of office walls and doors) and dynamic (e.g. location and appearance of devices and people) environments. It serves both as an ontology of prior information and as a source of context which is shared between applications. Fusion and inference are performed by Bayesian networks which model the probabilistic dependencies and reliabilities of different sources of information over time. It is shown that the fusion process significantly enhances the capabilities and robustness of the system, thus enabling it to maintain a richer and more accurate world model.