Integrating force and vision feedback within virtual environments for telerobotic systems

Traditional telerobotic systems often provide feedback to the user through a variety of sensing modalities, for example through live video imagery, force reflection, or acoustic signals. When a supervisor guides a robotic task while immersed in a virtual environment, geometric representations of the world are provided as feedback. In this case, the supervisor interacts with the environment by visually observing these virtual objects and directing their motion. At issue is how to appropriately use various sensing modalities provided by disparate sensors in a system of this configuration. This paper focuses on assimilating the disparate feedback provided by force and vision sensors for telerobotic systems guided from within virtual environments. A framework for feedback assimilation is described based on the concept of sensor resolvability. Sensor resolvability is used in two ways, to update the virtual environment and to guide the desired task in the real world. Resolvability selects the appropriate sensing modality to use in updating the virtual environment within which the supervisor is immersed. It is also used to direct semi-autonomous agents that interact directly with the real world to perform the desired task. Experimental results demonstrate the significant advantages of assimilating disparate sensory feedback throughout a telerobotic task using the concept of sensor resolvability