Interaction in robotics with a combination of vision, tactile and force sensing

Human dexterity relies on the simultaneous use of three perceptual modalities: vision, tactile and force. Existing robotic systems are still very far from humans in terms of their manipulation skills. This fact can be partly accounted for by a lack of integrated multimodal approaches to the problem. We present our research on the combination of vision, tactile and force sensing for manipulation with robotic hands, as part of our FPI approach the Framework for Physical Interaction, in which a grasp and a task are concurrently planned. A visual servoing loop is used to control the hand for task execution, which we combine with a tactile control signal, as well as an impedance force controller. The task can be performed even if a sensor is not available or provides inaccurate data. A number of experiments are reported for comparing the results of the integrated trimodal controller. Results show that it outperforms the bimodal or single-sensor approaches.