A sensorimotor account of visual and tactile integration for object categorization and grasping

The fusion of tactile and visual modalities is crucial for understanding objects and learning how to manipulate them. A common modus operandi in robotics is to deal with each of these modalities separately. We propose an integrated approach that associates to local visual features of an object, tactile feedback of the effector when touching that part of the object. Thus the agent learns to predict from a visual scene the shape/curvature properties of the object. The associated curvature properties are directly linked to grasp possibilities (as in approaches like [1] and [2]) but can also provide the agent with object categorization regarding the distribution of curvature classes.