Sensor-aided teleoperated grasping of transparent objects

This paper presents a method of augmenting streaming point cloud data with pretouch proximity sensor information for the purposes of teleoperated grasping of transparent targets. When using commercial RGB-Depth (RGB-D) cameras, material properties can significantly affect depth measurements. In particular, transparent objects are difficult to perceive with RGB images and commercially available depth sensors. Geometric information of such objects needs to be gathered with additional sensors, and in many scenarios, it is of interest to gather this information without physical contact. In this work, a non-contact pretouch sensor fixed to the robot end effector is used to sense and explore physical geometries previously unobserved. Thus, the point cloud representation of an unknown, transparent grasp target, can be enhanced through telerobotic exploration in real-time. Furthermore, real-time haptic rendering algorithms and haptic virtual fixtures used in combination with the augmented streaming point clouds assist the teleoperator in collision avoidance during exploration. Theoretical analyses are performed to design virtual fixtures suitable for pretouch sensing, and experiments show the effectiveness of this method to gather geometry data without collision and eventually to successfully grasp a transparent object.