Intuitive bimanual telemanipulation under communication restrictions by immersive 3D visualization and motion tracking

Robots which solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. As fully autonomous robots are not yet capable of operating in highly unstructured real-world scenarios, teleoperation is often used to embed the cognitive capabilities of human operators into the robotic system. The many degrees of freedom of anthropomorphic robots and communication restrictions pose challenges to the design of teleoperation interfaces, though. In this work, we propose to combine immersive 3D visualization and tracking of operator head and hand motions to an intuitive interface for bimanual teleoperation. 3D point clouds acquired from the robot are visualized together with a 3D robot model and camera images using a tracked 3D head-mounted display. 6D magnetic trackers capture the operator hand motions which are mapped to the grippers of our two-armed robot Momaro. The proposed user interface allows for solving complex manipulation tasks over degraded communication links, as demonstrated at the DARPA Robotics Challenge Finals and in lab experiments.