Point-cloud-based model-mediated teleoperation

In this paper, we extend the concept of model-mediated teleoperation (MMT) to six degrees-of-freedom in complex environments using a time-of-flight (ToF) camera. Compared to the original MMT method, the remote environment is no longer approximated by a simple planar surface, but by a point cloud model. Thus, object surfaces with complex geometry can be used in MMT. In our proposed system, the point cloud model is captured by the ToF camera with high temporal resolution (up to 160fps) and a flexible work range (10cm to 5m). Updating the model of the remote environment while the robot is in operation is thus easier compared to the original MMT approach. The point cloud model is transmitted from the teleoperator to the operator using a lossless H.264 codec. In addition, a simple point-cloud-based haptic rendering algorithm is adopted to generate the force feedback signal directly from the point cloud model without first converting it into polygons. Moreover, to compensate for the estimation error of the point cloud model, adaptive position and force control schemes are applied to enable stable and transparent teleoperation. Our experiments demonstrate the feasibility and benefits of utilizing the proposed method in MMT.