Predictive display models for tele-manipulation from uncalibrated camera-capture of scene geometry and appearance

In tele-robotics delayed visual feedback to the human operator can degrade task performance significantly. To improve this, predictive display, uses a scene model to estimate and render immediate visual feedback based on the operator´s control commands. Traditional predictive display involves the calibration and overlay of an a-priori model with the delayed real video feedback. In this paper we present an image-based method where the scene geometry and appearance is captured using structure-from-motion by an uncalibrated eye-in-hand camera mounted on the remote robot. The model is then compressed and transmitted to the operator site, where it is used to generate immediate feedback in response to the operators movements. Calibration problems are avoided since the model is captured by the same scene camera as is being simulated in the predictive display. We show experiments where we capture the appearance of a robot hand and transmit it over the network to the operator site where the model renders scene appearance change in response to operator viewpoint motion.