Replacing a Human Hand by a Virtual Hand and Adjusting Its Posture to a Virtual Object for Its Manipulation in an AR Environment

This paper discusses virtual object manipulation using a human hand in an AR environment. For this purpose, we need to measure the configuration, which includes the position, pose and posture, of the human hand. However, if we employ a data glove for the measurement, the glove appears in the video images synthesized by AR. If we otherwise employ a vision-based approach, the obtained configuration often includes larger errors due to occlusions among fingers. Moreover, the configuration of the human hand in anyway often includes some cognitive errors in manipulating the virtual object, which is only visible in the synthesized videos. To cope with these problems, we propose to replace the human hand with a data glove with a virtual hand in the video and adjust its configuration so that it properly grasps the virtual objects without errors.