Interactive Manipulation of Augmented Objects in Marker-Less AR Using Vision-Based Hand Interaction

In this paper, we present a vision-based framework to manipulate the augmented reality (AR) objects robustly in a marker-less AR system. It is known that one of the promising ways to develop a marker-less AR system is using human´s body such as hand or face for replacing fiducial markers. This paper presents a real-time method to manipulate the AR objects dynamically in a developed 3D hand coordinate system. To build the marker-less system, we utilize YCbCr skin-color model for hand shape detection and curvature-based fingertip detection from an input video images. Using the detected fingertips the camera pose are estimated to register virtual objects on the hand coordinate system. In order to manipulate the movements of the virtual objects rendered on the marker-less AR system dynamically, a vision-based hand control interface, which utilizes pattern recognition for the hand command initiation, is developed. From the experiments we can prove that the proposed hand mouse can manipulate the objects in a feasible fashion.