Hand tracking for interactive pattern-based augmented reality

Pattern-based augmented reality systems are considered the most promising approach for accurately registering virtual objects with real-time video feeds. The problem with existing solutions is the lack of robustness to partial occlusions of the pattern, which is important when attempting natural interactions with virtual objects. This paper describes a fast and accurate vision-based pattern tracking system that allows for autocalibrated 3D augmentation of virtual objects onto known planar patterns. The tracking system is shown to be robust to changes in pattern scale, orientation and, most importantly, partial occlusions. A method to detect a hand on top of the pattern is then described, along with a method to render the hand on top of the virtual objects.