High-fidelity visuo-haptic interaction with virtual objects in multi-modal AR systems

The driving force of our research is the precise combination of real and - possibly indistinguishable - virtual objects in an interactive augmented reality environment. This requires real-time, multimodal simulation, as well as stable and accurate overlay of the computer-generated objects. This paper describes several methods to improve accuracy and stability of our hybrid augmented reality system. In a comparison of two approaches to hybrid head pose refinement, we show that Quasi-Newton method enables high performance optimization for image space error minimization. Moreover, a 3D landmark refinement step is proposed, which significantly improves quality and robustness of the overlay process. The enhanced system is demonstrated in an interactive AR environment, which provides accurate haptic feedback from real and virtual deformable objects. Finally, the effect of landmark occlusion on tracking stability during user interaction is also analyzed.