A general approach for closed-loop registration in AR

Tracking and augmentation are usually handled in independent consecutive stages in augmented reality (AR). The result is that the real-virtual registration is “open loop”-inaccurate tracking leads to misregistration that is seen by the users but not the system. We propose a general approach to “close the loop” in the displayed appearance by using the visual feedback of registration for tracking. Specifically, a model-based method is introduced to simultaneously track and augment real objects in a closed-loop fashion, where the model is comprised of the combination of the real object to be tracked and the virtual object to be rendered. This method is applicable to paradigms including video-based AR, projector-based AR, and diminished reality.