Vision-based visual/haptic registration for WYSIWYF display

We have been working on developing a visual/haptic interface for virtual environments. The authors have previously (1996) proposed a WYSIWYF (What You See Is What You Feel) concept which ensures a correct visual/haptic registration so that what the user can see via a visual interface is consistent with what he/she can feel through a haptic interface. The key components of the WYSIWYF display are (i) vision-based tracking, (ii) video keying, and (iii) physically-based simulation. The first prototype has been built and the proposed concept was demonstrated. It turned out, however, that the original system had a bottleneck in the vision tracking component and the performance was not satisfactory (slow frame rate and large latency). To solve the problem of our first prototype, we have implemented a fast tracker which can track more than 100 markers in video-rate. In this paper, new experimental results are shown followed by the improvements of the vision-based tracking component