MirrorTrack - a vision based multi-touch system for glossy display surfaces

This paper presents a multi-touch system using vision-based algorithms that takes advantage of specular surfaces to detect and track finger locations with multiple cameras. The key innovation of the MirrorTrack approach is that by mounting cameras at a low azimuth to the horizontal display surface, the screen approaches a perfect mirror. Hence, we are able to track the user’s fingers together with its mirror reflection in the screen. The advantages of our system are that: 1. It works with any glossy flat panel display; 2. It avoids clumsy setups that require suspension of down-looking cameras with the concomitant screen glare problems; 3. It supports both touch and hover operations; 4. It can be extended to track objects other than fingers; and, 5. It can work with large vertical displays without the usual occlusion problems. We describe our algorithm and implementation in details and present a set of finger gestures interactions.