Real-time Volumetric Reconstruction and Tracking of Hands and Face as a User Interface for Virtual Environments

Enhancing desk-based computer environments with virtual reality technology requires natural interaction support, in particular head and hand tracking. Todays motion capture systems instrument users with intrusive hardware like optical markers or data gloves which limit the perceived realism of interactions with a virtual environment and constrain the free moving space of operators. Our work therefore focuses on the development of fault-tolerant techniques for fast, non-contact 3D hand motion capture, targeted to the application in standard office environments. This paper presents a table-top setup which utilizes vision based volumetric reconstruction and tracking of skin colored objects to integrate the users hands and face into virtual environments. The system is based on off-the-shelf hardware components and satisfies real-time constraints.