Real-time hands-free immersive image navigation system using Microsoft Kinect 2.0 and Leap Motion Controller

This paper presents a real-time hands-free immersive image navigation system that can respond to various gestures and voice commands. We combine Microsoft Kinect 2.0 and Leap Motion Controller, on a single platform, to define a wide range of hand and finger gestures, and voice commands. Using these we create a robust and natural immersive image navigation experience. At present, the immersive environment is simulated using an 1×3 array of 17 inch Dell monitors. For gesture recognition we use three different algorithms, depending on the nature of gesture and the sensor used to capture it. We use Microsoft Speech API to integrate voice commands to the system. Experiments are conducted with 5 subjects each performing 10 gestures each. The system achieves a user accuracy ranging from 94% to 100% for different hands-free controls. We are currently integrating other sensors including an Eye-Gaze Tracker and extending to 2×4 array of monitors for better immersive experience.