Vision-Guided "Point and Click" for Smart Rooms

Pointing is a natural method for humans to use to indicate a focus of their attention. Over the years, the ubiquitous, infra-red remote control has been used to control a variety of appliances by pointing. Despite its success, infra-red based pointing solutions have some limitations. In our research, we have explored other ways for mobile users to select objects inside a smart room via pointing or similar gesture. In this paper we describe the use of mobile phones with cameras as pointing devices to interact with smart environments. Mobile users select an object by simply taking a picture of it. We rely on two key ideas to simplify the image-processing problem: objects that can be recognized visually contain image-processing hints in their descriptions, and all the objects (and their descriptions) in the vicinity of the user can be found by a location-aware discovery protocol. The paper describes the design of a location-aware service discovery protocol, the integration of the service discovery protocol with image capture and analysis and our experience with a smart room application that uses image capture and recognition to select physical objects.