Guiding computational perception through a shared auditory space

Blind or visually impaired people want to know more about things they hear in the world. They want to know what other people can “see”. With its cameras, a robot can fill that role. But how can an individual make requests about arbitrary objects they can only hear? How can people make requests about objects they do not know either the exact location of, or any uniquely identifiable traits? This work describes a solution for querying the robot about unknown, audible objects in the surrounding environment through a combination of computational sound source localization and human input including pointing gestures and spoken descriptors.