Combining probabilistic reference resolution with simulated vision

We describe a probabilistic reference disambiguation mechanism developed for a spoken dialogue system mounted on an autonomous robotic agent. Our mechanism processes referring expressions containing intrinsic features of objects (lexical item, colour and size) and locative expressions, which involve more than one concept. The intended objects are identified in the context of the output of a simulated scene analysis system, which returns the colour and size of the seen objects and a distribution for their type. The evaluation of our system shows high resolution performance across a range of spoken referring expressions and simulated vision accuracies.