Generating Verbal Descriptions of Colored Objects: Towards Grounding Language in Perception

This paper addresses the issue of how verbal communication arises from the complex and uncertain representations that seem necessary to robustly carry out perception in real-world domains. We propose that the generation of natural language in such domains should be addressed as the optimization problem of finding, under various constraints, the verbalization that has the greatest probability of achieving a specific change that the speaker wants to induce in the mental state or behavior of the listener. This most likely effective or MLE strategy has the advantage of making the problem concrete, and allowing (possibly empathic) models of the perceptual and behavioral processes to be used in a principled way. We illustrate these issues in the context of the specific problem of describing real objects in native domains using basic color language (e.g. "mostly brown", "partly red"). The term "native domains" refers to real-world environments that have not been tailored to suit the application