Language grounding model: Connecting utterances and visual attributions

The job of language grounding is to research the relationship between language and external physical stimuli. In this paper, we build a language grounding model which is an extension of hidden Markov model. In a show-and-tell experiment, we use this model to learn the words meaning and simple bi-gram syntax, and finally generate the natural language description of special 2-D scenes automatically. The experiment results show the validity of our model in words categorization, semantic learning and phrase generation.