Addressers´ gesture changes according to addressees´ interpretation of communicative intention

To construct mind-reading robots, it is important to correctly identify the communicative intention of humans. In human environments, there are usually an enormous number of objects, and each object has many different parts. Therefore, referring to an object or an object part in current communication is not an easy task. We use both linguistic and nonlinguistic information to specify such referential intentions, but exactly how nonlinguistic information is used has not been examined in detail. In this study, video data from eight pairs of adult addressers and addressees who communicated about whole-object or object-part labels were analyzed. The addressers´ utterances were transcribed, and nouns and demonstrative words were extracted. Hand motions were divided into five categories: showing, pointing, stroking, functional action, and other. Gaze data obtained using eye movement recordings were also coded using the following categories: gaze on object, face, or body of the experimenter and other. Gaze on object was further divided into two categories; critical and noncritical object parts. The results showed that participants used different patterns of hand motions and gaze to refer to whole-object and object-part labels. When they taught whole-object labels, they showed the object and looked at the addressee´s face. When they taught object-part labels, they pointed at, stroked, and looked at the object part. Based on the results, we propose the concept of action contrast in human communication of referential intention. We propose that the need for specification and the use of actions in a contrasting manner are related. Nonlinguistic cues, such as pointing and showing, and timing of utterances are important sources of information that specify human referential intentions in the environment. Action contrast is an important source of information specifying addressers´ referential intentions, which can be utilized to construct mind-reading robots.