Instructing cooperating assembly robots through situated dialogues in natural language

We present an assembly cell consisting of two cooperating robots and a variety of sensors. It offers a number of complex skills necessary for constructing aggregates from elements of a toy construction set. A high degree of flexibility was achieved because the skills were realised only through sensory feedback not by resorting to fixtures or specialised tools. The operation of the cell is completely controlled through natural language. Results from experiments in cognitive sciences and computer linguistics were incorporated to integrate natural language with vision as well as to control the construction dialogue between a human instructor and the robotic system. The experimental setup is described, and a sample dialogue demonstrates the capabilities of the cell. A brief discussion of issues for further research concludes the paper