Understanding Rules in Human-Robot Instructions

This paper presents an overview of the systematic creation of a human-robot instruction system from a multi-modal corpus. The corpus has been collected from human-to-human card game instructions. A design procedure is introduced that helps creating a speech recognition grammar which is closely linked to semantics and the corpus, so avoiding unwanted over-generation. Particular attention is paid to rule-instructions, since they are more challenging to implement than sequential and knowledge manipulating instructions. A brief overview is given on how the robot stores knowledge coming from instructions using an ontological object-oriented form. Furthermore a problem-solver is described that can reason with the newly gained knowledge. The aim of the work is to enable users to naturally instruct robots without prior knowledge about the robot. A further aim is to simplify and expedite the process of implementing multi-modal human robot instruction svstems by engineers.