Generating Multimodal Grammars for Multimodal Dialogue Processing

This paper presents a new multimodal grammar generation system (MGGS) that allows defining a multimodal grammar in a very easy and intuitive way, overcoming the difficulties arising from the textual description of grammar production rules. The novelty of the proposed approach relies in adopting a “by example” paradigm to define a multimodal grammar. This paradigm consists in providing concrete examples of multimodal sentences, and enabling a grammar inference algorithm to automatically generate the grammar rules to parse those sentences. The main contribution of this approach is that it is general enough to be applicable for whatever modalities and in whichever domains. Moreover, the use of the grammar inference algorithm for automating the grammar writing and updating processes, reduces costs of the grammar development and maintenance. In the first part of this paper, we present the MGGS, describing in detail the methodology we have implemented in this system. More specifically, the multimodal attribute grammar and the grammar inference algorithm are illustrated. The second part describes the experiment aimed at observing participants while interacting with the system in order to provide some real data about the usability of the system. Results of this experiment showed that the proposed system facilitates the grammar definition and updating, and it is more suitable also for nonexpert people as it does not require the learning of the grammar notation.