Robust multimodal understanding

Contemporary multimodal prototypes provide an excellent proof of concept but are not sufficiently robust in their handling of user input to be adopted by real users engaged in complex tasks. Our goal is to investigate techniques that improve the robustness of multimodal understanding to the point where effective multimodal interfaces can be made feasible. We present two different approaches - a pattern-matching based approach and a classification-based approach to robust multimodal understanding. We compare these approaches by evaluating them on data collected in the context of a multimodal conversational system.