On the difficulty of improving hand-crafted rules in chat-oriented dialogue systems

In this work, we explore the difficulty of improving hand-crafted rules in chat-oriented dialogue systems. We first created an initial rule set in artificial intelligence markup language and revised it through an iterative cycle. Then, we tested the initial and revised rule sets by using human participants. The dialogue experiment showed that, despite the intensive revision process, the overall performance had little improvement. We investigated the errors made by the systems with each of the two rule sets, leading to the conclusion that it is the tracking of the context and the coverage of questions that are hindering the improvement of hand-crafted rules.