Off to see the Wizard: using a "Wizard of Oz" study to learn how to design a spoken language interface for programming

We are in the early stages of developing a spoken language interface that will help beginners write programs. Our goal is a system in which a student will talk to a computer using English sentences, in response to which the computer will generate syntactically correct Java source code. We believe that such a system would help beginning students by allowing them to focus on concepts instead of syntactic details, and that it would also be a boon to students with visual or mobility impairments. As a prelude to designing and implementing such a system, we evaluated the concept via a Wizard of Oz study. Volunteer subjects were told that they were helping us evaluate a working system. In reality, an accomplished programmer was playing the role of the purported system, and we were studying how the subjects interacted with it. We describe the system that we envision, discuss the process of running a Wizard of Oz study in the context of our own recently completed study, and summarize our preliminary results.