Puppet show for entertaining hearing-impaired, together with normal-hearing people — A novel application of human sensing technology to inclusive education

The authors propose a novel application of human sensing technology to inclusive education. We focus on puppet show to aspire the inclusive education for both the normal hearing and the hearing impaired. Thus, we are developing Puppet Theater system to entertain both hearing impaired and normal hearing. This Puppet Theater system consists of two functions. First, the lines are displayed as letters with voice for complementing information of sound. The system enables the hearing impaired to understand the contents of the puppet show. Second function presents branches of the story to audience for encouraging the audience to understand the content of the story. In this paper, we describe that we develope branches of the story as function to select the branches by body movements for the purpose of additional effect of understanding contents of the story. The body movements are measured and detected by Kinect sensor for Xbox. We anticipate additional benefit of understanding contents of the story because this new function enables the audience to participate in the progression of the puppet show. The authors performed the system to elementary school pupils with hearing impairment and found that the pupils enjoyed it. Effects of the function were investigated with questionnaire.