Evaluation of a speech translation system for travel conversation installed in PDA

For mobile use, we are developing a multilingual example-sentence-driven speech translation system that has a multi-modal input interface to retrieve the sentence. In addition to the basic speech input mode, we apply an associative keyword mode with a software keyboard and an associative domain selection mode. The paper discusses the characteristics of each input mode, the synergistic effect obtained by combining the modes and the results of evaluations that show the difference between system performance in the laboratory and in the real world. As the evaluation criteria, we adopted the retrieval time and the retrieval precision of the sentence. When all of the modes were available, the precision within 30 seconds was 86.8% for a closed test set and 76.8% for an open-test set. When the retrieval was completed with only one operation, the average time was 10.3 seconds for a closed set. The precision was 12.0% higher than the maximum precision obtained when only one of the modes was available. The results show that synergetic effect of the combined modes certainly exists and all the modes are necessary to improve the system´s usability.