Patterns of Multimodal Input Usage in Non-Visual Information Navigation

Multimodal input is known to be advantageous for graphical user interfaces, but its benefits for non-visual interaction are unknown. To explore this issue, an exploratory study was conducted with fourteen sighted subjects on a system that allows speech input and hand input on a touchpad. Findings include: (1) Users chose between these two input modalities based on the types of operations undertaken. Navigation operations were done primarily with touchpad input, while non-navigation instructions were carried out primarily using speech input. (2) Multimodal error correction was not prevalent. Repeating a failed operation until it succeeded and trying other methods in the same input modality were dominant error-correction strategies. (3) The modality learned first was not necessarily the primary modality used later, but a training order effect existed. These empirical results provide guidelines for designing non-visual multimodal input and create a comparison baseline for a subsequent study with blind users.