Multimodal User Guide for Experience Sharing Based on Activity Modeling

This paper presents a multimodal user guide system, comprised of wearable sensors, which provides interactive feedback to users for the purpose of sharing experiences. The system we propose captures and analyzes contextual information using wearable sensors, while a user is performing an activity. The information can subsequently be shared with other users and used to guide users in performing similar activities. Audiovisual feedback from the guiding system is provided to the users, in real time, in accordance with their activity state. To realize the system, we propose a probabilistic method for modeling of user experiences, and a multimodal feedback process based on the previous experience model. By applying the system to a short tour experience, we evaluated the feasibility and suggested uncovered potential future directions in regards to our proposed system.