A Decision--Theoretic Assistant Based on Gesture Recognition

This paper presents a new approach that combines computer vision and decision theory for an automatic human assistant. The setting is a washstand where a person is trying to complete the activity "cleaning the hands", guided by auditive instructions. To perform this activity the user interacts with surrounding objects. We assume that each step of the activity can be recognized based on previous and current hand gestures, and their interaction with the objects in the environment. The proposed approach combines context-based gesture recognition with a decision theoretic model to select the most adequate message. Gesture recognition is based on hidden Markov models, combining motion and contextual information, where the context refers to the relation of the position of the hand with other objects. The posterior probability of each gesture is used in a partially observable Markov decision process (POMDP) to select the best auditive instruction according to a utility function. The POMDP is implemented as a dynamic Bayesian network with certain lookahead. Preliminary tests on the system based on a comparison with a human assistant show promising results.