Sensor fusion approaches for observation of user actions in programming by demonstration

Good observation of a manipulation presentation performed by a human teacher is crucial to further processing steps in programming by demonstration which is of prime importance in interactive robot programming. This paper outlines a sensor fusion concept for hand action tracking by observing the hand posture, position and applied forces. The input sources include: a data glove which classifies several gestures and grasps, a stereo camera mounted head and several force sensors fitted on the finger tips. The hardware used is presented as well as the first implementation of measurement and fusion approaches. Accuracies of the experiment are also given.