Robotic manipulation in object composition space

Manipulating unknown objects in a cluttered environment is difficult because object composition is uncertain. Because of this uncertainty, earlier work has concentrated on finding the “best” object composition and based on this composition decided on manipulation actions. Contrary to earlier work, we 1) utilize different possible object compositions in decision making, 2) take advantage of object composition information provided by robot actions, 3) take into account the effect of different competing object hypothesis on the actual task to be performed. We cast the manipulation planning problem as a partially observable Markov decision process (POMDP) which plans over possible hypotheses of object compositions. The POMDP model chooses the action that maximizes the long-term expected task specific utility, and while doing so, considers the value of informative actions and the effect of different object hypotheses on the completion of the task. In experiments with a physical robot arm and an RGB-D sensor, our approach outperforms an approach that only considers the most likely object composition.