Predicting object functionality using physical simulations

It is challenging for a robot acting in the world to interact with and use novel objects. While a person may be able to look past visual differences and recognize the intended function of an object, doing so is more difficult for robots, which tend to rely on visual similarity to recognize categories of objects. A robot that recognizes and classifies objects based on their functional properties and potential capabilities is better prepared to use unknown objects. We propose a technique for functionally classifying objects using features obtained through physical simulations. The described method simulates spheres falling onto an object from above. We show how a feature vector can be derived from the results of the physics-based simulation, and that this feature vector is informative for a variety of affordance classification tasks. This process allows a robot equipped with a 3D sensor to determine the functionality of objects in its environment given only a few training examples from various function classes. We show that this method is able to accurately learn membership of 3D models in three function classes: “drinking vessel”, “table”, and “sittable”. We then show that this can be extended to 3D scans of objects using the models as training examples.