Resolving automated perception system failures in bin-picking tasks using assistance from remote human operators

We present an approach to resolve automated perception failures during bin-picking operations in hybrid assembly cells. Our model exploits complementary strengths of humans and robots. Whereas the robot performs bin-picking and proceeds to the subsequent operation like kitting or assembly, a remotely located human assists the robot in critical situations by resolving any automated perception problems encountered during bin-picking. We present the design details of our overall system comprising an automated part recognition system and a remote user interface that allows effective information exchange between the human and the robot that is geared toward solutions that minimize human operator time in resolving the detected perception failures. We use illustrative real robot experiments to show that human-robot information exchange leads to improved bin-picking performance.