A Multi-view camera-projector system for object detection and robot-human feedback

In this paper, we present a novel camera-projector system for assisting robot-human interaction. The system is comprised of a stereo camera pair and a DLP projector. The proposed system provides feedback information indicating the robot´s perception of the environment and what action a human user desires. Feedback is delivered by iteratively spotlighting objects in the environment using the projector. When the spotlight lands on an object that the human user wants the robot to retrieve, he or she can confirm the object selection, and the robot will perform a grasping task to retrieve the selected object. In this investigation, the proposed camera-projector performs three tasks: 1) Actively detect visually salient objects in the scene from the two camera views using a visual attention model. 2) Locate the detected objects using a trifocal tensor based object matching method, and project a spot of light on the objects to provide information back to the human user. 3) Reconstruct the 3D position of the target to plan robot motion and conduct vision-based control to move a robot manipulator to grasp the object. Experimental results of human-directed object grasping task are presented to demonstrate the functions of the proposed system.