Automatic Mobile Robotic Manipulation with Active Eye-to-Hand Binocular Vision

In this paper, a mobile robotic manipulation system equipped with an active eye-to-hand binocular vision system is proposed. The active vision system is able to observe arm gesture of a user by visually recognizing features on the arm and reconstructing the arm pose in Cartesian space. The target that the user wants to point at can thus be identified based on the arm pose. The user is then prompted with the image of the target on screen for confirmation by hand gesture. If confirmed, a vision- based control law is able to drive the robot toward the target position and to fetch the target by the robotic arm. Similarly, the target can be delivered to the user by real-time visual servoing. In fact, the proposed system can successfully interact with users by using binocular vision. Specifically, this robotic system could effectively help users to fetch targets that users designate by simple arm gesture.