An autonomous mobile grasping system using visual servoing and nonlinear model predictive control

This paper presents an autonomous grasping system using visual servoing and mobile robots. While this kind of system has many potential significant applications, there have been several key challenges, for example, localization accuracy, visibility and velocity constraints, obstacle avoidance, and so on, to prevent the implementation of such a system. The main contribution of this paper is to develop an adaptive nonlinear model predictive controller (NMPC) to meet all these challenges in one single controller. In particular, the model of the vision-based mobile grasping system is first derived. Then, based on the model, a nonlinear predictive control strategy with vision feedback is proposed to deal with the issues of optimal control and constraints simultaneously. Different from other work in this field, in order to improve the performance, an adaptive mechanism is proposed in the paper to update the model online so that it can track the nonlinear time-varying plant in a real time manner. To the best of our knowledge, this is the first work to apply model predictive control to mobile visual servoing and consider various constraints at the same time. The approach was validated with two physical experiments. It was shown that the system with the new control strategy was quite successful to carry out an autonomous mobile grasping task in a complex environment.