A direct visual-inertial sensor fusion approach in multi-state constraint Kalman filter

Pose estimation only using a monocular camera and an inertial sensor triggers increasing popularity in recent years. In this paper, we propose a method, that tightly combines direct image information (intensity and gradient) from monocular camera with inertial information from three-axis gyroscope and accelerometer in a multi-state constraint Kalman filter (MSCKF) based framework to perform an effective pose estimation. In contrast to other pose estimation methods using vision, our solution gets rid of traditional feature extraction and expression, instead using image patches with distinct gradient to represent a visual measurement from environment. We adopt sequential inertial information and the poses between two consecutive keyframes to construct the state vector, imposing constraints on the poses and marginalising out expired ones, which would reduce the computational complexity linear to the number of selected patches. Furthermore, we view the data from the inertial sensor as intrinsic information applied in filter propagation, providing fast rate estimation for the state. The result of our method has been tested on real flying data of a micro aerial vehicle in indoor and outdoor environments.