Stereo vision and IMU based real-time ego-motion and depth image computation on a handheld device

Real-time environmental depth perception and ego-motion estimation is essential for all mobile robotic systems. We present a system that computes high quality depth images with 0.5 MPixel resolution using Semi-Global Matching (SGM) and estimates the ego-motion by key frame based visual odometry fused with the data of an inertial measurement unit (IMU). The hardware includes a pair of cameras, a small Intel Core2Duo CPU board, a Spartan 6 FPGA board, an OMAP3530 ARM processor board as well as an IMU. The total weight of the experimental setup is 830 g and is, thus, also feasible for hand-held or flying platforms. Experiments show that the vision system runs at 14.6 Hz with a latency of around 250 ms and produces high quality depth images as well as reliable 6D ego-motion estimates. In the fusion algorithm of visual odometry and IMU data, time delays of the vision system are compensated and a system state estimate is available at the full data rate of the IMU which is important for system control. This paper presents the integration of different techniques into a fast, light weight, real-time system and validates its performance by experiments on real data.