System-on-Chip Solution of Video Stabilization for CMOS Image Sensors in Hand-Held Devices

Majority of CMOS image sensors in consumer market utilize a rolling shutter to increase sensitivity. However, it causes severe distortions, such as jitter, wobble, or skew. Since most of these kinds of sensors are used in hand-held devices, the approach of undistorting and generating stabilized images is restricted to resource limited systems. It has also been one of the major challenges to have a mathematical representation of CMOS rolling effect depicting the practical scenario, while keeping accuracy and stability. We propose that a CMOS sensor can be modeled by a section-wise charge-coupled devices model which has multiple homographies and exploit the observation that rolling shutter mechanism gives close relationships among them. We present a CMOS seven-parameter model, and show video stabilization algorithm by the iterative parameter estimation technique. We address four issues while accelerating our stabilization algorithm within resource limited environment: accuracy, stability, computation time, and resource utilization. We developed cache based optimization techniques to meet the requirement of the memory bandwidth and computational time for the iterative parameter estimation and final output image interpolation, and also proposed the incremental form of the seven-parameter model to greatly reduce resource consumption while maintaining the same results as the previous. The validity and effectiveness of our approach is demonstrated by experiments for different types of camera motions. The cache based optimization technique can be used to accelerate other types of iterative vision algorithms that require repetitive memory access: feature tracking, motion estimation, motion compensation, various types of image distortion correction, and also image warping and scaling.