High speed depth estimation by smart image sensor system

This work proposes a novel smart image sensor system for depth estimation by searching for the most in-focus position over multi-focus images (a sequence of images taken by changing a camera parameter "focus" little by little). On the smart sensor, in-focus position is detected pixel by pixel and all-in-focus image can be output in real time. A prototype sensor of 64×64 pixels has been designed and fabricated. It can be expected that the smart imaging system is able to generate depth map and all-in-focus images at video rate (30 Hz) or faster because the proposed sensor can operate at more than 1500 fps. This paper also proposes the method of removal and interpolation for depth map to improve accuracy of the depth map. Some experimental results are shown to verify the proposed method of the in-focus detection and the interpolation, and we show design of the proposed sensor.