Robust Real-Time Bi-Layer Video Segmentation Using Infrared Video

In this paper, we propose a novel method for the automatic segmentation of a foreground layer from a natural scene in real time by fusing infrared, color and edge information. This method improves on previous video foreground/background segmentation algorithms by making the system totally independent of changes in ambient lighting. A powerful data acquisition unit was developed using an optical technique that automatically gives synchronized and registered color video and infrared (IR) video at 850 nm. Using the fused information produced by the IR video one can then automatically initialize a pentamap, which is processed by the graph cuts algorithm. We show that the pentamap can simplify the graph construction process, improve the efficiency of the graph cuts algorithm, and allow to use a more reliable color Gaussian mixture model (GMM) than the usual trimap method. At the end of the processing pipeline, a simple border-blurring algorithm is used to simulate matting effects on the foreground-background boundary. Experimental results are presented.