Robust Temporal-Spatial Decomposition and Its Applications in Video Processing

In this paper, we propose a robust temporal-spatial decomposition (RTSD) model and discuss its applications in video processing. A video sequence usually possesses high correlations among and within its frames. Fully exploiting the temporal and spatial correlations enables efficient processing and better understanding of the video sequence. Considering that the video sequence typically contains slowly changing background and rapidly changing foreground as well as noise, we propose to decompose the video frames into three parts: the temporal-spatially correlated part, the feature compensation part, and the sparse noise part. Accordingly, the decomposition problem can be formulated as the minimization of a convex function, which consists of a nuclear norm, a total variation (TV)-like norm, and an l₁ norm. Since the minimization is nontrivial to handle, we develop a two-stage strategy to solve this decomposition problem, and discuss different alternatives to fulfil each stage of decomposition. The RTSD model treats video frames as a unity from both the temporal and spatial point of view, and demonstrates robustness to noise and certain background variations. Experiments on video denoising and scratch detection applications verify the effectiveness of the proposed RTSD model and the developed algorithms.