Enhancement for temporal resolution of video based on multi-frame feature trajectory and occlusion compensation

This paper presents a bidirectional motion estimation (ME) method based on tracking feature trajectories and compensating for occlusion to enhance the temporal resolution of an input video sequence. First, we extract features and estimate their trajectories in the forward direction. We continue to track features only if the reliability of their trajectory is sufficiently high. Accordingly, if a feature can be continuously tracked through multiple frames, the proposed method assumes that its trajectory is a true motion vector (MV). Then, these forward feature trajectories are used as reference motion directions for backward block-based ME. If a block does not include any feature trajectories, we use the MVs of neighboring blocks as candidate MVs to propagate the predetermined true MV and preserve the spatial correlation of MVs. Further, the proposed method can detect occlusion regions, where a continuously tracked feature does not have a corresponding point in the current frame. In the detected occlusion region, the intermediate frame is generated from either the previous frame or the current frame. Outside the occlusion region, we generate the intermediate frames by taking into consideration the neighboring MVs to reduce blocking artifacts. Experimental results show that the proposed temporal frame interpolation (TFI) method can improve visual quality over conventional TFI methods, both objectively and subjectively.