Decoder-Side Super-Resolution and Frame Interpolation for Improved H.264 Video Coding

In literature decoder-side motion estimation is shown to improve video coding efficiency of both H.264 and HEVC standards. In this paper we introduce enhanced skip and direct modes for H.264 coding using decoder-side super-resolution (SR) and frame interpolation. P- and B-frames are down sampled and H.264 encoded at lower resolution (LR). Then reconstructed LR frames are super-resolved using decoder-side motion estimation. Alternatively for B-frames, bidirectional true motion estimation is performed to synthesize a B-frame from its reference frames. For P-frames, bicubic interpolation of the LR frame is used as an alternative to SR reconstruction. A rate-distortion optimal mode selection algorithm determines for each MB which of the two reconstructions to use as skip/direct mode prediction. Simulations indicate an average of 1.04 dB PSNR improvement or 23.0% bit rate reduction at low bit rates when compared to H.264 standard. Average PSNR gains reach as high as 3.95 dB depending on the video content and frame rate.