Motion-adaptive sub-Nyquist sampling technique for multi-frame super-resolution

We have developed a motion-adaptive sub-Nyquist sampling technique to be used as a complement to super-resolution (SR) image reconstruction. These techniques are used together for pre- and post-processing with a conventional video codec (core codec). In this new video coding system, called “reconstructive video coding,” the sub-sampling process reduces the core codec´s compression ratio, thereby reducing such nonlinear distortion as block noise. On the decoder side, the decimated pixels are replenished with ones from neighboring frames through a multi-frame SR technique that compensates for inter-frame local motion. As a preliminary design study, we developed a systematic procedure (a sub-sampler) for a spatiotemporal allocation of sampling points to cover the objects in a scene as uniformly as possible by taking into account their motions. Testing of this procedure in combination with a multi-frame SR technique demonstrated that the proposed sub-Nyquist sampling technique is more effective than simple non-adaptive sub-sampling.