Semantic segmentation applied to image interpolation in the case of camera panning and zooming

An algorithm for image interpolation that takes into account both the motion of the camera and the motion of the imaged objects is described. First the estimated global motion parameters are used to compensate for the camera motion. The missing images are then interpolated by using the local motion field. Finally, these images are reconstructed in their correct dimension, considering the global motion information. Experiments have been carried out to extend the concept of object-background segmentation, developed for fixed camera sequences, to the case of camera zoom or pan. Simulations show that in order to obtain an effective semantic segmentation it is necessary to use a very accurate model of the camera lens system. The estimation of the global motion parameters is shown to be very accurate. For the interpolated images, an average gain of 4-5 dB with respect to the case of standard motion-compensated interpolation has been obtained.<>