Video Prediction Using Multi-Scale Deep Neural Networks

In video prediction, it is expected to predict the next frame of a video by providing a sequence of input frames. Whereas numerous studies exist that tackle frame prediction, a suitable performance is not still achieved, and therefore, the application is an open problem. In this work, multi-scale processing is studied for video prediction, and a new network architecture for multi-scale processing is presented. This architecture is in the broad family of autoencoders. It is comprised of an encoder and decoder. A pretrained VGG is used as an encoder that processes a pyramid of input frames at multiple scales simultaneously. The decoder is based on the 3D convolutional neurons. The presented architecture is studied using three different datasets with varying degrees of difficulty. In addition, the proposed approach is compared with two conventional autoencoders. It is observed that using the pretrained network and multi-scale processing results in a performant approach.