An efficient error-minimizing algorithm for variable-rate temporal video sampling

We provide a novel algorithm for selecting key frames from a video at all possible temporal sampling densities. We define a measure for video reconstruction error (VRE), and use VRE to evaluate the representational quality of the selected key frames. The algorithm uses a heap-based greedy algorithm to build a hierarchy of increasingly sparsely sampled temporal sequences, each consisting of key frames with minimal VRE. By exploiting the heap and a novel computation technique of "forward computing", the complete hierarchy is constructed in O(nlog(n)) time and O(n) space. The algorithm also ranks all frames according to their importance in recovering the original video, potentially useful for applications in temporally scalable video coding and video streaming. Experiments show that our algorithm outperforms other existing methods in VRE, as compared via peak signal-noise ratio, in computation time and in guaranteed convergence.