A Possible Pitfall in the Experimental Analysis of Tampering Detection Algorithms

This paper aims to give a contribute to the experimental evaluation of tampered image detection algorithms (i.e. Image Integrity algorithms), by describing a possible way to improve these experimentations with respect to the traditional approaches followed in this area. In particular, the paper focuses on the problem of choosing a proper test dataset allowing to keep low the bias on the experimental performance of these kind of algorithms. The paper first describes a JPEG image integrity algorithm, the Lin et al. algorithm, that has been used as benchmark during our experiments. Then, the experimental performance of this algorithm are presented and discussed. These performance have been measured by running it on the CASIA TIDE public dataset, which represents the de facto standard for the experimental evaluation of image integrity algorithms. The considered algorithm apparently performs very well on this dataset. However, a closer analysis reveals the existence of some statistical artifacts in the dataset that improve the performance of the algorithm. In order to confirm this observation, we assembled an alternative dataset. This new dataset has been conceived to not exhibit the statistical artifacts existing in the images of the CASIA TIDE dataset, while producing an uniform distribution of some physical image features such as the quality factor. Then, we repeated the same experiments conducted on the CASIA TIDE dataset, using this new dataset. As expected, we observed a performance degradation of the Lin et al. algorithm, thus confirming our hypotheses about the CASIA TIDE dataset being, in some way, flawed.