Revealing the dark side of a subjective study: Learnings from noise and sharpness ratings

Significant improvement in no-reference image quality assessment (NR-IQA) methods has been demonstrated in recent years. The demonstrated prediction performance of proposed NR-IQA methods, in terms of correlations between predicted and subjective scores, has reached similar performance levels with full-reference image quality assessment (FR-IQA) measures, on popular image datasets. However, in our work we found that these correlations drop significantly when NR-IQA measures are applied on images that represent real consumer-capture scenarios. This is due to the fact that the datasets used thus far for NR-IQA research are not representative of these consumer scenarios. In an attempt to tackle this issue, we created a set of consumer photos and conducted a subjective experiment. In this paper we describe the subjective experiment, which asked participants for subjective ratings for sharpness, noise, and overall image quality, but yielded counterintuitive subjective ratings for noise due to the complexity of the interaction between sharpness, noise, and perceived quality in consumer content.