Beauty is in the scale of the beholder: Comparison of methodologies for the subjective assessment of image aesthetic appeal

A first step towards creating automatic measures of image aesthetic appeal is understanding its appreciation via subjective testing. Nevertheless, reliably setting up such tests appears to be challenging, as aesthetic appeal is proven to be influenced by a number of subjective factors. In this paper we investigate four scale types for aesthetic appeal rating and assess their ability to provide general quantification of aesthetic appeal, as well as repeatable judgment across experiments. We asked 24 users to assess a representative image set constructed to uniformly cover a wide range of aesthetic appeal. Our experiments show that the Absolute Category Rating (ACR) 5-point scale provides the most consistent ratings across participants, which are also repeatable across different experiments.