Paired comparison for subjective multimedia quality assessment: Theory and practice

For subjective multimedia quality assessment, paired comparison has been used less frequently than other methodologies such as single stimulus and double stimulus methodologies, mainly due to its increased time complexity. However, it has been shown that paired comparison provides improved discriminability between stimuli, which consequently improves reliability of the test results. Analyzing comparison results involves several challenges in quality score computation, outlier detection, etc. This paper presents theoretical backgrounds on these challenges and use cases where paired comparison are successfully applied. In particular, the issue of the increased time complexity is discussed along with recently proposed solutions.