Assessing the validity of subjective QoE data through rating times and self-reported confidence

When users give ratings in subjective Quality of Experience experiments, they may experience a lack of confidence in assigning a score to a stimulus. This results in invalid votes that might need to be removed from the acquired data. We show results from a subjective video quality experiment in which we measured the rating times and asked observers for their confidence, in addition to just the quality score. It is shown that the average confidence is higher for extremely good or bad MOS values. The results also demonstrate that collecting rating times is an effective way to detect issues with single votes caused by observers struggling to decide for a rating.