Task-based subject validation: Reliability metrics

The quality obtained from a CCTV or a telemedicine system has to be good enough to correctly and with high probability recognize a face or a medical pathology. In order to measure “good enough” term subjective experiments are used. One of the problems related to subjective experiment data analysis is the subjects´ reliability or, more precisely, each individual subject´s reliability. In this paper two metrics which can be used to detect unreliable subjects in case of a task-based subjective experiment are presented. A task-based subjective experiment is an experiment in which a subject performs a task (recognizes a person, characters, pathology, etc.). In this paper metrics enabling measurement of a subject´s reliability are referred to as reliability metrics. Reliability metrics should be considered as a part of a standard of a task based subjective experiment like ITU P.912 [1].