Improving Viewer Training in Visual Assessment

European Workshop on Visual Information Processing, pp. 1–5

Abstract

Subjective assessment in the context of video compression performance evaluation measures the impairment or the quality of the observed video by the mean opinion score provided by naïve viewers. In a recent update of Recommendation ITU-R BT.500-14, this concept has been extended to expert viewers. Experts, who often have strong opinions on specific artifacts, provide wider confidence interval values than naïve viewers; this implies, therefore, a reduced ability to rank the results. This paper presents a new training procedure for viewers, aiming to reduce the spread of ratings either due to strong personal opinions (i.e., those most likely from expert viewers) or to hesitation in performing a new task (i.e., those most likely from naïve viewers). The new design of the training session includes suggesting of the expected range of scores, thereby helping the viewer to achieve a more consistent understanding of the scoring scale. In our experiment we apply the DSIS (Double Stimulus Impairment Scale) / DCR (Degradation Category Rating) test protocols. The same video clips were assessed by expert viewers both with and without the new training method, as well as by naïve viewers in a laboratory test. The analysis of the results shows a promising consistency in the results of the different tests.

Authors 2

1. Mathias Wien Aachen Lehrstuhl für Bildverarbeitung

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Lehrstuhl für Bildverarbeitung,Aachen,Germany

2. Vittorio Baroncini
   Affiliation as printed

   VABTech,London,UK

   VABTech, London, UK