Application of empirical methodology to evaluate information fusion approaches

The ETURWG subgroup defined a set of criteria structured in an ontology (URREF = Uncertainty Representation and Reasoning Evaluation Framework) to evaluate techniques for assessing, managing, and reducing uncertainty. Different techniques will be assessed by the members of the group by applying the fusion techniques to predefined use cases using available test data sets. We propose that the application of empirical methodology which has e.g. psychological background to evaluate the fusion techniques will help to improve the quality of evaluation results and give the chance to pose and evaluate additional questions. It will also help to be aware about the chosen evaluation design and its resulting advantages and shortcomings. We give a short overview about some of the manifold possible empirical methodologies to define the evaluating design for the assessment of fusion techniques. We selected one exemplary useful methodology of the presented methodologies to explicate this application. We describe the application of the selected approach to evaluation of fusion techniques and draft the resulting experimental design. The possible improvements and extensions of evaluation results are presented. The paper gives a first idea of the possibilities of these methodologies. Results of experiments already performed and complete description of experiment settings are not in its scope. We focus on the discussion and the proposed experimental settings to the URREF criteria set.