Comparing and Validating Measurements of Dependability Attributes

This paper investigates sources of uncertainty in measurement results obtained using three different fault injection techniques. Two software-implemented and one test port-based technique are characterized and compared. The three techniques can be used to inject the same faults, which are defined in a shared database. Due to the uncertainties associated with the techniques, which we identify and discuss, the results of injecting a given fault may differ to some extent. The paper analyzes the results of using the three techniques to inject faults into two experimental targets: a brake-by-wire controller and a partitioning operating system. The objective of the experiments is to determine whether the results of the different techniques are metrologically compatible and, consequently, meaningful when disseminated and compared. Our observations indicate that, even though the outcome of many individual experiments is affected by uncertainties, the three techniques produce similar average results over a large number of experiments.