The Operator or the Automation: In Whom Should We Place Ultimate Trust?

This paper investigates the conflict between two competing safety philosophies. According to one philosophy, dangerous system states should be explicitly prevented through system design. According to the alternate philosophy, human operators should be the final arbiters of safety. These principles come into conflict in situations where any attempt to prevent the system entering an unsafe state may limit the freedom of the operator to recover from a dangerous situation not envisaged by the system designers. We argue that conventional hazard analysis and mitigation techniques have limited power to resolve this dilemma. We present a framework for making decisions about where final authority in a system should lie, and explore the implications of this framework.