Redefining reliability evaluations for software-intensive systems

The Department of Defense has increased the use of Major Automated Information Systems (MAIS); in particular Enterprise Resource Planning (ERP) systems seek to simplify business operations, optimize processes, and maximize efficient financial and logistics management capabilities. These automated information systems integrate all departments and functions across an organization into a single, integrated, centralized database software suite which serves all the internal departments´ needs. Due to their unique characteristics and non-traditional programmatic structures, these system´s often benefit from the evaluation of uniquely defined metrics than what is typically used for hardware centric systems. Often MAIS reliability requirements are at such a high level (essentially accounting only for the combined experience of all users simultaneously) that they fail to fully capture the experience of the individual end user. While system level Reliability, Availability, and Maintainability (RAM) requirements are necessary, there is a need to focus the evaluation on supplementary metrics that address end user level failures. This paper will discuss innovative approaches to defining reliability evaluations for MAIS programs. This includes understanding how the reliability requirements are defined and what they intend to measure, appropriate use of metrics such as server reliability versus end user experience, and the utilization of additional metrics to provide a meaningful software reliability evaluation to decision makers.