Using security metrics coupled with predictive modeling and simulation to assess security processes

It is hard for security practitioners and decision-makers to know what level of protection they are getting from their investments in security, especially when they have invested in a number of technologies and processes which interact and combine together. It is even harder to estimate how well these investments can be expected to protect their organizations in the future as security policies, regulations and the threat environment are constantly changing. In this paper we propose that for measuring the effectiveness of security processes in large organizations, a greater emphasis needs to be put on process-based metrics, in contrast to the more commonly used symptomatic lagging indicators. We show, by means of two case studies, how these process-based metrics can be combined with executable, predictive models, based on a sound mathematical foundation, to both assess organizations\´ security processes under current conditions and predict how well they are likely to perform in potential future scenarios, which may include changes in working practices, policies or threat levels, or new investments in security. We present two case studies, in the areas of vulnerability threat management, and identity and access management, as significant examples to illustrate how this modeling and simulation-based approach can be used to provide a rich picture of how well existing security processes are protecting the organization and to answer "what-if" questions, such as exploring the effects of a change in security policy or an investment in new security technology. Our approach enables the organization to apply the metrics that are most relevant to its business, and provide a comprehensive view that shows the benefits and losses to the different stakeholders.