Of Massive Static Analysis Data

The Software Assurance Metrics and Tool Evaluation (SAMATE) project at the National Institute of Standards and Technology (NIST) has organized four Static Analysis Tool Expositions (SATE). SATE is designed to advance research in static analysis tools that find security-relevant defects in source code. Briefly, participating tool makers run their tools on a set of programs. Researchers led by NIST analyze the tool outputs. The results and experiences are reported at a workshop. These expositions have accumulated large amounts of data. This collection allowed for the development and validation of practical metrics in regard to static analysis tool effectiveness and independence. In this paper, we discuss the role of the data in determining which metrics can be derived. Specifically, we detail the three characteristics test data should exhibit and explain why the data we use express each combination of two out of these three properties.