UCBench: A user-centric benchmark suite for C code static analyzers

In recent years, many C code static analyzers, with different abilities of bug detection, have appeared and been applied in various domains. There are so many choices that it becomes hard for programmers to know in detail the strengths as well as limitations of all these analyzers and to find the most suitable ones for their code. In this paper, we propose a benchmark for C code static analyzers, named UCBench, to provide quantitative and qualitative measurements for evaluating analyzers. Being different from other benchmarks, UCBench concentrates more on users´ requirements rather than the improvements of bug detecting technique itself. The major components of UCBench include test case database, evaluation metrics and harness. We classify test cases into several groups according to their attributes and design various user-centric evaluation metrics. Besides, we develop some harness to automate the evaluation process. Finally, we demonstrate our benchmark suite over four C code static analyzers: Flawfinder, Cppcheck, Uno and Splint.