Measuring the Accuracy of Information Retrieval Based Bug Localization Techniques

Bug localization involves using information about a bug to locate affected code sections. Several automated bug localization techniques based on information retrieval (IR) models have been constructed recently. The "gold standard" of measuring an IR technique\´s accuracy considers the technique\´s ability to locate a "first relevant method." However, the question remains -- does finding this single method enable the location of a complete set of affected methods? Previous arguments assume this to be true, however, few analyses of this assumption have been performed. In this paper, we perform a case study to test the reliability of this "gold standard" assumption. To further measure IR accuracy in the context of bug localization, we analyze the relevance of the IR model\´s "first method returned." We use various structural analysis techniques to extend relevant methods located by IR techniques and determine accuracy and reliability of these assumptions.