Toward a metrics suite for source code lexicons

In this paper we present an empirical study of relationships between three source code lexicons: the identifier, comment, and literal lexicons. We conjecture that shared and unique properties of these lexicons for the given subject system can inform the configuration of a source code retrieval technique for a particular software understanding activity or software evolution task. Thus, we seek to discover these lexicon properties, and so we investigate five lexicon measures that consider term frequency, term density, and term provenance.