On the Coherence between Comments and Implementations in Source Code

Source code comments provide useful information on the implementation of a software and on the intent behind design decisions and goals. Writing informative and useful comments is far from being a trivial task. Moreover, source code comments tend to remain mostly unchanged during maintenance activities. As a consequence, the information provided in the comment of a method and in its corresponding implementation may be not coherent with each other (i.e., The comment does not properly describe the implementation). In this paper, we present the results of a manual assessment on the coherence between comments and implementations of 3636 methods, gathered from 3 Java open source software systems (for one of these systems, we considered 2 different subsequent versions). Resulting evaluations have been collected in a dataset, we made publicly available on the web. The defined protocol used for the creation of this dataset is also described. This lets researchers evaluate the goodness of our dataset and eases its future possible extensions. Another contribution of our paper consists in investigating on a possible link between coherence and lexical similarity between source code and comments. Our preliminary outcomes suggest that this similarity is higher in case the comment of methods and their implementations are coherent. However, the obtained similarity values are generally low and are not much higher than those for non-coherent method implementations and comments.