Empirical Measurement of Automated Recovery of Design Decisions and Structure

Several notations and techniques have been proposed to capture and represent software architecture decisions and structures. Previous work has suggested that using automated techniques to recover decisions and structures from whichever project artifacts are actually kept may be a reasonable solution, but empirical evaluations of this claim are scarce. This article describes: (1) the TREx design decision and architecture structure recovery technique and tool, (2) precision and recall measures of recovery effectiveness for system structure, and (3) an experimental design to evaluate design decision and structure recovery approaches. An initial experimental evaluation of manual versus tool-supported TREx shows that using the tool increases recall, but yield mixed results on precision: decrease structure recovery precision, but increase decision recovery. Also anecdotic evidence suggest that only Experts were systematically better in recovering decisions without tool-support. A replicable experimental package has been published.