The value of architecturally significant information extracted from patterns for architecture evaluation: a controlled experiment

We have developed an approach to identify and capture architecturally significant information from patterns (ASIP), which can be used to improve architecture design and evaluation. Our experimental goal was to evaluate whether the use of the ASIP improves the quality of scenarios developed to evaluate software architecture. Out of 24 subjects 21 were experienced software engineers who had returned to University for a postgraduate studies and remaining 3 were fourth year undergraduate students. All participants were taking a course in software architecture. The participants were randomly assigned to two groups of equal size. Both groups developed scenarios for architecture evaluation. One group (treatment group) was given ASIP information the other (control group) was not. The outcome variable was the quality of the scenarios produced by each participant working individually. The treatment group participants also completed a post-experiment questionnaire. Our results support the hypothesis that ASIP information assists scenario development in the context of architecture evaluation.