Empirical results from an experiment on value-based review (VBR) processes

As part of our research on value-based software engineering, we conducted an experiment on the use of value-based review (VBR) processes. We developed a set of VBR checklists with issues ranked by success-criticality, and a set of VBR processes prioritized by issue criticality and stakeholder-negotiated product capability priorities. The experiment involved 28 independent verification and validation (IV&V) subjects (full-time working professionals taking a distance learning course) reviewing specifications produced by 18 real-client, full-time student e-services projects. The IV&V subjects were randomly assigned to use either the VBR approach or our previous value-neutral checklist-based reading (CBR) approach. The difference between groups was not statistically significant for number of issues reported, but was statistically significant for number of issues per review hour, total issue impact, and cost effectiveness in terms of total issue impact per review hour. For the latter, the VBRs were roughly twice as cost-effective as the CBRs.