Scalability improvement in software evaluation methodologies

Evaluation of software is critical in a world that increasingly relies on software. Several software evaluation methodologies have been developed, but as software solutions increases in number and size, many of them do not scale. Improving scalability of software evaluation methodologies is a challenge and failing to reach a reasonable scalability level likely constrains the adoption of an evaluation methodology. In this paper, a framework for improving scalability of software evaluation methodologies is proposed. The proposed framework relies on three keystones: categorization of evaluation criteria, dependency among criteria, and methodology adaptation. A case study is conducted to demonstrate how the proposed framework is used to improve the scalability of a hybrid evaluation model which is used to rank commercial off-the-shelf (COTS) products for a library system and select the best candidate. The case study is also used to determine which keystones are effective in improving scalability.