A critical look at software capability evaluations

The methods used by the Software Engineering Institute (SEI´s) Software Capability Evaluation program (SCE) are assessed. The goal of the SCE program is to provide the US Defense Department with a method by which it can rank the overall capability of organizations to produce software in a timely, repeatable fashion. Because SEI assessments are a preparation for SCEs, the authors first describe the major steps in an SEI process assessment and highlight its strengths and weaknesses. The SCE grading methods are described, focusing on their statistical reliability. It is concluded that the system is so seriously and fundamentally flawed that it should be abandoned rather than modified or updated. SEI´s overall process-improvement paradigm, that is, the set of goals and directions it is trying to impart to the industry through its assessment and evaluation programs, is examined. It is suggested that the assessment program needs some type of structured, preferably graphical, method for recording the details of an organization´s existing processes.<>