Development and validation of evaluation indicators for a consortium of institutional repositories: A case study of dcollection

This study develops evaluation indicators for a consortium of Korean institutional repositories called “dCollection” and validates the indicators against actual data from the participants of this consortium. The literature review reveals a conceptual framework for institutional repository (IR) evaluation with four categories and 19 items. In developing the initial framework, equal amounts of emphasis are put on the assessments of procedural achievement and actual performance to pinpoint the procedural weaknesses of each IR and to determine its customized solution. A Delphi method of three rounds with the help of IR librarians reveals a converging tendency pertaining to the measures of importance ascribed to the categories and items. Through a focus-group interview with middle- to top-level managers, 39 indicators derived from 19 items are identified as possessing relevancy, measurability, data availability, and differentiability. Validation of evaluation indicators employs actual evaluation data from 32 university IRs. Factor analysis shows a simpler structural pattern containing 12 factors than that of the structural pattern of the conceptual framework that contains 19 items. Correlation analysis using the factor scores identifies six key factors: Registration Rate, Archiving, Resource Allocation, System Performance, Multifunctionality, and Use Rate. The results from regression analyses suggest that two different approaches can be employed to promote the Use Rate factor. In the content-oriented approach, the Registration Rate factor is crucial while in the policy-oriented approach the Archiving factor assumes this role; however, the System Performance factor plays a mediating role for the key factors, thus forming a contingency for either approach.