Effective standards for metadata in the GCMD data access system

The paper presents an information retrieval system for use by the Global Change Master Directory. The GCMD is a repository that contains Earth Science data collected by various agencies worldwide. The GCMD does not house the actual data, it contains descriptions of the data including the location of the actual data set. The GCMD also provides search services to locate these data descriptor files. For data to be included in the GCMD database, it must be submitted to the GCMD in the Directory Interchange Format (DIF). This DIF submission is currently done by data collectors manually submitting the DIF to the GCMD, but this manual system cannot keep pace with the amount of data being collected. Our proposed solution to keep pace with data being collected is to design and develop a data access system for the GCMD to automate the DIF creation process. Our data access system will be capable of autonomously searching Web sites for Earth Science data sets, extracting the metadata from these data sets, and creating a DIF for the file. The paper describes our prototype system that uses a URL pool to direct its search for Hierarchical Data Format (HDF) files. The HDF file is a self-describing format and contains metadata describing the contents of the files. This metadata is extracted and mapped to the DIF format. We present examples of DIFs created by our prototype to demonstrate that our approach is feasible, and discuss the need for a metadata standard among scientific data sets and how such a standard would enhance the effectiveness of our system and others in the Earth Science community