Evaluation of a Semi-automated Semantic Annotation Approach for Bootstrapping the Analysis of Large-Scale Web Service Networks

In recent years many methods have been proposed, which require semantic annotations of Web services as an input. Such methods include discovery, match-making, composition and execution of Web services in dynamic settings, just to mention few. At the same time automated Web service annotation approaches have been proposed for supporting application of former methods in settings where it is not feasible to provide the annotations manually. However, lack of effective automated evaluation frameworks has seriously limited proper evaluation of the constructed annotations in practical settings where the overall annotation quality of millions of Web services needs to be evaluated. This paper describes an evaluation framework for measuring the quality of semantic annotations of large number of Web services descriptions provided in form of WSDL and XSD documents. The evaluation framework is based on analyzing network properties, namely scale-free and small-world properties, of Web service networks, which in turn have been constructed from semantic annotations of Web services. The evaluation approach is demonstrated through evaluation of a semi-automated annotation approach, which was applied to a set of publicly available WSDL documents describing altogether ca 200 000 Web service operations.