Task-based assessment of performance and cost-effectiveness of automatic metadata extraction

Automatic metadata extraction tools can improve the effectiveness of media production processes. However, it is difficult to assess the applicability, expected performance and thus cost-effectiveness of a specific tool in a specific task context. We propose the introduction of a task-based approach in the domain of multimedia analysis, in order to assess the practical value of automatic information extraction tools for media production tasks. Using formalized machine readable models of these tasks, multimedia analysis tools can be assessed in the context of a real media production workflow rather than evaluating these tools in an isolated lab setting. We model dependencies of the performance of analysis tools on their input using a Bayesian network, and show that we obtain a better measure for the quality of the analysis results from this particular tool than with generic metrics. We also show that the method can be used for performance prediction based on content properties. The second contribution of this paper is the application of task models for cost simulation, comparing the effort of manual correction of results generated by automatic content analysis tools at different performance levels with a fully manual completion of the task. This enables determining a minimum performance level for cost-effective application of the automatic tools.