The conversion of diagrams to knowledge bases

If future electronic documents are to be truly useful, one must devise ways to automatically turn them into knowledge bases. In particular, one must be able to do this for diagrams. This paper discusses biological diagrams. The author describes the three major aspects of diagrams: visual salience, domain conventions and pragmatics. He next describes the organization of diagrams into informational and substrate components. The latter are typically collections of objects related by generalized equivalence relations. To analyze diagrams, the author defines graphics constraint grammars (GCGs) that can be used for both syntactic and semantic analysis. Each grammar rule describes a rule object and consists of the production, describing the constituents of the object, constraints that must hold between the constituents and propagators that build properties of the rule object from the constituents. The author discusses how a mix of parsing and constraint satisfaction techniques are used to parse diagrams with GCGs