Structured Natural-Language Descriptions for Semantic Content Retrieval of Visual Materials

Keyword search of multimedia collections lacks precision and automatic parsing of unrestricted natural language annotations lacks accuracy. We propose a structure for natural language descriptions of the semantic content of visual materials that requires descriptions to be (modified) keywords, phrases, or simple sentences, with components that are grammatical relations common to many languages. This structure makes it easy to implement a collectionʹs descriptions as a relational database, enabling efficient search via the application of well-developed database-indexing methods. Description components may be elements from external resources (thesaurus, ontology, database, or knowledge base). This provides a rich superstructure for the meaningful retrieval of images by their semantic contents.