Integrating knowledge, semantics and digital media into the multimedia generation process

In order to be satisfactorily adequate in generating relevant multimodal information, we argue that any multimedia and multimodal ontology has to incorporate three basic criteria. These are: (i) a conceptually and semantically clear distinction between the operational concept of Modality and Media (medium), (ii) describe a set of recursive formal rules that can allocate and vehicle the appropriate modality information through the most relevant media, taking into consideration human cognitive constraints of perceiving and interpreting relevant information and (iii) develop formal rules to ensure that the output knowledge about the different modalities that constitute a final multimodal presentation can be recombined, reinterpreted and regenerated. The relevant interaction of these criteria to ensure the generation of optimal and relevant multimodal information in the multimedia and multimodal systems requires the existence of a modality ontology which can formalise this interaction. This paper, which is mainly concerned with the presentation of a modality ontology is a step in that direction.