On the specification of dynamic visual languages

We study the dynamics of visual sentences produced during interaction between user and computer. Our approach states that the messages exchanged between the user and the computer not only should make explicit the current state of the interaction, but also the possible paths that it may take. We introduce the notion of “dynamic visual language” as a set of visual sentences characterised by the presence of a common frame which conveys the context in which users are operating. We present a formal model of derivation of dynamic visual sentences, in which each visual sentence specifies the possible actions which can be performed on it and the possible transformations it can go through. The concepts are illustrated by examples in visual navigation and visual querying