Formalising visual languages

We propose a formalisation of visual languages which allows a uniform approach to satisfying the needs of pattern recognition, image generation, and visual reasoning faced in visual human-computer communication. Such needs comprise managing the full screen image as seen on the computer display, handling interpretation of icons even when ambiguous, or generating multiple representations to convey one same meaning as required by different users for diverse tasks. We also formalise the way the machine associates a computational meaning with an image (including the whole screen image), and conversely, the way it generates an image on the screen from a computation. A definition of visual sentence as interpreted image is proposed, and the visual language is viewed as a set of visual sentences in a user-computer dialogue. Examples of how the proposed formalism is suitable for ambiguity control and multiple representations of meanings are provided