How to integrate text and images interactively

‘Show, don’t tell’ dictates the famous credo in creative writing. Maybe it should say: show and tell. Communication often benefits from the combination of text and images. Within one message, some meaning is best encoded using written language while other information is best expressed in a picture, a graph, a diagram or a map. Combinations of these representational modalities are part of all kinds of genres, like news reports, textbooks, scientific articles, advertisements, comics, and even novels.

Although text and images can complement each other and thus can benefit comprehension, their combination also forces the readers to split their attention between the two. To reduce this split attention effect it is mandatory that text and image are optimally integrated. This can be achieved using design (for example by placing related textual and graphical elements close together, or linking them with lines or arrows) or by creating linguistic relations, like a textual reference to an image (‘see figure’), or its content (‘as the figure shows, there was a peak in January’).

Digital media have added a powerful new way to integrate text and images: interactivity. Let me explain by example. The model on the right, based on the dissertation of Cengiz Acartürk (pdf) , shows how a combination of correlated text and image (a so-called multimodal constellation) consists of a text layer and a image layer. The first contains the written text, the second is divided in two sublayers: one containing graphical elements the other textual elements (for example a title, axis labels, numbers and verbal annotations). All layers and sublayers can be linked together by referential relations. Examples are written words in the text layer that refer to the image as a whole (link 1) or to the content of the graphical sublayer (link 2). Within the image layer, textual elements can be linked to graphical elements (link 3), for example by pointers. As this paragraph illustrates, all these links also can be interactive.

Several researchers have experimented with such hypertext connections between text and images. In the late 1990s, human-computer interaction researchers Pete Faraday and Alistair Sutcliffe studied interactive co-references between text and animations and measured a positive effect on comprehension. More recently, educational psychologists Tina Seufert and Roland Brünken also concluded that hyperlinks can guide the attention of the reader to relevant connections between text and images.

Interestingly enough, interactive text-image references are rarely used on the web. Sometimes they appear in long forms, the storytelling format that has become quite popular the last couple of years. For example in Snowfall , the archetype of this new genre created by The New York Times, hyperlinks in the text can be used to start correlated video. More sophisticated are Bret Victor’s explorable explanations and reactive documents, that allow users to manipulate text and images thus enabling what Victor dubbed ‘active reading’. Be sure to try out his redesigned version of Watts and Strogatz’s seminal paper on ‘small world’ networks.

From there it’s a small step to interactive links between text and data visualizations. Especially when visualization is used to communicate rather than to analyze data, good annotation is essential. Accompanying text can tell the audience not only what the visualization is about but also how to read and use it, for example by highlighting interesting parts or applying filters.

Let me again illustrate with a example. This chart shows the increase and decrease of the population of Amsterdam during the last decades. The red line indicates the total growth or decline. Except for the years 1994, 1995 and 1996 the population has been growing. This is mainly due to natural increase : more births than deaths. The last couple of years the net domestic migration also has been positive: more people in The Netherlands move to the capital than vice versa. The same is true for the net international migration although the influx of citizens from abroad is considerably lower than in the early nineties.

This chart is part of a project I did for the Research and Statistics Bureau of the City of Amsterdam that contains several more examples. If you have different thoughts on the interactive integration of text and images, or other examples, feel free to share them in the comments below!