Joris Dormans (2004)
The advent of new media in western society is accompanied with a few buzz words. 'Visual' and 'nonlinear' are terms that are often used to describe those media artefacts that are the harbingers of things to come. In this article I will examine what it means for a text to be nonlinear and tie the notion of nonlinear to a visual or graphic structuring of information. In doing so, I am not trying to join a chorus of voices to praise these new-found cultural qualities, nor do I wish to join a conservative camp and condemn visual culture. Visual culture is here to stay, for better or for worse. What is lacking, however, is our understanding of it, and that is what I wish to address in this article.
In his influential Course in General Linguistics Ferdinand de Saussure identified the linear character of the auditory signal as one of the principles of the verbal sign (1983: 103). Since then Saussurian semiology has been applied to many different types of languages and many different media, with written text probably studied the most extensively. This practice might have given rise to the misconception that written text, like spoken text of which it is a record, is linear. This misconception is only recently identified, not in the least, because of the rise of theories of (so called) nonlinear hypertexts.
It is my view that the linearity of the text is often mistaken for its one-dimensional organization. This often leads to confusion when notions such as 'linear access' are applied to these text. Let me explain.
There is nothing wrong with Saussure's statements as he confined himself to the spoken text. Spoken text indeed "occupies a certain temporal space, and [ ] this space is measured in just one dimension: it is a line" (ibid. 103; italics in original). However, the same claim cannot be made for written text which occupies a certain spatial space, and this space is measured in two dimensions. Written text uses its two-dimensional form to record an essentially one-dimensional object: spoken text, and therefore retains much of the original's one-dimensional character. Books are mostly filled with lines. However, the two dimensions of the page and the graphic nature of the written record superimposes a structure onto the written text that might be called extra-linguistic (in a sense that this structure is not and cannot be part of spoken text). It is the aim of this article to identify and study this extra-linguistic structure. For now it is sufficient to identify headings, marked use of typography, lists and tables as examples of this extra-linguistic structure.
The nature of this particular construction (a two-dimensional representation of a one-dimensional object) has largely gone unnoticed. Whereas its reversal (a two-dimensional representation of a three-dimensional object, i.e. painting, photography) and the mechanics associated with it are far better understood. Failure to notice this construction has contributed to much confusion in the debate on the (non)linearity of the written text. As Espen Aarseth points out, the written text is far from linear: it allows random access. There is no reason why a reader of a book could not skip parts, reread whole sections or even start at the final page. In fact, many so-called nonlinear hypertexts are much more limited in this way than printed books. Hypertext typically provide the reader with only a few points of entry, and to reach the end he needs to work his or her way through the maze of links (Aarseth 1997: 46-47).
Still, hypertext is often heralded for its nonlinearity. What is usually meant by these claims is that a hypertext is a network of interconnected texts. The network has a structure that is multi-dimensional: the links open connections to other points in the text, they are paths in directions different from the forward and backward of the one-dimensional text. This multi-dimensional character is best illustrated by the spatial metaphor of navigation associated with hypertext. But navigation is only a metaphor and we should not confuse the multi-dimensional organisation of the hypertext network with a spatial one. At best, a typical network allows for topological (as opposed to topographical) representation in the form of network diagrams or maps, which might facilitate our understanding of the hypertext as a whole. This is not to say that a topographical organization is impossible within hypertext. Many hypertexts, such as text adventures, restrict their links to resemble a topographical structure. But it is important to see that this is a self-inflicted restriction, albeit one that facilitates the representation of a fictional spatial world.
Multi-dimensional organization of texts is becoming increasingly important in contemporary culture. The rise of the internet and its dominant structure of the hypertext, caused multi-dimensionality to trickle down into other forms of representation. Most importantly this has led to a reconfiguration of how we structure and use all texts. The inclusion of more graphic elements into written text and the growing importance of these elements in the text's structure are important aspects of the change. In so far we are heading towards a 'visual culture', it is my view that this is not due to the slow replacement of words by images, but to the growing prominence of the extra-linguistic structure or rather the written text's graphic dimension and the resulting importance of something that we might call a 'visual literacy'; the skills that we need to 'navigate' multi-dimensional texts (cf. Kress & Van Leeuwen 1996: 3).
This article is not about the links or active elements in a hypertext. It examines the graphic dimension of both printed texts and individual hypertext pages. Its quest is to unravel the ways in which "the writer [can] specify which sequences of reading are available to the reader" (Aarseth, 1997: 77). Something which might be more prominent in the links of a hypertext but which is not restricted to it. It studies the way headings in a text signify preferred points of entry, how multiple texts on a single page interact, and how a multi-modal text can inspire different readings, among other things. All these are (at least partly) the result of marked use of typography (headings) or visual signs (cadres, images, colours). To unlock the power of the graphic dimension we need to understand the workings of its visual elements and signs. To this end hypertext pages are interesting objects of study because they often have a pronounced graphical dimension: they sport an elaborate layout and feature many images and often present different texts organized in different frames. And it is exactly this aspect of the hypertext page that is trickling down into many printed media.
The use of graphical signs within text predates the hypertext. It might even be that it is with us from the moment somebody thought of using visual signs to record spoken words, although I do not intend to delve into history that far. Traditionally written texts are divided (or articulated to use Saussure's terminology) into distinct units: words, sentences, paragraphs, sections, chapters, etc. This structure is communicated with visual signs: spaces separate words, points mark the end of a sentence, paragraphs start at a new line, sections are preceded by a blank line, chapters often start at the top of the page. All these visual signs contribute to what might be called the rhythm of the written text. At least these perform a similar function as longer and shorter pauses in spoken text. In addition typographical conventions such as the use of italics or a bold font to signify stress mimics some effects of pronunciation and stress that is used to the same effect. The effect of capitalizing words is sometimes referred to as SHOUTING for obvious reasons and can be as effective as the exclamation mark. So far these visual signs are used to structure a text in a way that can also be achieved by speaking. Other signs such as brackets, quotation marks and slashes stem from the written medium. These useful tools for written expression have found their way into contemporary non-verbal (quotation marks) and verbal communication ("his wife slash secretary"). The question mark is a sign that is sometimes redundant because in many languages the status of sentence is often implied by the order of the words, but it might simply signify the typical alteration of tone to that is used in verbal communication to mark a question.
In the internet age creative users have come up with a whole new range of signs to be included in written texts: emoticons (for example the well known smiley: ":)"). These signs add an extra dimension to the written text by coding information that would normally be communicated by gestures, facial expression or special intonations that cannot be captured easily with the signs and symbols discussed above. For example, a few days prior to the writing of this text I wrote in an e-mail directed at a friend "otherwise I would be very disappointed ;)". The dramatic pause (" ") and the 'wink' emoticon (";)") indicate that my friend should not take these words too 'literal'. The many references to gestures and actions other than speech in chat rooms and bulletin boards ("ElfGirl: lol / Julia bows.") are similar tendencies to escape the text's limited dimension of sound in time (see also Turkle 1995: 183).
The latter examples cannot be found frequently in printed media. And all the examples so far do not really exist in dimensions outside the singular dimension of the written text proper. However, written texts - and especially academic texts - have long time included content pages, labelled sections, indexes and cross-references that facilitate a non-linear access and notes that provide the reader with short auxiliary discussions and annotations. These textual devices have always relied on their visual representation in the form of marked use of typography (headings), the spatial organization of the ordered list (index, contents) and the reference to spatial location of certain sequences in the text (page numbers). These devices are today so much part of the text that it is easy to overlook their graphic nature. The power of these means to order a text so that the reader is actually discouraged to read a book from start to finish but return to it for brief visits is one of the reasons for Ludwig Wittgenstein to order his Philosophical Investigations into a series of short numbered paragraphs: "for it compels us to travel over a wide field of thought criss-cross in every direction" (1953: vii). For Wittgenstein the one-dimensional character of the text was insufficient to capture the multi-dimensional character of his philosophy, so he devised a new textual mode to better represent the latter.
If we compare at a glance an academic text of today with one of, say, one-hundred years ago the advance of visual culture is striking. Where in the old book letters and words dominate, the most striking feature of the modern book is likely to be its carefully designed layout. Section names are clearly distinguishable from the main text, definitions or important passages are often framed or otherwise visually marked, sometimes keywords are placed next to the text at appropriate locations for ease of reference. Visual material in the form of coloured illustrations, diagrams and charts are much more frequent. Today it is even possible (and accepted) to publish academic text in the form of a comic book! (see McLoud 1993). A few developments have contributed to these differences. First of all the production and reproduction costs have become a lot lower, making it cheaper to experiment with new formats. Second, new reproduction techniques such as full-colour prints and the use of computers have revolutionized the field by making possible new design and making it available to a much larger group of people. Third, one-hundred years ago modern visual design was still in its infancy; then it was a new field, pioneered by printers, painters, and architects. Today visual design is an establish art that is taught at art academies and practised by an increasingly large group of professionals. All these have contributed to the development of visual design and opened up the possibility for new modes of reading and writing.
There is also a danger to the emergence of the multi-dimensional text. Old texts are very open: lacking many visual cues that refer to a textual structure imposed on it by either author, editor, designer or publisher. Academic use of these texts allowed for personal method of reading, scanning, skipping and reference. Improper use of visual markers or bad design can actually interfere with this process, steering the 'reader' down paths that would not be his own. That is probably why I personally find it very annoying when somebody leaves marks and notes in library books. Recall that Espen Aarseth associated hypertexts with author control over the sequence of reading (see above). The author proclaimed death by Roland Barthes (1977) rattles his bones again but with this renewed control also comes responsibility.
The use of visual sign in everyday communication has grown ever more sophisticated. Once images were predominantly figurative illustrations that mainly supported written text, relying on transparent codes and iconic signs. Today the use of graphic signs and the graphic dimension of texts has increasingly been subjected to particular conventional use. The visual signs used at Schiphol Airport (designed by Total Design) have been adopted by many other airports all over the world. This is mainly because even in a more formal use, graphic signs often retain some of the image's natural transparency. This is in line with arguments of Kress and Van Leeuwen who point out that their visual grammar take the biological constraints of perception as it basis (1996). I have expanded on this elsewhere and argued that this is due to a direct form of correspondence between the visual means of articulation and the signified of the expression, but at the same time I have shown that the marked divergence of use of visual signs in different cultural contexts should be understood as the emergence of different languages of visual design (Dormans 2004). The use of visual signs is thus rooted in both the biological constraints of perception and an inherently ambiguous, cultural context.
A visual grammar is required to study these visual languages and to study those aspects of multi-dimensional texts that are built on a graphic structure. In Reading Images (1995) Gunter Kress and Theo van Leeuwen do exactly this. They present a lot of typical graphic compositions and structures and comment on their use and meaning. They expose and discuss concepts such as narrative processes, conceptual processes, framing, salience, composition and visual modality. There are a number of problems with Kress and Van Leeuwen's, however. Charles Forceville points out some of these problems: the very complicated and rigid hierarchy of different types of visual structures in which some obvious options seem to have been omitted, the unclear and sometimes fuzzy categories on which this hierarchy is built, and in some instances flawed interpretations of some examples (Forceville 1999). Forceville's solution to escape Kress and Van Leeuwen's rigid hierarchy is to interpret their structures as typical or prototypical schema's. This seems to be a good alternative. By focussing on the key notions of Reading Images such as the different types 'participants' and 'vectors' of narrative processes (Kress & Van Leeuwen 1995: 45-69) and the way these combine an infinite set of possible structures opens up. On a formal level this kind of grammar resembles the transformative generative grammars devised by Noam Chomsky (1957). The gains of such a construction are obvious: we can use the experience of five decades of linguistics research for the study of graphic structures, and it would make it easier to study both words and images in unison. But there is a danger also.
There have been many attempts to assimilate the domain of visual design to the study of language in general. Such attempts sometimes provoke hostile reactions (for example Wollheim 1996). And we should do well to listen to these reactions with care. All too often linguistic expeditions into the visual domain fail to recognise the visual aspects of graphic design. The translation of a picture into words for further procession by a linguistic theory or practice is an example of a method that fails to do justice to the graphic qualities of the image. Semiotics can often be accused of exactly this kind of practice, because semiotics is a theory that has been designed with spoken (and truly linear) text in mind and needs to expand its methods and concepts before it can accommodate the graphic dimension. What is needed is an approach to visual grammar that shares the formalistic rigour of linguistics and semiotics but at the same time is able to recognize those aspects of the image that are truly graphic. Works like Reading Images lack this rigour and do point the right way in dealing with the visual.
One, and perhaps the most, significant part of reading is the making of references between the various statements. When reading any text "the 'glue' binding disparate elements spread out over several hundred pages is our ability to perceive references between sentences and paragraphs" (Douglas 2000: 111). Reading requires the cognitive effort of constructing of a conceptual whole of what is being read. This aspect of reading is fore-grounded by the modernist novel with its fragmentary narratives, in which often crucial events are omitted, and thereby requiring more, and more conscious, effort to construct a meaningful whole out of the fragments. It is exactly for the same reason that the art of comics must be looked for between the panels (McLoud 1993: 90-93), or why the cut is so prominent in theories of cinema (cf. Borwell & Thompson 1997). Into these non-spaces of omission the reader projects his or her ideas of the story or the meaning of the text. The author of a text can point the way for these ideas to develop, but never truly control them. In fact, it might be the author's wish to control them as little as he can, or be deliberately ambiguous and let the reader choose between several options.
This aspect of reading is very profound in the multi-dimensional text. As discussed above, a multi-dimensional text offers the reader a set of textual units that can be read in any sequence, parts can be skipped, and a non-sequential conceptual representation can be constructed from the reading process. The multi-dimensional text is very well suited to represent non-sequential knowledge or experiences, and since only a relatively small subset of all knowledge and experiences are sequential it is no surprise that the multi-dimensional text is growing more important within our visual culture. In fact, I have argued that truly one-dimensional (use of) texts always have been very rare. Most things in life are simply more complex than a single chain of cause and effect, even though such a chain may be sufficient to explain some aspects of life. In its essence, the multi-dimensional textual form offers more ways to facilitate and to control the making of references by the reader.
For the author of multi-dimensional text this brings a new responsibility. Within a multi-dimensional text the author is responsible for all the links and relations that suggest themselves from the text's form. When the intention of the author is to communicate a clear message s/he better make sure that the graphic dimension of the text supports this intention. The multi-dimensional text incorporates a graphical rhetoric for better or worse. For an author with artistic intentions the graphic dimensions of a text offer new and relatively unexplored mechanisms or poetics. The graphic dimension can be used to create ambiguous meaning, new forms of paradoxes, metaphors, etc. Experiments with this kind of visual poetics can be traced to before the rise of computer mediated text, with the visual poetry by Apollinaire being one of the best examples.
From the reader this requires a new form of literacy. The ability to understand text and to quickly process vast amounts of information is very important in modern society. People who have much experience with multi-dimensional texts have developed skills that facilitate their reading. With the growing dominance of multi-dimensional texts these skills also become more important. However, one must not confuse this visual literacy with the notion of visual grammar as presented in this article. A visual grammar is a model that attempts to formalize the graphic constructions in multi-dimensional text. Visual grammar should be used to study multi-dimensional text and our interpretation of such text. As one can become literate without studying linguistics, a reader does not need to have academic knowledge of visual grammar to understand a multi-dimensional text. It might help the critical reader, but to those interested in the workings of multi-dimensional text visual grammar is an important tool.
The status of svisual grammars that have been devised up to this point (Kress & Van Leeuwen 1995, Dormans 2004) are far from definite. The graphic dimension of text has been neglected for a long time. Only since the emergence of theories like semiotics and gestalt psychology did visual representations come to the attention of academics, and still only a select group of scholars have studied these in detail. A lot of work remains to be done. However, the need to develop a visual grammar should be evident. The principles underlying visual grammar the existence of the graphical dimension, the significance of the (implied) relation between textual units, the social context of visual languages, the embodiment of visual perception are well founded. These principles oblige us to look at multi-dimensional text in a particular way: to recognize the graphic resources the text draws on and to investigate these in their graphical essence. This should allow us to found speculations about the meaning of a text on elements that are actually present within the text, discuss these in objective terms and compare these with the structure of other texts. It should help us to identify and use the visual aspect of visual culture, it should help us to better construct and understand nonlinear text. Ends worthy of pursuit.
 "lol" is an abbreviation for 'laughing out loud'. 'Julia bows' is adapted from some well-documented transcriptions of conservation of co-produced by the 'chatterbot' Julia: an artificial program developed to log on to Multi-User Dungeons and to act as if it were another player (Murray 1997: 216).
Aarseth, Espen J. (1997) Cybertext, Perspectives on Ergodic Literature. Baltimore, The Johns Hopkins University Press.
Barthes, Roland (1977) "The Death of the Author" in Image Music Text (pp. 142-147). New York, Hill and Wang.
Bordwell, David & Kristin Thompson (1997) Film Art, An Introduction, Fifth Edition. New York, MacGraw-Hill.
Chomsky, N. (1957) Syntactic Structures. The Hague, Mouton Publishers.
Dormans, Joris (2004) Visual Grammar, Constructing a Grammar of Visual Design. MA thesis, Free University Amsterdam. Available at: http://www.jorisdormans.nl/article.php?ref=visualgrammar
Douglas, J. Yellowlees (2000) The End of Books Or Books without End? Reading Interactive Narratives. Ann Arbor, The University of Michigan Press.
Forceville, Charles (1999) "Educating the Eye?, Kress and Van Leeuwen's Reading Images: The Grammar of Visual Design (1996)" in Language and Literature 1999, 8, 2, pp. 163-177.
Kress, Gunter & Theo van Leeuwen (1996) Reading Images, The Grammar of Visual Design. London, Routledge.
McLoud, Scott (1993) Understanding Comics, The Invisible Art. New York, HarperCollins Publishers.
Murray, Janet H. Hamlet on the Holodeck, The Future of Narrative in Cyberspace. Cambridge, The MIT Press.
Saussure, Ferdinand de (1983) Course in General Linguistics. Chicago, Open Court.
Turkle, Sherry (1995) Life on the Screen, Identity in the Age of Internet. New York, Touchstone.
Wittgenstein, Ludwig (1953) Philosophical Investigations. Oxford, Basil Blackwell.
Wollheim, Richard (1996) "On the assimilation of pictorial art to language" in Jon Thompson (ed.), Towards A Theory Of The Image. Maastricht, Jan van Eyck Akademie.