We ran a MODE seminar on Analysing Space and time in mobile and online environments this week. We introduced key concepts from multimodality related to space, place and time and demonstrated how these could be used to analyse blogs, and the use of mobile applications such as Layar, Runtastic, and Foursquare.
Towards the end of the day one of the participants pointed to the ideological and political connections among the concepts we were drawing on from theories of Space (e.g. Spatial practices (de Certeau), Spatial flows (Castells), as well as the work of Soja and Foucault and multimodality. He noted that the ways in which social semiotic multimodality was being used was focused on critique (albeit critique with the possibility to inform design and re-imagining of technological and social spaces) that aligned with the broadly Marxist underpinnings of this work. Was this a part of multimodality he asked. An interesting question: though in work now turned around into a focus on design, which can, in its turn, be used for purposes of critique. We endeavoured to answer this with a potted history of multimodality – which I will try to re-create here.
Multimodal social semiotics has two parts: “multimodality” which names the domain of application; and Social Semiotics, which names the theory used in accounting for issues of meaning in that domain. The latter has its origins in the work of the linguist Michael Halliday. He had moved away from the work of structuralist linguists (such as Saussure) to a functional approach (drawing on the work of anthropologist B. Malinowski, and an implicit Marxism) to treat language as a social semiotic – that is language as something that is lived and used, fluid and changing, and shaped by its social uses – shaped by what people use it to do. From this came the understanding of language as produced in social functions: an embedding social order of meaning in which language is used to realise three meaning functions: the ideational to express what is ‘going on’, ‘the case’ in the world; the interpersonal to express relationships of participants in interaction; and the textual to provide coherence – the relationships between elements in the text and of the text with its environment. Bob Hodge and Gunther Kress drew on this work in Language as Ideology (1979/1993) in which the relation of social power (as a generative source of meaning) and language is traced in detail. Marx and Freud were explicitly there as the central intellectual foundations, while Halliday’s linguistics provided the basic theory, with the important addition of ‘transformation’ from Chomsky’s theory. Realizing that language was one among other cultural resources, they extended their earlier work in Social Semiotics (1988), to include attention to the visual, to fashion, to sculpture, and so on. From 1986 on, Gunther Kress and Theo van Leeuwen started to articulate the visual as a social semiotic system, resulting in their book Reading Images: A grammar of Visual Design (1996). They made the theoretical assumption that as all modes are the product of social action, then anything that is considered a mode will realise meanings in the three functions. They drew in a range of theoretical resources from Art History and Sociology in relation e.g. to symbolism (Panofsky), colour (Arnheim), and framing (Goffman), as well as work of theorists such as Edward Hall, and Foucault, both of whom contribute to understanding of space and power.
This work has been and is being further developed (by the MODE team and others) to explore the semiotic resources of sound, music, gesture, colour, and so on. So yes, social semiotic multimodal analysis is built on the foundation of Marxist theory (as one significant strand in understanding how power is articulated through representational resources, be they speech, image, writing, movement, or the multimodal configurations of these resources into technological platforms such as the applications of Layar, Runtastic and Foursquare and how they shape and influence social interaction in time and space. So thanks for the question!