Reflections on Multimodal Transcriptions

Author: Marianne Wurtz, Aalborg University, Denmark

I am researching students’ production of multimodal texts in a Danish primary school. As I am interested in the choices the students make in their design process, I focus on the dialogues in the group of students. These dialogues are often non-verbal and therefore I have used multimodal transcriptions. My aim is to try to make my reflections, during the process of transcriptions of few minutes of a video observation, more transparent.

The original interaction is framed as students’ transduction of “Jairus’s daughter” from the Bible to a multimodal text. In the video clip the students are previewing their realization of the scene where Jairus receives the message of his daughter’s death and suddenly they are laughing. Why do they laugh? In what way is their laughter a response to the sign which they had made?

In order to answer these questions I first made a written transcript. I used a tabular format, with temporality on a vertical axis and modal separation horizontally, in order to analyze the students’ response across modes unfolded in time. I started with 4 columns and rich description. Then I elaborated my template to get a better overview, with columns for distance from the screen, gaze, body posture, speech and voice. I ended up with 12 columns because it also became necessary to divide the multimodal text, which the students were previewing, into modes in order to answer the question. I highlighted the most significant modes by positioning them to the left.

In the beginning of the preview of the scene the students all gaze at the screen. After the lines: “Is she?” three of the students rapidly gaze at Michael (who is outside of the webcam), at the screen and at Michael while they laugh.The scene they are previewing is communicating grief. Grief is represented across modes: bodily in the students acting the characters, through voice in ‘the acting with voice’ of the servant expressing compassion and in the image which foregrounds Jairus by his position in the center and by the gazes from the servant and Jesus.  It is Michael who has recorded both the lines of the servant and the lines of Jairus. In order to let the audience know that it is two different persons speaking, he has changed his voice as he speaks the lines of Jairus. The result is a voice quality that is more like a girl speaking than that of a man.

I see the students’ laughter as a resource for forming a hypothesis about the students’ interest. The students are engaged in communicating the grief of Jairus, and they laugh because the use of voice is a contrast to the grief they want to communicate.  In the preview situation they shift positions from makers of the text to interpreters of the text.  Their laughter is a sign of disagreement of the recorded speech as the most apt use of voice in relation to the ensamples of modes. Learning a body of knowledge like Danish includes interpretations of literary texts. In the students’ text interpretation as transduction they have made their interpretation of this scene material. The preview turned their interpretation as an outward sign into a prompt and their laughter is a response to this sign as the most apt realization of this scene. The laughter was a sign of learning.

My first transcripts gave me analytical insight and helped me select the modes that were significant in order to answer my question. The next step was how to represent my findings in order to persuade an audience. My representational choices were made during my stay in London Knowledge Lab in the social context of the MODE group and inspired by their Blog2013 10 08comments on my different transcriptions.


My final template (Figure 1) is framed by my research question, interpretation of literary texts as part of the curriculum for Danish and by a multimodal social semiotic approach to learning. I have excluded the modes which were not significant and chosen to represent my findings in image and writing as the most apt choice to communicate this single episode.

I chose to use images in order to stay closer to the video and at the same time give the reader the possibility to ‘see for himself’. Audiences might have difficulties separating the text the students are previewing from their response to the text. I have therefore used color and layout as signifiers: Purple signifies the text, and blue signifies the students’ response to their text. Speech bubbles separates voice from the text from voice in the response. Title and icons in the screen dumps separates images from the text from images in the response. Images from the text are used to document the grief of Jaurus and images of the students’ response to document gaze and laughter. To make the gazes more clear I added arrows. Time was significant to document the shift from serious engagement to laughter exactly after the lines “Is she”. And I changed the layout so time now is horizontal and text and response vertical. By placing the students response above the text the transcript becomes a visual documentation for when the students gaze at the text and when they gaze at Michael.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s