Multi-modal work has been an increasing, and welcome, presence in ROLSI, and here Jackson Tolins gives us a blow-by-blow account of working with complex data.

For researchers of language and social interaction, capturing the intricacies of spontaneous talk-in-interaction has always posed a challenge. We hope to present in our manuscripts not only what is said, but the subtle details of how it was said, capturing the critical features of the talk relevant to how the interlocutors create, coordinate, and respond in conversation. For talk, this challenge has been met with the standardized transcription practices such as the Jeffersonian system.

As research on language in social interaction has expanded in an embodied direction, we again come up against the issue of portraying data in our communications with the field. Analysts must tackle decisions on how best to present the use bodily movements, hand gestures, gaze, and other modalities. Playing videos at conferences meets this task, but for manuscripts, the field of possibilities is wide open, including of image stills series, diagrams, novel annotation schemes layered on to the speech transcription, and a variety of other techniques. Decisions on how best to present multimodal data are left in the hands of the authors themselves.

An example – transcribing music music teaching

My own recent work can serve as an example, centering on the use of nonverbal, embodied communication in music pedagogy. Having never done particularly well in any sort of art class, for me drawings and diagrams were mostly out of the question (think Mac Paint stick figures). For capturing bodily movement, this left me with a combination of describing the gesture in the transcript (Figure 1) …

… and making use of a series of still shots (Figure 2).

I had originally toyed with the idea of including arrows demarking direction in the series (Figure 3), but decided that this would be best left off, leaving the momentum to be filled in by the reader.

For capturing the vocalizations that the musicians use in their enactments of performance, the typical pitch transcription symbols seemed inefficient, (↑uuurr↓lll ↓iii↓↓aaaa?). Apart from presenting a full spectral visualization, which I thought would be heavy handed, this left me with the music notation itself (Figure 4), (and crossed fingers that most of the readership would have enough experience with music that this would be a useful way of presenting the melody mimicked by the voice).

Even within this single embodied display, a variety of techniques for representation were available, and there’s always the possibility that an alternative would have lead to more easily digestible representations. Similarly, given a limit on manuscript length, there were dimensions of the enactments that simply could not be given their full dues.

Authors must be artists

Authors must be artists of a sort, filtering and condensing the multiple channels and creating images that pull the many modalities of communication onto the page.