Skip to content

Advertisement

  • Research
  • Open Access

Embodied meaning: a systemic functional perspective on paralanguage

Functional Linguistics20196:1

https://doi.org/10.1186/s40554-018-0065-9

  • Received: 10 September 2018
  • Accepted: 22 October 2018
  • Published:

Abstract

This paper develops a framework for analysing paralanguage, initially inspired by systemic functional linguistic (hereafter SFL) research on early child language development. A distinction is drawn between non-semiotic behaviour (somasis) and meaning (semiosis), and within semiosis between language and paralanguage (using the term paralanguage to refer to semiosis dependent on language and realised through both sound quality and body language, the latter including facial expression, gesture, posture and movement). Within paralanguage a distinction is drawn between sonovergent resources in sync with or in tune with the prosodic phonology of spoken language, and semovergent resources supporting the ideational, interpersonal and textual meaning resources of spoken language’s content plane. The paper closes with a brief discussion of the intermodal relations among language, paralanguage and other modalities of communication.

Embodied meaning

In this paper we introduce a social semiotic framework for analysing paralanguage. Our approach was inspired by Chris Cléirigh’s contributions1 to New South Wales Youth Justice Conferencing research consolidated in Zappavigna and Martin (2018) and Martin and Zappavigna (2018).2 Cléirigh’s work drew on Matthiessen’s synopses (Matthiessen 2004; Matthiessen 2007; Matthiessen 2009) of Systemic Functional Linguistic (hereafter SFL) research on early child language development. Following Matthiessen (2009) we use the term paralanguage to refer to gestural resources arranged along what McNeill (1992) christened as ‘Kendon’s Continuum’ (gesticulation, pantomime and emblems), along with the vocalisations outlined in van Leeuwen (1999) not usually included in linguistic descriptions of the segmental and prosodic phonology of spoken language (timbre, tempo, tension, pitch range etc.). In this paper however we will consider only gestural systems.

There are of course many ways to classify gestural resources. Kendon 2004 (Chapter 6) provides a thorough historical survey. The most useful vantage point from which to compare classifications is Kendon’s Continuum. The introductory chapters in McNeill (McNeill 2000a, 2000b; McNeill 2012) include clear presentations of the model outlined in Fig. 1 below (taken from Sekine et al. 2013). We will cross-reference our work to this model as we present our framework, setting aside the sign languages of deaf communities (ASL, BSL, Auslan, LSF etc.) since these are languages in their own right (themselves involving paralanguage; Johnston 2018).
Fig. 1
Fig. 1

Kendon’s Continuum (as rendered in Sekine et al. 2013)

Following on from Martinec’s ground-breaking SFL inspired studies (Martinec 1998, Martinec 2000a, 2000b, Martinec 2001, Martinec 2004, Martinec 2008; see also Muntigl 2004) we will organise our description around the kinds of meaning being made – in SFL terms the trilogy of ideational, interpersonal and textual meaning.3 Ideational meaning involves resources for construing reality, interpersonal meaning involves resources for enacting social relations and textual meaning involves resources for managing information flow. Textual meaning corresponds roughly to beats and pointing/deictics in Fig. 1; ideational and interpersonal meaning involves both iconic and metaphoric gestures. The correspondences with Kendon’s distinction between representational and pragmatic functions of gestures is outlined in Table 1 below. Ideational gestures would be representational in his terms; and his pragmatic gestures (defined as not referring to referential or propositional content) include both interpersonal and textual functions.
Table 1

SFL metafunctions (ideational, interpersonal and textual meaning)

SFL metafunctions

‘function’

Kendon (2004: 158–159)

Ideational

Construing reality

Representational

Interpersonal

Enacting social relations

Pragmatic (modal, interactive, performative)

Textual

Managing information flow

Pragmatic (parsing)

There are a number of reasons why our SFL interpretation of paralanguage is timely. One has to do with the explosion of SFL inspired work on modalities other than language triggered by Kress and van Leeuwen’s (1996, 2006) monographs, which focused on single static images. As reviewed in Martinec (2005), O’Halloran et al. n.d.(in press) and Taylor (2017) this work has now been extended to the study of diagrams, Powerpoint slides, webpages, comics, picturebooks, moving images, sound and music, architecture, sculpture, toys and behaviour. Since so many texts involve one or more of these modalities, it is advantageous when studying inter-modal relations to be able to draw on descriptions informed by the same theoretical principles. The concept of metafunction introduced above for example allow us to compare like with like as far as convergence and divergence of meaning across modalities is concerned (Painter et al. 2013). Paralanguage is so closely coordinated with spoken language and so regularly implicated in inter-modal texts of several kinds that the utility of a common metalanguage is clear.

Alongside theoretical integration, SFL is particularly well-suited to the study of paralanguage in a number of ways. One is that it provides a linguistically informed model of prosodic phonology (Halliday 1967, 1970; Halliday and Greaves 2008; Smith and Greaves 2015; van Leeuwen 1992; Martinec 2002) which can be used to make explicit the coordination of rhythm and intonation in spoken language with beats and strokes in gesture. This has facilitated Martinec’s development of Kendon’s early work (Kendon 1972) in this area, taking into account later work by Tuite (1993). We will in fact suggest that SFL’s tone group, analysed for rhythm and tone, provides an essential unit of analysis for work on paralanguage as far as questions of synchronicity across modalities are concerned.

Another advantage of SFL is the clear distinction it draws between paradigmatic and syntagmatic relations (system vs structure in SFL terminology). As is well-attested, there is more variation in the language structures realising systemic options than in the underlying systems themselves (Caffarel et al. 2004). This is even more true when comparing the structural realisation of systems from one modality to another. Kendon’s (2004: 186–187) well-known example of the different trajectories of the gestures accompanying “sliced the wolf’s head off” vs “sliced the wolf’s stomach open” illustrates this point. The swinging arm motions are very different structurally from the clause structures in play; but from the perspective of system, the oppositions in meaning are comparable.4 Systematically separating system from structure is crucial when comparing and contrasting modalities.

We also feel that further development of Martinec’s pioneering modelling is timely in light of theoretical and descriptive developments in SFL since his work. This has mainly to do with a clearer articulation of the stratification of language as levels of phonology, lexicogrammar and discourse semantics (e.g. Martin 2010; 20112014; Martin and Rose 2007). Martinec’s work draws largely on Halliday’s lexicogrammatical systems (those proposed in Halliday 1985), the same systems which inspired Kress and van Leeuwen’s (1996) breakthrough. We have found it illuminating to further develop this work by drawing on ideational, interpersonal and textual systems at the level of discourse semantics (ideation, connexion, negotiation, appraisal, identification and periodicity). Work on appraisal (the language of evaluation) in particular (Martin and White 2005) has a number of ramifications for models of paralanguage, especially in relation to the relative marginalisation of these resources in canonical work by Calbris (Calbris 2011), Kendon (Kendon 1997) and McNeill (McNeill 2006).

In this paper we accordingly proceed as follows. In section “Language development (ontogenesis)” we briefly review SFL research on early child language development. We then move on to draw a distinction between behaviour (somasis) and meaning (semiosis), outlining our current framework in sections “Non-semiotic behaviour (somasis)" and "Embodied meaning (semiosis)”. As noted above, for this framework we adopt the term paralanguage to refer to semiosis dependent on language and realised through both sound quality and body language (including facial expression, gesture, posture and movement). We close with a brief discussion of the relations among language, paralanguage and other modalities of communication.

Language development (ontogenesis)

SFL research on language development in young children is a useful starting point for work on paralanguage in two respects. On the one hand, the emergence of the first signs (protolanguage) highlights the issue of what counts as semiosis and what does not. On the other, the realisation of these first signs is multimodal – linguistic and paralinguistic resources are not differentiated at this stage.

Matthiessen (2004, 2007, 2009) reviews the emergence of language and other semiotic systems based on SFL studies of child language development by Halliday (1975, 2003), Painter (1984, 2003) and Torr (1991). These studies show that language develops out of a protolinguistic system in which children draw on sounds, facial expressions and gestures to enact signs. With the emergence of language proper however, these resources become specialised in distinctive ways. Segmental articulation and prosody (rhythm and intonation) are marshalled as the phonology of spoken language.5 But vocal resources such as timbre, tempo, tension and loudness (explored in detail in van Leeuwen 1999) continue as expressive resources, often referred to as sound quality. And gesture, posture and facial expression develop as resources often referred to as body language. As Matthiessen points out, sound quality and body language are then coordinated with language as texts unfold: ...certain interpersonal contrasts in language are realized vocally by contrasts in tone (pitch movement) accompanied by facial contrasts involving eyebrow movements; textual contrasts in deicticity are often accompanied by pointing gestures; talking to babies may involve rounding, pouting lips – a feature that affects the sound but which is also visible; and as detailed studies have shown, there is a complex relationship between addressing somebody in language and gazing at them. (2007: 6–7).

In this paper we will follow Matthiessen (2009: 21–22) in referring to the resources of both sound quality and body language as paralanguage.

Non-semiotic behaviour (somasis)

One basic challenge that has to be faced when working on paralanguage is how to distinguish it from behaviour – separating semiosis from non-semiosis in other words.6 This is of course the challenge faced by specialists in ontogenesis as they track the emergence of protolanguage out of the pre-linguistic interaction, as explored by Trevarthen (Trevarthen 2005). For Halliday and Painter the key criteria are that i. the act in question is interpretable as one of a systematic system of content/expression pairs (i.e. signs with valeur), and ii. the act in question is used on a number of different occasions7 (i.e. not simply iterated in a single interaction). Halliday (1984/2003: 240) for example notes three signs oriented to action in his son’s protolanguage at 8 months of age (Fig. 2). The signs are constituted as the following content/expression pairs: ‘I want it/grasp firmly’, ‘I don’t want it/touch lightly’ and ‘do that with it/touch firmly’. The contrast between semiosis and non-semiosis is evident here, perhaps most clearly in the contrast between touching something lightly (semiosis) and pushing it away (non-semiosis). In Peircian terms8 we might say that the semiosis symbolises the intention of the speaker while the non-semiosis indexes it.
Fig. 2
Fig. 2

Early protolanguage (action systems); from Halliday 1984/2003: 240, Fig. 9

From this point on we will use the term somasis for non-semiotic behaviour, and semiosis for systems of signs. As far as somasis is concerned we have found it useful to draw on Halliday’s proposals for an evolutionary typology of systems (Halliday 1996: 388, Halliday 2005: 67–68). He recognises four orders of complexity, with semiotic systems evolving out of social systems, social systems out of biological ones and biological ones out of physical ones. We have adapted this framework in our classification of somatic behaviour, distinguishing physical activity, biological behaviour and social communion. Physical activity covers material action involving some change in the relationship of one physical entity to another (walking, running, jumping, throwing, breaking, cutting, digging, pulling etc.). Biological behaviour can be divided into changes that restore comfort (sneezing, coughing, scratching, laughing, adjusting garments or hair etc.) and changes that index discomfort (nail biting, fiddling, fidgeting, wriggling, blushing, shivering, crying etc.). Social communion can be divided into mutual perception (sharing gaze, pitch, proximity, touch, smell etc.) and reciprocal attachment (tickling, cradling, holding hands, hugging, stroking, hugging, kissing, mating etc.). These proposals are outlined in Fig. 3.
Fig. 3
Fig. 3

A model of behaviour (somasis)

Trained as we are as linguists and semioticians we are not ourselves in a strong position to further develop this model.9 But we have found it useful to try and compile a range of behaviours that border on semiosis and which can be interpreted by social semiotic animals as indexing purposeful activity. As Halliday and Painter have shown, early protolinguistic semiosis involves a reconstrual of some of these activities as the expression face of signs. And all of the behaviour outlined above has the potential to be used as signs – for example stamping one’s foot in frustration, coughing to remind a meeting of one’s presence, shivering to indicate one is cold, sniffing to object to an odour, kissing on the cheek as a greeting and so on. In these cases there is some degree of deliberation involved, as manifested in the fact that the behaviour will synchronise with the prosodic phonology and turn-taking structure of an interaction and will be responded to as meaningful by co-participants.10

The model of non-semiotic and semiotic behaviour we have developed to this point is outlined as Fig. 4.
Fig. 4
Fig. 4

Somasis and semiosis

Embodied meaning (semiosis)

In their work on intermodal relations in children’s picture books (Painter and Martin 2012; Painter et al. 2013) Painter and her colleagues suggest a model involving degrees of convergence between verbiage and image. The model is organised by metafunction – degrees of concurrence for ideational meaning, degrees of resonance for interpersonal meaning and degrees of synchronicity for textual meaning (for illustrative text analysis see Martin 2008, Painter and Martin 2012). The relevant terminology is presented in Table 2 below.
Table 2

Convergent verbiage/image relations in children’s picture books

Verbiage

Convergence

Image

Ideational

Concurrence

Ideational

Interpersonal

Resonance

Interpersonal

Textual

Synchronicity

Textual

We have drawn on this terminology to deal with two dimensions of the relation between language and paralanguage introduced by Cléirigh as ‘linguistic body language’ and ‘epilinguistic body language’.11 The basic distinction here is between paralanguage that is in tune with (resonance) or in sync with (synchronicity) the prosodic phonology (i.e. rhythm and intonation) of spoken language on the one hand and on the other paralanguage that expresses meanings made possible by having language – in Cléirigh ‘s terms linguistic vs epilinguistic body language respectively. We have preferred a more transparent terminology, derived from Table 2, with phonologically convergent paralanguage referred to as sonovergent and semantically convergent paralanguage as semovergent. This revised terminology is outlined in Table 3.
Table 3

Sonovergent and semovergent paralanguage

Cléirigh (unpublished)

Revised terminology

Linguistic body language

Sonovergent paralanguage (phonologically convergent)

Epilinguistic body language

Semovergent paralanguage (semantically convergent)

Sonovergent and semovergent paralinguistic systems will be introduced in turn below, drawing on examples from a Youtube video titled, ‘Let’s Talk. | Random Chatty Vlog’, used here with the presenter’s permission (https://youtu.be/YRx-zDoPbVw). A ‘vlog’ (derived from ‘blog’12) is a video in which a user recounts, or presents, some form of personal activity (e.g. a ‘day in the life’ vlog where the user shows highlights from their activity over a day). The following is the description accompanying the video posted by the vlogger:

Grab a cup of coffee and a snack. Let's just sit down and talk today. I chat about annoying people who follow me in the parking lot, my kids begging for food all summer, my hair, feet, Youtube...etc. I have no trouble rambling on. If you like this, PLEASE give this video a thumbs up so I know! I want to know what you guys like seeing. Thank you for watching! Subscribe so you don't miss another video. I post every Monday, Tuesday and Thursday at 2pm EST.

A full transcription, including intonation13 analysis, of the hair dye episode of this Vlog from which we take most of our examples is provided as Appendix 1.

Sonovergent paralanguage

Sonovergent paralanguage converges with the prosodic phonology of spoken language (Halliday 1967, 1970; Halliday and Greaves 2008; Smith and Greaves 2015). From an interpersonal perspective, it resonates with tone and involves a body part (e.g. eyebrows or arms) moving up and down in tune with pitch movement in a tone group (tone and marked salience). From a textual perspective it involves a body part14 (e.g. hands, head) beating in sync with the periodicity of speech15 – which might involve beats aligned with a salient syllable of a foot, the tonic syllable of a tone group, or a gesture co-extensive with a tone group (i.e. in sync with tonality, tonicity or rhythm). An outline of this sonovergent paralanguage is presented in Table 4.
Table 4

Sonovergent paralinguistic systems

Verbiage

Sonovergent

Paralanguage (‘prosodic’)

Interpersonal:

Tone

Resonance

Interpersonal:

In tune

Textual:

Tonality, Tonicity, Rhythm

Synchronicity

Textual:

In sync

The phonological system of tone is realised through pitch movement. In example (1) the vlogger’s eyebrows move up in tune with the rising tone (tone 2) on the syllable prev (Fig. 5).
Fig. 5
Fig. 5

Example 1

The phonological system of tonality organises spoken language into waves of information called tone groups, with one salient syllable carrying this tone movement. Gestures tend to be co-extensive with this periodic unit. In examples (2) and (3) the vlogger makes a sweeping right-to-left16 gesture referencing past time; the gestures unfold in sync with the temporal extent of the tone group (Figs. 6 and 7).
Fig. 6
Fig. 6

Example 2

Fig. 7
Fig. 7

Example 3

The phonological system of tonicity highlights a peak of informational prominence by positioning the major pitch movement of a tone group (its tone) on one or another of its salient syllables (its culminative salient syllable in the unmarked case). In example (4) the vlogger claps on the syllable realising the tone group’s major pitch movement – hair (Fig. 8).
Fig. 8
Fig. 8

Example 4

The phonological system of rhythm is realised in English through the timing of the salient syllables beginning each foot (relatively equal timing between salient syllables in a stress-timed language). In the following example, the vlogger beats with her hands in time with the salient syllables of the feet / not /, / find the / and / hair dye that I /. The last of these in fact syncs with the tonic syllable hair (Fig. 9).
Fig. 9
Fig. 9

Example 5

Salient syllables other than the tonic syllable can be given additional prominence through various means. In the following sequence the vlogger’s pitch on the first tone group is unusually high, and contrasts with the descending lower pitch of the following tone group (a sing/song effect).

//3 hopefully / next ↑time I will

//1 get my / ↓hair colour / back //

And the vlogger’s eyebrows move up in tune and in sync with the higher pitch on / hopefully /, before lowering again by the end of the following tone group (Fig. 10).
Fig. 10
Fig. 10

Example 6

The same sing/song effect follows on and culminates this section of the vlog, with a high pitch on the tonic syllable / now // contrasting with the low pitch on / do //. The vlogger’s eyebrows once again move up and down in tune and in sync with the contrasting pitch salience (this time on contrasting tonic syllables).

//3 [handclap] / um /but for / ↑now

//3 this will / ↓do //

These rhythmic in-tune gestures reinforce the attitudinal import of the rhythm and tonicity (cf. section “Evaluation (interpersonal semovergent paralanguage)” below).

The contribution of sonovergent paralanguage to the vlog is interrupted in tone group 15 of Appendix 1, suspended for tone groups 16–19, and resumes for tone group 20 – to allow for a somatic phase during which the vlogger uses her left hand to scratch her right arm. This phase unfolds as follows:

//3 lighter than it / was a few /days ago

//1 ^ but /yeah it’s

//1 such a / bummer and then I

//2 went to / Target

//3 ^ like / two days / later and there / was a //

The vlogger stops looking at her followers and begins scratching in the final foot of tone group 15 (Fig. 11).
Fig. 11
Fig. 11

Example 7

The scratching and absence of gaze continues for two tone groups (Figs. 12 and 13).
Fig. 12
Fig. 12

Example 8

Fig. 13
Fig. 13

Example 9

Gaze resumes in the final foot of tone group 18 (Fig. 14).
Fig. 14
Fig. 14

Example 10

And the vlogger then resumes gesturing (Fig. 15).
Fig. 15
Fig. 15

Example 11

It is interesting to note that the vlogger does not scratch in sync with the rhythm, tonicity and tonality of the text; the scratching lasts for two and a half tone groups, and does not match the timing of salient and tonic syllables. But the paralanguage remains in sync, stopping precisely at the tonic syllable of tone group 15 (/ days ago //), resuming with a smile precisely at the tonic syllable of tone group 18 (/ Target //) and resuming with gesture precisely at the beginning of tone group 19. This indicates that synchronicity with prosodic phonology can function as a demarcating criteria for distinguishing somatic from semiotic behaviour.

Gesture converging with meaning (semovergent paralanguage)

Semovergent paralanguage is convergent with the lexicogrammar and discourse semantics of spoken language (its content plane). We adopt a discourse semantic perspective on these meaning making resources here (Martin and Rose 2007). Ideational paralanguage is ‘mimetic’, concurring with ideation and connexion17 systems; interpersonal paralanguage is ‘expressive’, resonating with negotiation and appraisal systems; and textual body language is ‘deictic’, syncing with identification and periodicity18 systems. These convergences are outlined in Table 5.19
Table 5

Converging paralinguistic and discourse semantic systems

Verbiage

Sonovergent

Paralanguage (‘prosodic’)

Ideational:

Ideation/Connexion

Concurrence

Ideational:

‘mimetic’

Interpersonal:

Tone

Resonance

Interpersonal:

‘expressive’

Textual:

Tonality, Tonicity, Rhythm

Synchronicity

Textual:

‘deictic’

Representation (ideational semovergent paralanguage)

From an ideational perspective we need to take into account how spoken language combines entities, occurrences, qualities and spatiotemporal circumscriptions as figures (ideation), and how these figures are connected to one another (connexion). Semovergent paralanguage supports these resources with hand shapes, which potentially concur with entities, and hand/arm motion, which potentially concurs with occurrences; the hand/arm motion is optionally directed, potentially concurring with spatiotemporal direction (to/from there in space, to/from then in time). We say “potentially concurring” because ideational paralanguage can be used on its own, without accompany spoken language; see the discussion of mime in section "Multidimensionality (multiplying meaning)" below.

By way of illustration we now move to the next section in the vlog, which concerns a visit to the vlogger’s dermatologist (for treatment for granuloma). The sequence of figures we are interested in unfolds verbally in tone groups as follows (for the complete anecdote see Appendix B):

// and so the dermatologist um took like this needle

// and under each like bump

// and injected this like steroid

// and like it all bubbled up //

From the perspective of language, this sequence makes explicit four entities (dermatologist, needle, bump, steroid). The paralanguage uses handshape to concur with two of these (needle and bump) (Fig. 16). The ‘needle’ is first rendered as a tiny pointed entity the vlogger holds between thumb and index finger, and then with the hand shape used for holding a syringe. The ‘bump’ is not actually visualised until the fourth tone group, where it renders the shape of the steroid bubbling up. As we can see, the meanings construed in language and paralanguage can either correspond or complement one another. In terms of commitment (i.e. the amount of meaning specified across semiotic modes; Martin 2010, Painter et al. 2013), the dermatologist and steroid are committed in the language but not the paralanguage; but the needle is more delicately committed in the paralanguage as a tiny pointed entity and then as a syringe. And the paralinguistic commitment of the bump in fact takes place two tone groups after it is committed verbally.
Fig. 16
Fig. 16

Example 12

Turning from a static to a dynamic perspective, the language of this sequence makes explicit three occurrences  (took, injected, bubbled). The paralanguage concurs with these, and in addition uses six rapid piercing gestures to make explicit the occurrences implied by the second tone group (Figs. 17 and 18).
Fig. 17
Fig. 17

Example 13

Fig. 18
Fig. 18

Example 14

In each case the entity indicated by the hand shape is in motion, as the dermatologist picks the needle up, pierces the bumps, injects the steroid and the bump bubbles up (Figs. 19 and 20).
Fig. 19
Fig. 19

Example 15

Fig. 20
Fig. 20

Example 16

As with imagic sequences in film, animations, graphic novels, comics, cartoons and picture books, the gesture sequence does not make explicit the conjunctive relations between figures. These have to be abduced (Bateman 2007) from the sequence and concurring language. In the case of this sequence conjunctive relations of time and cause are not made explicit linguistically either; only the additive linker and, is used. A defeasible reading of the sequence is offered below.

// and so the dermatologist um took like this needle

(temporal sequential)

// and under each like bump

(temporal overlapping)

// and injected this like steroid

(causal)

// and like it all bubbled up //

As noted above, for this paralinguistic sequence handshape and motion are combined. In other cases handshapes occur on their own (Fig. 21). In the following sequence our vlogger concentrates on the size of the snack she has given her children, without setting the bowl in motion.
Fig. 21
Fig. 21

Example 17

// then they had a snack I

// gave them each a bowl

// like a heaping bowl full of Chex Mix and applesauce squeeze //

Motion can also occur on its own, without a handshape concurring with an entity (Fig. 22). For example the vlogger uses a circular hand motion (two rotations) concurrent with the tone group // tried washing it out and it's //.
Fig. 22
Fig. 22

Example 18

Motion can also be used to support direction in space or time (Fig. 23). Above in section “Sonovergent paralanguage” we illustrated two examples of hands sweeping right-to-left towards the past, concurring with the tone groups // bought previously when I // and // loved the first time //. These contrast with left-to-right movement towards the future, concurrent with // hopefully next time I will //. This motion to the right is reinforced by a pointing gesture, which we discuss in section “Information flow (textual semovergent paralanguage)” below (as textual semovergence).
Fig. 23
Fig. 23

Example 19

Evaluation (interpersonal semovergent paralanguage)

From an interpersonal perspective we need to take into account how spoken language inscribes attitudes, grades qualities and positions voices other than the speaker’s own (appraisal). We also need to account for how speakers exchange feelings, greetings, calls for attention, information and goods & services in dialogue (negotiation). Semovergent paralanguage potentially resonates with appraisal resources through facial expression, bodily stance, muscle tension hand/arm position and motion (Hood 2011, Ngo n.d. in press) and voice quality. Whereas spoken language can make explicit attitudes of different kinds (emotional reactions, judgements of character and appreciation of things), paralanguage can only enact emotion. A further interpersonal restriction (as suggested by Cleírigh), setting aside emblems (discussed in Section “Emblems” below; Kendon 2004, McNeill 2012), is that semovergent paralanguage cannot be used to distinguish move types in dialogic exchanges (although sonovergent paralanguage can of course support tone choice in relation to these moves).

Paralanguage deploys facial expression and bodily stance to share attitude (Fig. 24). In the following example our vlogger nuances her appreciation (exciting) of a neighbourhood get-together she has dressed up for with raised eyebrows, lopsided mouth expression20 (which we might read as indicating that some followers might not find it all that exciting).
Fig. 24
Fig. 24

Example 20

As outlined by Martin and White (2005) attitude may not be explicitly inscribed in language, but invoked by ideational choices a speaker expects a reaction to. We introduced an example of this in section “Sonovergent paralanguage” above, as the vlogger introduces the good news that her hair dye is back in stock at Target. Her smiling face makes explicit the affect that her language does not (Fig. 25).
Fig. 25
Fig. 25

Example 21

A good example of a combined face and body commitment of affect in the vlog we are drawing our examples from comes as the vlogger is complaining about being hassled for her parking spot before she is ready to leave. The relevant tone groups are presented below, and we will return to this example in our discussion of mime in section “Emblems” below (for a complete transcription of this narrative see Appendix C). At this point we are simply interested in the way the vlogger’s facial expression and arm position are used to express the hassler’s exasperation (Fig. 26).
Fig. 26
Fig. 26

Example 22

// some guy was sitting there

// and there was cars behind him

// and he was like

// [mimics man’s expression]

// [mimics man’s gesture] like

// waving me out //

Turning to graduation, as noted by Hood (2011) the size of hand shapes and the range of hand/arm motion can be used to support graded language. In the following example the sweeping extent of the hand/arm motion resonates with the large quantity of hair dye in stock (whole stack) (Fig. 27).
Fig. 27
Fig. 27

Example 23

The most striking example of intensification in the hair colour episode occurs when the vlogger uses whole body movement to enact her reaction to how dark her hair is. She throws her head back and leans back as her arms move rise up – literally overwhelmed with emotion (Fig. 28).
Fig. 28
Fig. 28

Example 24

Alongside paralanguage of this kind converging with force, Hood notes the potential for precise hand shapes and muscle tension to resonate with focus. In the following example, introduced in section “Representation (ideational semovergent paralanguage)” above, the vlogger tightens her grip on the tiny virtual needle she is holding and frowns slightly in concentration as she role plays the precision involved in the dermatologist piercing her bumps (Fig. 29).
Fig. 29
Fig. 29

Example 25

Hao and Hood (in press). draw attention to the use of what they call de-centering postures to soften focus, using the example of a shoulder shrug converging with fairly non-contractile in a biology lecture. The paralinguistic generalisation here would appear to be loss of equilibrium – e.g. asymmetrical facial expression, out of kilter posture or a rotating prone hand (interpretable as between prone and supine). Clear examples in our data are the faces the vlogger pulls as she struggles to name her skin condition in the second tone group below, the second of which is accompanied by two shakes of her head (Fig. 30).
Fig. 30
Fig. 30

Example 26

// anyway

//^ ‘it was / some / granuloma /: / /^ / something

// I don’t know- it’s / called- it’s

// some sort of / skin thing //

Turning to engagement, Hood notes the significance of hand position as far as supporting the expansion and contraction of heteroglossia is concerned – with supine hands opening up dialogism and prone hands closing it down. In the following example the vlogger’s supine hands converge with the modalisation probably, reinforcing acknowledgement of the viewers voice (Fig. 31).
Fig. 31
Fig. 31

Example 27

Two moves later the hands flip over to prone position in support of the negative move shutting down the expectation that the vlogger was in control of the new colour of her hair (Fig. 32).
Fig. 32
Fig. 32

Example 28

Voice quality was noted in section “Sonovergent paralanguage” above in relation to the sing/song pitch (high then low) movement the vlogger uses in her last four tone groups to close down her hair dye narrative. From the perspective of appraisal the sound quality resonates with her resignation. Further work on this interpersonal aural dimension of paralanguage, drawing on van Leeuwen 1999, is beyond the scope of our current research.21

Information flow (textual semovergent paralanguage)

From a textual perspective22 we need to take into account how spoken language introduces entities and keeps track of them once there (identification) and how it composes waves of information in tone groups, clauses and beyond (periodicity). Semovergent paralanguage potentially supports these resources with pointing gestures and whole body movement and position.

As far as pointing deixis is concerned we can return to the examples contrasting past and future in sections “Sonovergent paralanguage" and "Representation (ideational semovergent paralanguage)” above. Alongside motioning to the past the vlogger’s hand points there. And alongside motioning to the future both the vlogger’s index fingers point there (Figs. 33 and 34).
Fig. 33
Fig. 33

Example 29

Fig. 34
Fig. 34

Example 30

Fig. 35
Fig. 35

Example 31

As far as longer wave lengths of information flow are concerned,23 our vlogger is seated and so whole body movement from one location to another is not a factor (as it would be for example for a lecturer roaming to and fro across a stage; cf. Hood 2011). As noted in sections “Sonovergent paralanguage and Information flow (textual semovergent paralanguage)” above however the vlogger does end the episode with a contrasting high then lowered pitch (Fig. 35). The higher pitch penultimate tone group begins rhythmically speaking with a handclap foot and then a foot comprising the ‘filler’ / um /.
Fig. 36
Fig. 36

Example 32

This is followed by the low pitch tone group; the vlogger is winding down. Following this there is a suspension of both language and paralanguage as her the vlogger’s eyes shut and her head slumps forward (Fig. 36).

The preceding episode to the one we are using to explore sonovergence here ends in a similar way (lowered pitch, with eyes shut, head down) (Fig. 37). So shutting down language and paralanguage and handing over to somasis is clearly a strategy for punctuating longer waves of discourse. It is at these points that the vlogger cuts from one filmic segment to the next (as she thinks of something more to say).
Fig. 37
Fig. 37

Example 33

Multidimensionality (multiplying meaning)

The sonovergent and semovergent paralinguistic systems discussed thus far are outlined in Fig. 38 (including cross-references to Cléirigh’s original terminology). Although presented as a simple taxonomy, all five subtypes of paralanguage can combine with one another in support of a single tone group (Fig. 38).
Fig. 38
Fig. 38

Sonovergent and semovergent paralanguage

Several examples of multiple dimensions of paralanguage converging on the same tone group were in fact presented above (for example, the combination of motion towards the future and pointing deixis in Example (19) of section “representation (ideational semovergent paralanguage)”). It is probably safe to say that whenever semovergent paralanguage is deployed, it will be coordinated with tonality, tonicity and rhythm; this is tantamount to arguing that semovergence implies sonovergence. Sonovergent paralanguage on the other hand can be deployed without semovergence, through gestures in tune with or in sync with prosodic phonology (but no more).

An important exception to these principles is what is commonly referred to as mime. In terms of our model mime is semovergent paralanguage that does not accompany language, an apparent contradiction in terms. To explore this further we will return to the miming segment in the parking lot narrative referred to above. The vlogger sets up what happened as follows:

Oh another thing that has been really annoying this summer is you know when you go to a parking lot and it’s a busy place. You get in your car and you don’t necessarily want to leave immediately. Like you might want to- I might want to have Henry test his blood sugar, give the kids snacks. Or if we were at the pool, like change or look at my phone or send a text message or whatever. It drives me crazy when a car is like sitting there following you and then they just wait for you to leave. I cannot stand that. And that has happened so many times. And I was just at the Mall of America and I got back to my car and I went into-. And I met up with a Kimmy from the Dodge family and I went to- I wanted to like Instagram a picture of us and FaceBook whatever. And as I was doing that I- I had...

This is followed by a specific parking lot incident, presented in tone groups below.

// just got in my car

// got my phone and

// as I was doing that

// some guy was sitting there and

// there was cars behind him and

// he was like

// [mimics man’s paralanguage]

// [mimics man’s paralanguage] like

// waving me out and

// I was so:: upset like

// I immediately got up

// put my phone down

// I immediately drove away a bit

// I wasn’t even thinking I

// shouldn’t have done that I

// should not have done that

// but it was just like “What!”

// There’s a guy sitting there

// waving and

// angry at me be-

// cause I was sitting in my car //

In terms of the tonality of this sequence, there are two miming segments where tone groups might have been. For each, the vlogger mimes the paralanguage of her parking spot assailant. In the first slot she mimes his interpersonal attitude paralanguage, as discussed in section “Evaluation (interpersonal semovergent paralanguage)” above (Fig. 39).
Fig. 39
Fig. 39

Example 34

In the second she mimes his ideational motion paralanguage as she twice gestures leaving (the second time including a textual pointing gesture) (Figs. 40 and 41).
Fig. 40
Fig. 40

Example 35

Fig. 41
Fig. 41

Example 36

The third time his motion gesture is mimed in fact concurs with language (Fig. 42).
Fig. 42
Fig. 42

Example 37

As we can see, the two miming segments are heavily co-textualised by language that makes explicit what is going on. The orientation to the narrative introduces the recurrent problem of someone following the vlogger in a parking lot and waiting for her to leave. The miming segments are themselves introduced with the incomplete tone group // he was like... //, with a missing tonic segment. The vlogger then mimes the expected information, before making it linguistically explicit in a tone group converging with the third iteration of the gesture.

Setting aside pantomime (the ‘art of silence’ Marcel Marceau referred to it), we can predict that co-textualisation of this kind is a generalisable pattern as far as semovergent paralanguage in the absence of language is concerned. What the moment of mime does not provide as far as language is concerned, the immediately preceding and following co-text does provide. So the convergent nature of semovergent paralanguage is clear.

Emblems

It remains to introduce our treatment of what Kendon (Kendon 2004) refers to as emblems, drawing on Ekman and Friesen (1969). Included here are gestures such as thumbs up or thumbs down (as praise or censure), index finger touching lips (for ‘quiet please’), hand cupped over ear (for ‘I can’t hear’), middle finger vertical (for ‘get fucked’) and so on. Our vlogger uses one of these gestures to introduce the first of her explanations as to why her hair is darker than usual – raising her index finger as an emblem for the numeral ‘1’ (Fig. 43).
Fig. 43
Fig. 43

Example 38

These gestures differ from the semovergent ones illustrated thus far in critical ways (cf. McNeill 2012: 7–10). For one thing they commit very specific meanings and can be readily recognised without accompanying co-text. As part of this specificity they can enact moves in exchange structure on their own – e.g. the statements and requests noted above, alongside greetings and leave-takings (hand waving), calls (beckoning gestures), agreement (nodding head), disagreement (shaking head), challenges (upright palm facing forward for ‘stop’) and so on. For another they are much more easily called to consciousness, as the first thing that comes to mind when someone mentions gesture. And in this regard they are often commented on as culturally specific (e.g. the difference between an Anglo supine hand beckoning gesture and its Filipino prone hand equivalent). In both respects emblems contrast with common-sense dismissals of the paralanguage (introduced in sections “Sonovergent paralanguage" and "Gesture converging with meaning (semovergent paralanguage)”) as idiosyncratic (although none of us has any trouble successfully interpreting another speaker’s sonovergent and semovergent systems). From the perspective of the sign language of the deaf, emblems most strongly resemble signs; they are expression form gestures explicitly encoding meaning. Similarly, from the perspective of character based writing systems (such as those of Chinese), emblems most strongly resemble characters (but gestured rather than scribed).

This indicates that emblems are better treated as part of language than as a dimension of paralanguage. The relationship we are emphasising between emblems and alternative expression form systems is outlined in Fig. 44, using the words zero, one, two, three, four and five as examples. These words can be alternatively expressed in English through segmental phonology (e.g. /tuw/), graphological characters (e.g. ‘2’) or hand gestures (index and middle finger vertical).
Fig. 44
Fig. 44

Alternative realisations of expression form (gesture, graphology, phonology)

An outline of the place of emblems in our overall system in presented in Fig. 45. Rather than treating them as a dimension of paralanguage, we have moved them over to language proper, as an alternative manifestation of its expression form.
Fig. 45
Fig. 45

Emblems as gestural signs

Intermodality

In this paper we have outlined a model distinguishing behaviour from meaning (somasis vs semiosis), and within semiosis, language from paralanguage. Paralanguage itself was then divided into sonovergent and semovergent systems according to their convergence with either the expression plane or content plane of language. Sonovergent systems enact interpersonal meaning in tune with and compose textual meaning in sync with the prosodic phonology of language; semovergent systems construe ideational meaning, enact interpersonal meaning and compose textual meaning convergently with the discourse semantics of language (and its realisation through lexiogrammar).

Compared to other modalities of communication, paralanguage has a distinctive relation to language in that it is coordinated with prosodic phonology. This is obviously true, by definition, for sonovergent paralanguage. But semovergent paralanguage is also coordinated with tonality, tonicity rhythm and tone, since gestures, facial expression, bodily stance and sounds unfold in measures of time converging with units of rhythm and intonation. Even brief episodes of mime follow this principle, filling in for ‘missing’ tonic segments or tone groups as a whole. Alongside this expression form of temporal dependency, paralanguage is dependent on the content form of language because of its inherent generality. Semovergent paralanguage typically commits meaning far less specifically than spoken language can; instantiations are by and large interpretable with respect to what is said. With respect to these two dependencies, the prefix para- (understood as ‘beside’) is appropriate.

But what about the stem -language which para- is prefixed to? The drift of consensus in gesture studies, as reviewed and promoted by Fricke (Fricke 2013) appears to be towards treating aspects of what we have been calling paralanguage here as part of language (in fact as part of grammar in Fricke’s work). From the perspective of SFL this argues for a re-interpretation of the taxonomy in Fig. 44 above as Table 6 below, with paralanguage positioned not alongside language but as part of its expression form. In this model, the content form of face-to-face linguistic communication can be realised as phonology (of spoken language) or sign (including the sign languages of deaf communities and the ‘emblems’ of hearing ones), plus in both cases sonovergent and semovergent paralanguage; and for many languages we have a graphological system used for written communication. This leaves us with the terminological challenge of how best to name the sound quality and gestural resources we have been calling paralanguage in this paper (since they wouldn’t be para- anymore); we will not attempt to improve on our usage here.
Table 6

Paralanguage as expression form

Content form

Expression form

Discourse semantics

Lexicogrammar

Phonology

Sign

Paralanguage

Graphology

This of course makes research into the relation between language and paralanguage an interesting case study as far as research into intermodality in general is concerned, possibly helping to clarify some of the theoretical and descriptive challenges posed in Martin 2011.

Our evolving work on these dependencies can be tracked through Martin et al. (2010), Hood (2011), Martin (2011), Martin, Zappavigna, Dwyer, and Cléirigh (2013) Martin and Zappavigna, 2018, Zappavigna and Martin (2018), and Hao and Hood (in press). From the perspective of SFL the most pertinent work on relations between modalities to compare with these studies is Painter et al. 2013 (on language and image in children’s picture books). Beyond these initiatives, multimodal discourse analysis research is best guided by Bateman et al. (2017).

As we stressed at the beginning of the paper building models of intermodality is facilitated if the descriptions of distinct modalities are informed by the same theoretical principles; and this is important for applications. Work in educational linguistics, for example Hood (2011) and Hao and Hood (in press), regularly has to deal with the interaction of language, paralanguage and imaging on Power Point slides. And for forensic linguistics, for example Martin and Zappavigna (2013) and Martin and Zappavigna, 2018, Zappavigna and Martin (2018), language and paralanguage interact with the semiotics of the location of the legal proceedings (which are very different for courtrooms and Youth Justice Conferences). The model of intermodal convergence (ideational concurrence, interpersonal resonance and textual synchronicity) presented in Table 2 above is far easier to operationalise when each of the modalities involved is interpreted from the perspective of SFL.

Our model of paralanguage might also prove of interest as a contribution to the growing field of interactional linguists (Ochs et al. 1996; Fox et al. 2013; Couper-Kuhlen and Selting 2001, 2018). These linguists see language structure as an emergent phenomenon which can only be understood in relation to the use of language in dialogue, and they draw heavily on Conversation Analysis (CA) in their research. This brings paralanguage and other modalities of communication into the picture as far as our understanding of language is concerned (cf. Heath and Luff 2013). SFL’s perspectives on multimodality creates an opportunity for linguistics to make a stronger contribution to this transdisciplinary exercise (Martin forthcoming).

Footnotes
1

Cléirigh ‘s paper is titled ‘Gestural and postural semiosis: a systemic-functional linguistic approach to ‘body language’; his model informs work published as Zappavigna et al. (2010), Hood (2011), Martin (2011), Martin et al. (2013), Martin & Zappavigna, 2018, Zappavigna and Martin (2018).

 
2

We acknowledge here the contributions of our research group (Susan Hood, Thu Ngo, Clare Painter and Brad Smith). For an elaboration of our work see Ngo et al. forthcoming.

 
3

A comparable multifunctional perspective has been proposed (Kok 2016; Kok et al. 2016) within the framework of Functional Discourse Grammar (FDG). In contrast to SFL, FDG recognises two rather than three ‘levels’ – representational and interpersonal (the later concerned with SFL’s interpersonal and textual meaning); and within these levels DFG involves an alternative set of semantic parameters.

 
4

Kendon in fact uses this example to argue that the gestures differentiate two meanings of the verb slice. We would argue on the other hand that the gestures are converging with the meaning of two different clause structures (distinguishing two different actions): the hunter…sliced the wolf’s head off involves a phrasal verb (cf. the hunter sliced off the wolf’s head) affecting the wolf, whereas the hunter…sliced the Wolf’s stomach open involves the non-phrasal verb slice, again affecting the wolf, but this time with the effect of the action specified (as the resulting attribute open). Kendon’s misreading reflects the logocentric (word-centred) bias in much of the gesture literature.

 
5

Just as gesture and facial expression are marshalled as the expression plane of the sign language of deaf communities (cf. Johnston 2013, 2018 on the use of paralanguage alongside signing in sign language).

 
6

Cf. Martinec (e.g. Martinec 2000b) who includes non-semiotic behavior in his modeling.

 
7

Halliday and Painter set the minimum number of occasions required for recognition of a sign at 3.

 
8

For an introductory overview of Peirce’s model of signs see https://plato.stanford.edu/entries/peirce-semiotics/.

 
9

For exemplary forays into this realm of inquiry see Martinec (1998, 2000a, 2000b, 2001, 2004), who models somasis as if it was semiosis, drawing on SFL theory to do so.

 
10

To put this another way, we are arguing that the behaviours outlined in Figure 3 can be treated as paralinguistic or not depending on whether or not they are negotiated as meaningful in interaction. We also need to acknowledge that what we are calling somatic behaviour has the potential to be imbued with cultural norms (e.g. a style of walking, norms for coughing or spitting etc.); these need to be taken into account in future work on somatic behaviour.

 
11

Cléirigh’s dimension of protolinguistic body language has been subsumed in our model as subtypes of somasis and interpersonal semovergent paralanguage. This avoids the problem of using the term protolinguistic for a paralinguistic system making meaning alongside language (protolanguage, as initial emergent semiosis, by definition cannot accompany language); and it makes room for paralinguistic systems enabled by the discourse semantic system of appraisal, a system of meaning beyond the scope of Cléirigh's study.

 
12

A ‘blog’ (truncated form of ‘weblog’) is a website comprised of posts displayed in reverse chronological order. Most often they involve personal diary-style entries composed by individuals; corporations and organisations may also incorporate blogs into their online material.

 
13

We are indebted to Brad Smith for his unfailing support for this analysis.

 
14

For wave lengths longer than a tone group whole body motion is involved.

 
15

SFL’s interpretation of the ‘beats’ in Figure 1 was first pursued by Martinec (2000a, 2002), but is anticipated in Kendon 1972 (who makes reference to Halliday’s work on intonation and rhythm).

 
16

Lateral gesture movement will be named from the perspective of the vlogger.

 
17

The term connexion has been adopted from Hao n.d.. (in press) in place of conjunction, to reinforce the differentiation of discourse semantic and lexicogrammatical terminology.

 
18

Semovergent synchronicity is concerned with the syncing of paralanguage with periodic structure composed above and beyond prosodic phonology.

 
19

Our discourse semantic perspective distinguishes our work from Martinec (1998, 2000a, 2000b, 2001); our metafunctional perspective distinguishes our classification from the work underpinning Kendon’s Continuum (Figure 1).

 
20

The ‘out of kilter’ mouth here can be interpreted as soft focus, converging with kind of.

 
21

We also need to acknowledge that a metalanguage for facial expression, in some sense comparable in specificity to SFL work on attitude in the appraisal framework, remains to be developed.

 
22

Martinec (1998) interprets textual meaning as realized through cohesion, following Halliday and Hasan (1976); here we follow Martin (1992) who reinterprets cohesion as discourse semantics, organised metafunctionally in Martin and Rose (2007) as ideational resources (ideation, connexion), interpersonal resources (negotiation, appraisal) and textual resources (identification, periodicity).

 
23

van Leeuwen (1985, 1992) and Martinec (2002) argue that SFL’s phonological hierarchy can be pushed up several wave lengths beyond the tone group and their work suggests that higher level rhythm would converge with higher level periodicity in Martin's framework.

 
24

For transcription conventions see Halliday and Greaves (2008).

 

Declarations

Acknowledgements

An earlier abridged version of this paper was published in Chinese in the journal Contemporary Rhetoric as Martin and Zappavigna (2018).

Authors’ contributions

Jointly constructed article; equal share. Both authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
University of Sydney, N366 John Woolley Building, Sydney, NSW, 2006, Australia
(2)
School of Arts and Media, University of New South Wales, Sydney, NSW, 2952, Australia

References

  1. Bateman, J., J. Wildfeuer, and T. Hiippala. 2017. Multimodality: Foundations, research and analysis – a problem-oriented introduction. Berlin: De Gruyter Mouton.View ArticleGoogle Scholar
  2. Bateman, J.A. 2007. Towards a grande paradigmatique of film: Christian Metz reloaded. Semiotica 2007 (167): 13–64.View ArticleGoogle Scholar
  3. Caffarel, A., J.R. Martin, and C.M.I.M. Matthiessen, eds. 2004. Language typology: A functional perspective. Amsterdam: Benjamins (Current Issues in Linguistic Theory).Google Scholar
  4. Calbris, G. 2011. Elements of meaning in gesture. Amsterdam: Benjamins (Gesture Studies 5).View ArticleGoogle Scholar
  5. Couper-Kuhlen, E., and M. Selting. 2001. Studies in interactional linguistics. Amsterdam: Benjamins.Google Scholar
  6. Couper-Kuhlen, E., and M. Selting. 2018. Interactional linguistics studying language in social interaction. Cambridge: Cambridge University Press.Google Scholar
  7. Ekman, P., and W.V. Friesen. 1969. The repertoire of nonverbal behavioural categories: Origins, usage and coding. Semiotica 1: 49–98.View ArticleGoogle Scholar
  8. Fox, B.A., S.A. Thompson, C.C. Ford, and E. Couper-Kuhlen. 2013. Conversation analysis and linguistics. In The handbook of conversation analysis, ed. J. Sidnell and T. Sirens, 726–740. London: Blackwell.Google Scholar
  9. Fricke, E. 2013. Towards a unified grammar of gesture and speech: A multimodal approach. In Body – language – communication: An international handbook on multimodality in human interaction (handbooks of linguistics and communication science 38.1), ed. C. Müller, A. Cienki, E. Fricke, S.H. Ladewig, D. McNeill, and S. Teßendorf, 733–754. Berlin: De Gruyter Mouton.Google Scholar
  10. Halliday, M.A.K. 1967. Intonation and grammar in British English. The Hague: Mouton.View ArticleGoogle Scholar
  11. Halliday, M.A.K. 1970. A course in spoken English: Intonation. London: Oxford University Press.Google Scholar
  12. Halliday, M.A.K. 1975. Learning how to mean: Explorations in the development of language. London: Edward Arnold (Explorations in Language Study).View ArticleGoogle Scholar
  13. Halliday, M.A.K. 1984. Language as code and language as behaviour: A systemic-functional interpretation of the nature and ontogenesis of dialogue. In R. Fawcett, M.A.K. Halliday, S.M. Lamb & A. Makkai .The Semiotics of language and culture: Vol 1: Language as social semiotic. London: Pinter. 3–35. [reprinted in Halliday 2003. 227–250].Google Scholar
  14. Halliday, M.A.K. 1985. An introduction to functional grammar. London: Arnold.Google Scholar
  15. Halliday, M.A.K. 1996. On grammar and grammatics. In Functional descriptions: Theory into practice, ed. R. Hasan, C. Cloran, and D. Butt, 1–38. Amsterdam: Benjamins.Google Scholar
  16. Halliday, M.A.K. 2003. The Language of early Childhood Collected works v.4. Ed. J.J. Webster. London: Continuum.Google Scholar
  17. Halliday, M.A.K. 2005. On matter and meaning. Linguistics and the Human Sciences 1 (1): 59–82.Google Scholar
  18. Halliday, M.A.K., and W.S. Greaves. 2008. Intonation in the grammar of English. London: Equinox.Google Scholar
  19. Halliday, M.A.K., and R. Hasan. 1976. Cohesion in English. London: Longman.Google Scholar
  20. Hao, J. n.d. in press. Nominalisations in scientific English: a tristratal perspective. Functions of Language.Google Scholar
  21. Hao, J. & S. Hood. n.d. in press Valuing science: The role of language and body language in a health science lecture. Journal of Pragmatics (Special edition, edited by D. Banks & E. Di Martino). in pressGoogle Scholar
  22. Heath, C., and P. Luff. 2013. Embodied action and organizational activity. In The handbook of conversation analysis, ed. J. Sidnell and T. Sirens, 283–307. London: Blackwell.Google Scholar
  23. Hood, S. 2011. Body language in face-to-face teaching. In Semiotic margins: Meaning in multimodalities, ed. S. Dreyfus, S. Hood, and M. Stenglin, 31–52. London: Continuum.Google Scholar
  24. Johnston, T. 2013. Formational and functional characteristics of pointing signs in a corpus of Auslan (Australian sign language): Are the data sufficient to posit a grammatical class of ‘pronouns’ in Auslan? Corpus Linguistics and Linguistic Theory 9 (1): 109–159.Google Scholar
  25. Johnston, T. 2018. A corpus-based study of the role of headshaking in negation in Auslan (Australian Sign Language): Implications for signed language typology. Linguistic Typology 22 (2): 185-231. Google Scholar
  26. Kendon, A. 1972. Some relationships between body motion and speech: An analysis of an example. In Studies in dyadic communication, ed. A.W. Siegman and B. Pope, 177–210. New York: Pergamon.View ArticleGoogle Scholar
  27. Kendon, A. 1997. Gesture. Annual Review of Anthropology 26: 109–128.View ArticleGoogle Scholar
  28. Kendon, A. 2004. Gesture: Visible action as utterance. Cambridge University Press, Cambridge.Google Scholar
  29. Kok, K. 2016. The grammatical potential of co-speech gesture: A functional discourse grammar perspective. Functions of Language 23 (3): 149–178.View ArticleGoogle Scholar
  30. Kok, K., K. Bergmann, A. Cienki, and S. Kopp. 2016. Mapping out the multifunctionality of speakers’ gestures. Gesture 15 (1): 37-59.Google Scholar
  31. Kress, G., and T. van Leeuwen. 1996. Reading images: The grammar of visual design. 1st ed. London: Routledge.Google Scholar
  32. Kress, G., and T. van Leeuwen. 2006. Reading images: The grammar of visual design. 2nd ed. London: Routledge.View ArticleGoogle Scholar
  33. Martin, J. R. 1992. English text: System and structure. Amsterdam: Benjamins.Google Scholar
  34. Martin, J. R. 2008. Intermodal reconciliation: Mates in arms. In New literacies and the English curriculum: Multimodal perspectives, ed. L. Unsworth, 112–148. London: Continuum.Google Scholar
  35. Martin, J. R. 2010. Semantic variation: Modelling system, text and affiliation in social semiosis. In New discourse on language: Functional perspectives on multimodality, identity and affiliation, ed. M. Bednarek and J.R. Martin, 1–34. London: Continuum.Google Scholar
  36. Martin, J. R. 2011. Multimodal semiotics: Theoretical challenges. In Semiotic margins: Meaning in multimodalities, ed. S. Dreyfus, S. Hood, and M. Stenglin, 243–270. London: Continuum.Google Scholar
  37. Martin, J. 2014. Evolving systemic functional linguistics: Beyond the clause. Functional Linguistics 1(3).Google Scholar
  38. Martin, J.R. forthcoming. Once more with feeling: Negotiating evaluation. Language, Context and Text.Google Scholar
  39. Martin, J. R. & D. Rose. 2007. Working with discourse: Meaning beyond the clause (2nd revised edition). London: Continuum.Google Scholar
  40. Martin, J. R. & P.R.R. White. 2005. The language of evaluation: Appraisal in English. London: Palgrave.Google Scholar
  41. Martin, J. R. & Zappavigna, M. 2013. Youth Justice Conferencing: ceremonial redress. International Journal of Law, Language and Discourse 2013: 103–142.Google Scholar
  42. Martin, J. R. & Zappavigna, M. 2018. Embodied meaning: a systemic functional perspective on body language. Chinese edition Contemporary Rhetoric 1 (Special issue on multimodality edited by Wang Zhenhua). Vol. 1, 2–29 詹姆斯·马丁 米歇尔·扎帕维尼娅 2018.副语言意义研究——系统功能语言学视角。《当代修辞学》.Google Scholar
  43. Martin, J.R., M. Zappavigna, P. Dwyer & C. Cléirigh. 2013. Users in uses of language: Embodied identity in youth justice conferencing. Text & Talk 33 (4/5): 467–496.Google Scholar
  44. Martinec, R. 1998. Cohesion in action. Semiotica 120 (1/2): 161–180.Google Scholar
  45. Martinec, R. 2000a. Rhythm in multimodal texts. Leonardo 33 (4): 289–297.View ArticleGoogle Scholar
  46. Martinec, R. 2000b. Types of process in action. Semiotica 130 (3/4): 243–268.Google Scholar
  47. Martinec, R. 2001. Interpersonal resources in action. Semiotica 135 (1/4): 117–145.Google Scholar
  48. Martinec, R. 2002. Rhythmic hierarchy in monologue and dialogue. Functions of Language 9 (1): 39–59.View ArticleGoogle Scholar
  49. Martinec, R. 2004. Gestures which co-occur with speech as a systematic resource: The realisation of experiential meanings in indexes. Social Semiotics 14 (2): 193–213.View ArticleGoogle Scholar
  50. Martinec, R. 2005. Topics in multimodality. In Continuing discourse on language: A functional perspective, ed. R. Hasan, C.M.I.M. Matthiessen, and J.J. Webster, vol. 1, 157–181. London: Equinox.Google Scholar
  51. Martinec, R. 2008. Review of Cienki & Muller. Linguistics and the Human Sciences 4 (3): 1–19.Google Scholar
  52. Matthiessen, C.M.I.M. 2004. The evolution of language: A systemic functional exploration of phylogenetic phases. In The development of language: Functional perspectives on species and individuals, ed. A. Lukin and G. Williams, 45–90. London: Continuum.Google Scholar
  53. Matthiessen, C.M.I.M. 2007. The multimodal page. In New directions in the analysis of multimodal discourse, ed. T. Royce and W. Bowcher, 1–62. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  54. Matthiessen, C.M.I.M. 2009. Multisemiosis and context-based register typology. In The world told and the world shown: Multisemiotic issues, ed. E. Ventola and M. Guijarro, 11–38. London: Palgrave Macmillan.Google Scholar
  55. McNeill, D. 1992. Hand and mind. Chicago: Chicago University Press.Google Scholar
  56. McNeill, D. 2000a. Introduction. In Language and gesture, ed. D. McNeill, 1–10. Cambridge: Cambridge University Press.View ArticleGoogle Scholar
  57. McNeill, D., ed. 2000b. Language and gesture. Cambridge: Cambridge University Press.Google Scholar
  58. McNeill, D. 2006. Gesture: A psycholinguistic approach. In The encyclopedia of language and linguistics, ed. E. Brown and A. Anderson, 58–66. Amsterdam: Elsevier.View ArticleGoogle Scholar
  59. McNeill, D. 2012. How language began: Gesture and speech in human evolution. Cambridge: Cambridge University Press.View ArticleGoogle Scholar
  60. Muntigl, P. 2004. Modelling multiple semiotic systems: The case of gesture and speech. In Perspectives on multimodality, ed. E. Ventola, C. Charles, and M. Kaltenbacher, 31–50. Amsterdam: Benjamins.View ArticleGoogle Scholar
  61. Ngo, T. n.d.(in press). Gesture as transduction of characterisation in children’s literature animation adaptation. Australian Journal of Language and Literacy.Google Scholar
  62. Ngo T, S Hood, JR Martin, C Painter, B Smith, M Martin & M. Zappavigna. forthcoming. Modelling paralanguage from the perspective of Systemic Functional Linguistics: theory and application. London: Routledge.Google Scholar
  63. O’Halloran, K.L., S. Tan, & P. Wignell n.d. (in press) SFL and Multimodal Discourse Analysis. In G. Thompson, W.L. Bowcher, L. Fontaine, & D. Schonthal (eds.), The Cambridge handbook of Systemic Functional Linguistics. Cambridge: Cambridge University Press.Google Scholar
  64. Ochs, E., E.A. Schegloff, and S.A. Thompson, eds. 1996. Interaction and grammar. Cambridge: Cambridge University press (studies in interactional Sociolinguistics 13).Google Scholar
  65. Painter, C. 1984. Into the mother tongue: A case study of early language development. London: Pinter.Google Scholar
  66. Painter, C. 2003. Developing attitude: An ontogenetic perspective on APPRAISAL. Text 23 (2): 183–209.View ArticleGoogle Scholar
  67. Painter, C., and J.R. Martin. 2012. Intermodal complementarity: Modelling affordances across verbiage and image in children’s picture books. Studies in Systemic Functional Linguistics and Discourse Analysis (III), 132–158. Beijing: Higher Education Press.Google Scholar
  68. Painter, C., J.R. Martin, and L. Unsworth. 2013. Reading visual narratives: Image analyses of children’s picture books. London: Equinox.Google Scholar
  69. Sekine, K., M.L. Rose, A.M. Foster, M.C. Attard, and L.E. Lanyon. 2013. Gesture production in aphasic discourse: In-depth description and preliminary predictions. Aphasiology 27 (9): 1031–1049.View ArticleGoogle Scholar
  70. Smith, B.A., and W.S. Greaves. 2015. Intonation. In The Bloomsbury companion to MAK Halliday, ed. J.J. Webster, 291–313. London: Bloomsbury.Google Scholar
  71. Taylor, C. 2017. Reading images: Including moving ones. In The Routledge handbook of Systemic Functional Linguistics, ed. T. Bartlett and G. O’Grady, 575–590. London: Routledge.Google Scholar
  72. Torr, J. 1991. From child tongue to mother tongue: A case study of language development in the first two and a half years (Monographs in Systemic Linguistics no.9). University of Nottingham: Dept. of English Studies.Google Scholar
  73. Trevarthen, C. 2005. First things first: Infants make good use of the sympathetic rhythm of imitation, without reason or language. Journal of Child Psychotherapy 31 (1): 91–113.View ArticleGoogle Scholar
  74. Tuite, K. 1993. The production of gesture. Semiotica 93 (1/2): 83–105.Google Scholar
  75. van Leeuwen, T. 1985. Rhythmic structure of the film text. In Discourse and communication: New approaches to the analysis of mass media discourse and communication, ed. T.A. van Dijk, 216–232. Berlin: De Gruyter.Google Scholar
  76. van Leeuwen, T. 1992. Rhythm and social context: Accent and juncture in the speech of professional radio announcers. In Systemic phonology, ed. P. Tench, 231–262. London: Pinter.Google Scholar
  77. van Leeuwen, T. 1999. Speech, music, sound. London: Macmillan.View ArticleGoogle Scholar
  78. Zappavigna, M. & Zappavigna, M. 2018. Discourse and diversionary justice: An analysis of youth justice conferencing. London: Palgrave Macmillan.Google Scholar
  79. Zappavigna, M., C. Cléirigh, P. Dwyer & J. R. Martin. 2010. The coupling of gesture and phonology. In M. Bednarek, & J.R. Martin (eds.), New discourse on language: Functional perspectives on multimodality, identity and affiliation. London: Continuum. 219–236.Google Scholar

Copyright

© The Author(s). 2019

Advertisement