Abstract
Natural human interaction involves the fast-paced exchange of speaker turns. Crucially, if a next speaker waited with planning their turn until the current speaker was finished, language production models would predict much longer turn transition times than what we observe. Next speakers must therefore prepare their turn in parallel to listening. Visual signals likely play a role in this process, for example by helping the next speaker to process the ongoing utterance and thus prepare an appropriately-timed response.
To understand how visual signals contribute to the timing of turn-taking, and to move beyond the mostly qualitative studies of gesture in conversation, we examined unconstrained, computer-mediated conversations between 20 pairs of participants while systematically manipulating speaker visibility. Using motion tracking and manual gesture annotation, we assessed 1) how visibility affected the timing of turn transitions, and 2) whether use of co-speech gestures and 3) the communicative kinematic features of these gestures were associated with changes in turn transition timing.
We found that 1) decreased visibility was associated with less tightly timed turn transitions, and 2) the presence of gestures was associated with more tightly timed turn transitions across visibility conditions. Finally, 3) structural and salient kinematics contributed to gesture’s facilitatory effect on turn transition times.
Our findings suggest that speaker visibility--and especially the presence and kinematic form of gestures--during conversation contributes to the temporal coordination of conversational turns in computer-mediated settings. Furthermore, our study demonstrates that it is possible to use naturalistic conversation and still obtain controlled results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alibali, M.W., Heath, D.C., Myers, H.J.: Effects of visibility between speaker and listener on gesture production: some gestures are meant to be seen. J. Mem. Lang. 44, 169–188 (2001)
Bates, D., Maechler, M., Bolker, B., Walker, S.: Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67(1), 1–48 (2015). https://doi.org/10.18637/jss.v067.i01
Bavelas, J.B., Chovil, N., Coates, L., Roe, L.: Gestures specialized for dialogue: Pers. Soc. Psychol. Bull. (1995). https://doi.org/10.1177/0146167295214010
Bavelas, J.B., Chovil, N.: Visible acts of meaning: an integrated message model of language in face-to-face dialogue. J. Lang. Soc. Psychol. 19(2), 163–194 (2000). https://doi.org/10.1177/0261927X00019002001
ter Bekke, M., Drijvers, L., Holler, J.: The predictive potential of hand gestures during conversation: an investigation of the timing of gestures in relation to speech. (2020). https://doi.org/10.31234/osf.io/b5zq7
Boersma, P., Weenink, D.: Praat (Version 4.5. 25) (2007)
ten Bosch, L., Oostdijk, N., Boves, L.: On temporal aspects of turn taking in conversational dialogues. Speech Commun. 47(1), 80–86 (2005). https://doi.org/10.1016/j.specom.2005.05.009
Campisi, E., Özyürek, A.: Iconicity as a communicative strategy: recipient design in multimodal demonstrations for adults and children. J. Pragmat. 47(1), 14–27 (2013). https://doi.org/10.1016/j.pragma.2012.12.007
Drijvers, L., Özyürek, A.: Visual context enhanced: the joint contribution of iconic gestures and visible speech to degraded speech comprehension. J. Speech Lang. Hearing Res. 60(1), 212–222 (2017). https://doi.org/10.1044/2016_JSLHR-H-16-0101
Gerwing, J., Bavelas, J.: Linguistic influences on gesture’s form. Gesture 4, 157–195 (2004)
Heldner, M., Edlund, J.: Pauses, gaps and overlaps in conversations. J. Phon. 38(4), 555–568 (2010). https://doi.org/10.1016/j.wocn.2010.08.002
Holler, J., Kendrick, K.H., Levinson, S.C.: Processing language in face-to-face conversation: questions with gestures get faster responses. Psychon. Bull. Rev. 25(5), 1900–1908 (2018). https://doi.org/10.3758/s13423-017-1363-z
Holler, J., Levinson, S.C.: Multimodal language processing in human communication. Trends Cogn. Sci. 23(8), 639–652 (2019). https://doi.org/10.1016/j.tics.2019.05.006
Holler, J., Wilkin, K.: An experimental investigation of how addressee feedback affects co-speech gestures accompanying speakers’ responses. J. Pragmat. 43(14), 3522–3536 (2011). https://doi.org/10.1016/j.pragma.2011.08.002
Jadoul, Y., Thompson, B., de Boer, B.: Introducing parselmouth: a python interface to praat. J. Phon. 71, 1–15 (2018). https://doi.org/10.1016/j.wocn.2018.07.001
Kendon, A.: Some functions of gaze-direction in social interaction. Acta Psychologica 26, 22–63 (1967). https://doi.org/10.1016/0001-6918(67)90005-4
Kendon, A.: Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge (2004)
Kita, S., van Gijn, I., van der Hulst, H.: Movement phases in signs and co-speech gestures, and their transcription by human coders. In: Wachsmuth, I., Fröhlich, M. (eds) Gesture and Sign Language in Human-Computer Interaction. GW 1997. Lecture Notes in Computer Science, vol. 1371, pp. 23–35. Springer, Berlin, Heidelberg (1998). https://doi.org/10.1007/BFb0052986
Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)
Levinson, S.C.: Turn-taking in human communication—origins and implications for language processing. Trends Cogn. Sci. 20(1), 6–14 (2016). https://doi.org/10.1016/j.tics.2015.10.010
Levinson, S.C., Torreira, F.: Timing in turn-taking and its implications for processing models of language. Front. Psychol. 6, 731 (2015). https://doi.org/10.3389/fpsyg.2015.00731
McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
Pouw, W., Harrison, S.J., Dixon, J.A.: Gesture–speech physics: the biomechanical basis for the emergence of gesture–speech synchrony. J. Exp. Psychol. Gen. 149(2), 391–404 (2020). https://doi.org/10.1037/xge0000646
R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing (2019). https://www.R-project.org/
Ripperda, J., Drijvers, L. Holler, J.: Speeding up the detection of non-iconic and iconic gestures (SPUDNIG): A toolkit for the automatic detection of hand movements and gestures in video data. Behav. Res. 52, 1783–1794 (2020). https://doi.org/10.3758/s13428-020-01350-2
Roberts, S.G., Torreira, F., Levinson, S.C.: The effects of processing and sequence organization on the timing of turn taking: a corpus study, 509th edn, vol. 5. Frontiers in Psychology (2015). https://doi.org/10.3389/978-2-88919-825-2
Sacks, H., Schegloff, E. A., Jefferson, G.: A simplest systematics for the organization of turn-taking for conversation. 50(40) (1974)
Stivers, T. et al.: Universals and cultural variation in turn-taking in conversation. PNAS. 106, 10587–10592 (2009)
Trujillo, J.P., Simanova, I., Bekkering, H., Özyürek, A.: Communicative intent modulates production and comprehension of actions and gestures: a Kinect study. Cognition 180, 38–51 (2018). https://doi.org/10.1016/j.cognition.2018.04.003
Trujillo, J.P., Simanova, I., Bekkering, H., Özyürek, A.: The communicative advantage: how kinematic signaling supports semantic comprehension. Psychol. Res. 84(7), 1897–1911 (2019). https://doi.org/10.1007/s00426-019-01198-y
Trujillo, J.P., Simanova, I., Özyürek, A., Bekkering, H.: Seeing the unexpected: how brains read communicative intent through kinematics. Cereb. Cortex 30(3), 1056–1067 (2020). https://doi.org/10.1093/cercor/bhz148
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix 1
Appendix 1
Parameter overview of kinematic analyses
Model parameter | Parameter estimate* | Standard deviation | t-value |
|---|---|---|---|
Overlaps | |||
Formula: turn_transition_time ~ utterance_duration + maxdist + peakvel*grade + MN_mode + submovements*grade + holdtime + volume + (1 | participant + (1 | utterance) | |||
(Intercept) | −442.20 | 3318.00 | −0.13 |
utterance duration | 0.00 | 0.00 | −1.25 |
blur grade | 898.10 | 331.60 | 2.71 |
maximum distance | 184.80 | 126.70 | 1.46 |
peak velocity | −20.71 | 227.50 | −0.09 |
McNeillian mode | 49.83 | 40.27 | 1.24 |
submovements | −366.10 | 295.20 | −1.24 |
holdtime | 136.80 | 235.60 | 0.58 |
volume | −176.70 | 91.83 | −1.92 |
peak velocity * grade | −60.62 | 22.12 | −2.74 |
submovements * grade | 88.97 | 39.00 | 2.28 |
Gaps | |||
Formula: turn_transition_time ~ utterance_duration + maxdist + peakvel + MN_mode * grade + submovements + holdtime* grade + volume + (1 | dyad/participant) + (1 | utterance) | |||
(Intercept) | 1454.00 | 2044.00 | 0.71 |
utterance duration | 0.00 | 0.00 | −6.55 |
blur grade | −40.13 | 11.99 | −3.35 |
maximum distance | 79.07 | 81.79 | 0.97 |
peak velocity | −53.15 | 127.30 | −0.42 |
McNeillian mode | −123.30 | 50.80 | −2.43 |
Submovements | −58.75 | 120.50 | −0.49 |
holdtime | 224.10 | 220.20 | 1.02 |
volume | 106.60 | 56.58 | 1.88 |
MCcNeillian Mode * grade | 28.78 | 8.08 | 3.56 |
holdtime * grade | −34.44 | 31.19 | −1.10 |
*Note that because kinematic values were normalized, parameter estimates cannot be directly interpreted in terms of effect on turn transition time, but rather should be interpreted in terms of effects relative to other kinematic features | |||
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Trujillo, J.P., Levinson, S.C., Holler, J. (2021). Visual Information in Computer-Mediated Interaction Matters: Investigating the Association Between the Availability of Gesture and Turn Transition Timing in Conversation. In: Kurosu, M. (eds) Human-Computer Interaction. Design and User Experience Case Studies. HCII 2021. Lecture Notes in Computer Science(), vol 12764. Springer, Cham. https://doi.org/10.1007/978-3-030-78468-3_44
Download citation
DOI: https://doi.org/10.1007/978-3-030-78468-3_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78467-6
Online ISBN: 978-3-030-78468-3
eBook Packages: Computer ScienceComputer Science (R0)