research-article

Analyzing and Recognizing Interlocutors' Gaze Functions from Multimodal Nonverbal Cues

Authors:

Kazuhiro OtsukaAuthors Info & Claims

ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction

Pages 33 - 41

https://doi.org/10.1145/3577190.3614152

Published: 09 October 2023 Publication History

Abstract

A novel framework is presented for analyzing and recognizing the functions of gaze in group conversations. Considering the multiplicity and ambiguity of the gaze functions, we first define 43 nonexclusive gaze functions that play essential roles in conversations, such as monitoring, regulation, and expressiveness. Based on the defined functions, in this study, a functional gaze corpus is created, and a corpus analysis reveals several frequent functions, such as addressing and thinking while speaking and attending by listeners. Next, targeting the ten most frequent functions, we build convolutional neural networks (CNNs) to recognize the frame-based presence/absence of each gaze function from multimodal inputs, including head pose, utterance status, gaze/avert status, eyeball direction, and facial expression. Comparing different input sets, our experiments confirm that the proposed CNN using all modality inputs achieves the best performance and an F value of 0.839 for listening while looking.

References

[1]

Reginald Adams and Robert Kleck. 2005. Effects of Direct and Averted Gaze on the Perception of Facially Communicated Emotion. Emotion 5 (2005), 3–11.

[2]

Michael Argyle. 1988. Bodily communication – 2nd ed.Routledge, London and New York.

[3]

Michael Argyle and Mark Cook. 1976. Gaze and Mutual Gaze. Cambridge University Press, London and New York and Melbourne.

[4]

Michael Argyle and Janet Dean. 1965. Eye-Contact, Distance and Affiliation. Sociometry 28, 3 (1965), 289–304.

[5]

Michael Argyle, Luc Lefebvre, and Mark Cook. 1974. The meaning of five patterns of gaze. European Journal of Social Psychology 4, 2 (1974), 125–136.

[6]

Tadas Baltrušaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Facial Behavior Analysis Toolkit. In Proc. 2018 13th IEEE Int. Conf. Automatic Face and Gesture Recognition (FG 2018). 59–66. https://doi.org/10.1109/FG.2018.00019

Digital Library

[7]

Cigdem Beyan, Francesca Capozzi, Cristina Becchio, and Vittorio Murino. 2018. Prediction of the Leadership Style of an Emergent Leader Using Audio and Visual Nonverbal Features. IEEE Transactions on Multimedia 20, 2 (2018), 441–456. https://doi.org/10.1109/TMM.2017.2740062

Digital Library

[8]

Max Boholm and Jens Allwood. 2010. Repeated head movements, their function and relation to speech. In Proc. Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality. LREC 2010. 1–5.

[9]

Dan Bohus and Eric Horvitz. 2010. Facilitating Multiparty Dialog with Gaze, Gesture, and Speech. In Proc. Int. Conf. Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction(ICMI-MLMI ’10). Article 5, 8 pages.

Digital Library

[10]

Starkey Duncan. 1972. Some Signals and Rules for Taking Speaking Turns in Conversations. Journal of Personality and Social Psychology 23 (1972), 283–292.

[11]

Starkey Duncan and George Niederehe. 1974. On signalling that it’s your turn to speak. Journal of Experimental Social Psychology 10, 3 (1974), 234–247.

[12]

Daniel Gatica-Perez. 2009. Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 27, 12 (2009), 1775–1787.

Digital Library

[13]

Anjith George and Aurobinda Routray. 2016. Real-time eye gaze direction classification using convolutional neural network. In 2016 Int. Conf. Signal Processing and Communications (SPCOM). 1–5.

[14]

Charles Goodwin. 1981. Conversational organization: Interaction between speakers and hearers. New York : Academic Press.

[15]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arxiv:1512.03385 [cs.CV]

[16]

M. Imamura, K. Takeda, S. Kumano, and K. Otsuka. 2023. Recognition of Synergetic Functions between Head Movements and Facial Expressions in Conversations. IEICE Tran. Fundamentals (Japanese Edition) J106-A, 3 (2023), 1–18.

[17]

M. Imamura, A. Tashiro, S. Kumano, and K. Otsuka. 2023. Analyzing Synergetic Functional Spectrum from Head Movements and Facial Expressions in Conversations. In Proc. ACM ICMI’23.

[18]

Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26 (1967), 22–63.

[19]

Chris L. Kleinke. 1986. Gaze and Eye Contact: A Research Review. Psychological Bulletin 100, 1 (1986), 78–100. https://doi.org/10.1037/0033-2909.100.1.78

[20]

Terry K. Koo and Mae Y. Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine 15, 2 (2016), 155–163.

[21]

Takashi Mori and Kazuhiro Otsuka. 2021. Deep Transfer Learning for Recognizing Functional Interactions via Head Movements in Multiparty Conversations. In Proc. 23rd ACM Int. Conf. Multimodal Interaction. 370–378.

Digital Library

[22]

Shumpei Otsuchi, Yoko Ishii, Momoko Nakatani, and Kazuhiro Otsuka. 2021. Prediction of Interlocutors’ Subjective Impressions Based on Functional Head-Movement Features in Group Meetings. In Proc. 23rd ACM Int. Conf. Multimodal Interaction. 352–360.

Digital Library

[23]

Kazuhiro Otsuka. 2011. Conversation Scene Analysis [Social Sciences]. IEEE Signal Processing Magazine 28, 4 (2011), 127–131.

[24]

Kazuhiro Otsuka, Keisuke Kasuga, and Martina Köhler. 2018. Estimating Visual Focus of Attention in Multiparty Meetings Using Deep Convolutional Neural Networks. In Proc. 20th ACM Int. Conf. Multimodal Interaction. 191–199. https://doi.org/10.1145/3242969.3242973

Digital Library

[25]

Kazuhiro Otsuka, Yoshinao Takemae, Junji Yamato, and Hiroshi Murase. 2005. A Probabilistic Inference of Multiparty-Conversation Structure Based on Markov-Switching Models of Gaze Patterns, Head Directions, and Utterances. In Proc. 7th ACM Int. Conf. Multimodal Interfaces(ICMI ’05). 191–198.

Digital Library

[26]

Kazuhiro Otsuka and Masahiro Tsumori. 2020. Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural Networks. IEEE Access 8 (2020), 217169–217195.

[27]

Kazuhiro Otsuka, Junji Yamato, Yoshinao Takemae, and Hiroshi Murase. 2006. Conversation Scene Analysis with Dynamic Bayesian Network Basedon Visual Head Tracking. In Proc. 2006 IEEE Int. Conf. Multimedia and Expo. 949–952.

[28]

Kazuhiro Otsuka, Junji Yamato, Yoshinao Takemae, and Hiroshi Murase. 2006. Quantifying Interpersonal Influence in Face-to-Face Conversations Based on Visual Attention Patterns. In Proc. CHI ’06 Extended Abstracts on Human Factors in Computing Systems. 1175–1180.

Digital Library

[29]

Miles L. Patterson. 1982. A sequential functional model of nonverbal exchange. Psychological Review 89 (05 1982), 231–249.

[30]

Frano Petric, Damjan Miklić, and Zdenko Kovačić. 2016. Probabilistic Eye Contact Detection for the Robot-assisted ASD Diagnostic Protocol. In Proc. Croatian Computer Vision Workshop, Year 4. 3–8.

[31]

Salwa O. Slim, Ayman Atia, Marwa M.A. Elfattah, and Mostafa-Sami M.Mostafa. 2019. Survey on Human Activity Recognition based on Acceleration Data. International Journal of Advanced Computer Science and Applications 10, 3 (2019), 84–98.

[32]

Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard. 2009. Social signal processing: Survey of an emerging domain. Image and Vision Computing 27, 12 (2009).

[33]

Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. 2021. Time Series Data Augmentation for Deep Learning: A Survey. In Proc. 13th Int. J. Conf. Artificial Intelligence, IJCAI-21. 1–8.

Cited By

Tashiro AImamura MKumano SOtsuka K(2024)Exploring Interlocutor Gaze Interactions in Conversations based on Functional Spectrum AnalysisProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685708(86-94)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685708
Ito KIshii YIshii REitoku SOtsuka K(2024)Exploring Multimodal Nonverbal Functional Features for Predicting the Subjective Impressions of InterlocutorsIEEE Access10.1109/ACCESS.2024.342653712(96769-96782)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3426537

Index Terms

Analyzing and Recognizing Interlocutors' Gaze Functions from Multimodal Nonverbal Cues
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI theory, concepts and models

Recommendations

Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment
Speech and Computer
Abstract
The paper explores multimodal collaboration in expository discourse considering verbal and nonverbal moves used by the participants in turn-taking. It reports the results of an experiment which tested speech, gesture and gaze alignment as affected ...
Looking for Laughs: Gaze Interaction with Laughter Pragmatics and Coordination
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

Laughter and gaze have an important role in managing and coordi-nating social interactions. In the current work, using a multimodal corpus of dyadic taste-testing interactions, we explore whether laughs performing different pragmatic functions are ...
Predicting evaluations of entrepreneurial pitches based on multimodal nonverbal behavioral cues and self-reported characteristics
ICMI '22 Companion: Companion Publication of the 2022 International Conference on Multimodal Interaction

Acquiring funding for a startup venture often involves pitching a business idea to potential investors. Insight into the nonverbal behavioral cues that impact the investment decision making process can help entrepreneurs to improve their persuasion ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '23: Proceedings of the 25th International Conference on Multimodal Interaction

October 2023

858 pages

ISBN:9798400700552

DOI:10.1145/3577190

Editors:
Elisabeth André
University of Augsburg
,
Mohamed Chetouani
Sorbonne University
,
Dominique Vaufreydaz
Univ. Grenoble Alpes
,
Gale Lucas
USC Institute for Creative Technologies
,
Tanja Schultz
University of Bremen
,
Louis-Philippe Morency
Carnegie Mellon University
,
Alessandro Vinciarelli
University of Glasgow

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Public Foundation of Chubu Science and Technology Center
JSPS KAKENHI
Telecom Advanced Technology Research Support Center,

Conference

ICMI '23

Sponsor:

SIGCHI

ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 9 - 13, 2023

Paris, France

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
145
Total Downloads

Downloads (Last 12 months)76
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tashiro AImamura MKumano SOtsuka K(2024)Exploring Interlocutor Gaze Interactions in Conversations based on Functional Spectrum AnalysisProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685708(86-94)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685708
Ito KIshii YIshii REitoku SOtsuka K(2024)Exploring Multimodal Nonverbal Functional Features for Predicting the Subjective Impressions of InterlocutorsIEEE Access10.1109/ACCESS.2024.342653712(96769-96782)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3426537

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten