research-article

Using self-context for multimodal detection of head nods in face-to-face interactions

Authors:

Laurent Nguyen,

Jean-Marc Odobez,

Daniel Gatica-PerezAuthors Info & Claims

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Pages 289 - 292

https://doi.org/10.1145/2388676.2388734

Published: 22 October 2012 Publication History

Get Access

Abstract

Head nods occur in virtually every face-to-face discussion. As part of the backchannel domain, they are not only used to express a 'yes', but also to display interest or enhance communicative attention. Detecting head nods in natural interactions is a challenging task as head nods can be subtle, both in amplitude and duration. In this study, we make use of findings in psychology establishing that the dynamics of head gestures are conditioned on the person's speaking status. We develop a multimodal method using audio-based self-context to detect head nods in natural settings. We demonstrate that our multimodal approach using the speaking status of the person under analysis significantly improved the detection rate over a visual-only approach.

References

[1]

J. Allwood and L. Cerrato. A study of gestural feedback expressions. In Proc. First Nordic Symposium on Multimodal Communication, 2003.

Google Scholar

[2]

L. Dong, Y. Jin, L. Tao, and G. Xu. Recognition of multi-pose head gestures in human conversations. In Proc. Int. Conf. on Image and Graphics (ICIG), Aug. 2007.

Digital Library

Google Scholar

[3]

R. Gifford, C. F. Ng, and M. Wilkinson. Nonverbal cues in the employment interview: Links between applicant qualities and interviewer judgments. Applied Psychology, 70(4):729--736, 1985.

Crossref

Google Scholar

[4]

U. Hadar, T. Steiner, E. Grant, and F. Rose. Kinematics of head movements accompanying speech during conversation. Human Movement Science, 2(1-2):35--46, June 1983.

Crossref

Google Scholar

[5]

U. Hadar, T. J. Steiner, and F. Clifford Rose. Head movement during listening turns in conversation. Nonverbal Behavior, 9(4):214--228, 1985.

Crossref

Google Scholar

[6]

A. Kapoor and R. W. Picard. A real-time head nod and shake detector. In Proc. Workshop on Perceptive User Interfaces (ICMI-PUI), number 544, 2001.

Digital Library

Google Scholar

[7]

T. V. McGovern, B. W. Jones, and S. E. Morris. Comparison of professional versus student ratings of job interviewee behavior. Counseling Psychology, 26(2):176--179, 1979.

Crossref

Google Scholar

[8]

L. Morency, I. de Kok, and J. Gratch. Context-based recognition during human interactions: automatic feature selection and encoding dictionary. In Proceedings of the 10th international conference on Multimodal interfaces, pages 181--188. ACM, 2008.

Digital Library

Google Scholar

[9]

L. Morency and I. D. Kok. Predicting Listener Backchannels: A Probabilistic Multimodal Approach. Journal of Autonomous Agents and Multi-Agent Systems, pages 1--14, 2008.

Digital Library

Google Scholar

[10]

L.-P. Morency, C. Sidner, C. Lee, and T. Darrell. Contextual recognition of head gestures. In Proc. Int. Conf. on Multimodal Interfaces (ICMI), 2005.

Digital Library

Google Scholar

[11]

J. Odobez and P. Bouthemy. Robust multiresolution estimation of parametric motion models. Visual Communication and Image Representation, 6(4):348--365, 1995.

Crossref

Google Scholar

[12]

E. Ricci and J. Odobez. Learning large margin likelihoods for realtime head pose tracking. In Proc. Int. Conf. on Image Processing (ICIP), Nov. 2009.

Digital Library

Google Scholar

[13]

W. Tan and G. Rong. A real-time head nod and shake detector using HMMs. Expert Systems with Applications, 25(3):461--466, Oct. 2003.

Crossref

Google Scholar

Cited By

View all

Parra MFavela JCastro LGatica-Perez D(2024)Towards Wine Tasting Activity Recognition for a Digital SommelierCompanion Proceedings of the 26th International Conference on Multimodal Interaction10.1145/3686215.3686217(108-112)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3686215.3686217
Vuillecard PFarkhondeh AVillamizar MOdobez J(2024)CCDb-HG: Novel Annotations and Gaze-Aware Representations for Head Gesture Recognition2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581954(1-9)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581954
Kleinlogel ESchmid Mast MJayagopi DShubham KButera A(2023)“The interviewer is a machine!” Investigating the effects of conventional and technology‐mediated interview methods on interviewee reactions and behaviorInternational Journal of Selection and Assessment10.1111/ijsa.1243331:3(403-419)Online publication date: 8-May-2023
https://doi.org/10.1111/ijsa.12433
Show More Cited By

Index Terms

Using self-context for multimodal detection of head nods in face-to-face interactions
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

A real-time head nod and shake detector
PUI '01: Proceedings of the 2001 workshop on Perceptive user interfaces

Head nods and head shakes are non-verbal gestures used often to communicate intent, emotion and to perform conversational functions. We describe a vision-based system that detects head nods and head shakes in real time and can act as a useful and basic ...
A probabilistic multimodal approach for predicting listener backchannels

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important ...
Automatically Generating Head Nods with Linguistic Information
Social Computing and Social Media. Technologies and Analytics
Abstract
In addition to verbal behavior, nonverbal behavior is an important aspect for an embodied dialogue system to be able to conduct a smooth conversation with the user. Researchers have focused on automatically generating nonverbal behavior from ...

Comments

Information & Contributors

Information

Published In

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

October 2012

636 pages

ISBN:9781450314671

DOI:10.1145/2388676

General Chairs:
Louis-Philippe Morency
University of Southern California, USA
,
Dan Bohus
Microsoft Research, USA
,
Hamid Aghajan
Stanford University, USA
,
Program Chairs:
Justine Cassell
Carnegie Mellon University, USA
,
Anton Nijholt
University of Twente, Netherlands
,
Julien Epps
The University of New South Wales, Australia

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '12

Sponsor:

SIGCHI

ICMI '12: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 22 - 26, 2012

California, Santa Monica, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
207
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Parra MFavela JCastro LGatica-Perez D(2024)Towards Wine Tasting Activity Recognition for a Digital SommelierCompanion Proceedings of the 26th International Conference on Multimodal Interaction10.1145/3686215.3686217(108-112)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3686215.3686217
Vuillecard PFarkhondeh AVillamizar MOdobez J(2024)CCDb-HG: Novel Annotations and Gaze-Aware Representations for Head Gesture Recognition2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG59268.2024.10581954(1-9)Online publication date: 27-May-2024
https://doi.org/10.1109/FG59268.2024.10581954
Kleinlogel ESchmid Mast MJayagopi DShubham KButera A(2023)“The interviewer is a machine!” Investigating the effects of conventional and technology‐mediated interview methods on interviewee reactions and behaviorInternational Journal of Selection and Assessment10.1111/ijsa.1243331:3(403-419)Online publication date: 8-May-2023
https://doi.org/10.1111/ijsa.12433
Renier LSchmid Mast MDael NKleinlogel E(2021)Nonverbal Social Sensing: What Social Sensing Can and Cannot Do for the Study of Nonverbal Behavior From VideoFrontiers in Psychology10.3389/fpsyg.2021.60654812Online publication date: 27-Jul-2021
https://doi.org/10.3389/fpsyg.2021.606548
Vijay RShubham KRenier LKleinlogel EMast MJayagopi D(2021)An Opportunity to Investigate the Role of Specific Nonverbal Cues and First Impression in Interviews using Deepfake Based Controlled Video GenerationCompanion Publication of the 2021 International Conference on Multimodal Interaction10.1145/3461615.3485397(148-152)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3461615.3485397
Otsuka KTsumori M(2020)Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural NetworksIEEE Access10.1109/ACCESS.2020.30416728(217169-217195)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3041672
Rahman AAnam AYeasin M(2020)Robust modeling of epistemic mental statesMultimedia Tools and Applications10.1007/s11042-020-09145-5Online publication date: 16-Jun-2020
https://doi.org/10.1007/s11042-020-09145-5
Okada SNguyen LAran OGatica-Perez D(2019)Modeling Dyadic and Group Impressions with Intermodal and Interperson FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/326575415:1s(1-30)Online publication date: 24-Jan-2019
https://dl.acm.org/doi/10.1145/3265754
Murphy NHall JRuben MFrauendorfer DSchmid Mast MJohnson KNguyen L(2018)Predictive Validity of Thin-Slice Nonverbal Behavior from Social InteractionsPersonality and Social Psychology Bulletin10.1177/014616721880283445:7(983-993)Online publication date: 7-Nov-2018
https://doi.org/10.1177/0146167218802834
Diehl D(2018)Language and interaction: applying sociolinguistics to social network analysisQuality & Quantity10.1007/s11135-018-0787-553:2(757-774)Online publication date: 4-Jul-2018
https://doi.org/10.1007/s11135-018-0787-5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

A real-time head nod and shake detector

A probabilistic multimodal approach for predicting listener backchannels

Automatically Generating Head Nods with Linguistic Information

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations