Supervised systems require human labels for training. But, are humans themselves always impartial during the annotation process? We examine this question in the context of automated assessment of human behavioral tasks. Specifically, we investigate whether human ratings themselves can be trusted at their face value when scoring video-based structured interviews, and whether such ratings can impact machine learning models that use them as training data. We present preliminary empirical evidence that indicates there are biases in such annotations, most of which are visual in nature.

References

[1]

Lei Chen, Gary Feng, Jilliam Joe, Chee Wee Leong, Christopher Kitchen, and Chong Min Lee. 2014. Towards automated assessment of public speaking skills using multimodal cues. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, 200–203.

Digital Library

Google Scholar

[2]

Lei Chen, Ru Zhao, Chee Wee Leong, Blair Lehman, Gary Feng, and Mohammed Ehsan Hoque. 2017. Automated video interview judgment on a large-sized corpus collected online. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 504–509.

Crossref

Google Scholar

[3]

Jose M Cortina, Nancy B Goldstein, Stephanie C Payne, H Kristl Davison, and Stephen W Gilliland. 2000. The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Personnel Psychology 53, 2 (2000), 325–351.

Crossref

Google Scholar

[4]

Allen I Huffcutt, James M Conway, Philip L Roth, and Nancy J Stone. 2001. Identification and meta-analytic assessment of psychological constructs measured in employment interviews.Journal of Applied Psychology 86, 5 (2001), 897.

Google Scholar

[5]

Julia Levashina, Christopher J Hartwell, Frederick P Morgeson, and Michael A Campion. 2014. The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology 67, 1 (2014), 241–293.

Crossref

Google Scholar

[6]

Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque. 2015. Automated prediction and analysis of job interview performance: The role of what you say and how you say it. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1. IEEE, 1–6.

Crossref

Google Scholar

[7]

Laurent Son Nguyen, Denise Frauendorfer, Marianne Schmid Mast, and Daniel Gatica-Perez. 2014. Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE transactions on multimedia 16, 4 (2014), 1018–1031.

Google Scholar

[8]

Laurent Son Nguyen and Daniel Gatica-Perez. 2015. I would hire you in a minute: Thin slices of nonverbal behavior in job interviews. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 51–58.

Digital Library

Google Scholar

[9]

Terence Parr. 2018. Feature importances for scikit random forests. https://github.com/parrt/random-forest-importances

Google Scholar

[10]

Vikram Ramanarayanan, Chee Wee Leong, Lei Chen, Gary Feng, and David Suendermann-Oeft. 2015. Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ACM, 23–30.

Digital Library

Google Scholar

[11]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135–1144.

Digital Library

Google Scholar

[12]

Lisa M Schreiber, Gregory D Paul, and Lisa R Shibley. 2012. The development and test of the public speaking competence rubric. Communication Education 61, 3 (2012), 205–233.

Crossref

Google Scholar

[13]

Rutuja Ubale, Yao Qian, and Keelan Evanini. 2018. Exploring End-To-End Attention-Based Neural Networks For Native Language Identification. In 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 84–91.

Google Scholar

[14]

David H Wolpert. 1992. Stacked generalization. Neural networks 5, 2 (1992), 241–259.

Google Scholar

Cited By

View all

Fabris ABaranowska NDennis MGraus DHacker PSaldivar JZuiderveen Borgesius FBiega A(2024)Fairness and Bias in Algorithmic Hiring: A Multidisciplinary SurveyACM Transactions on Intelligent Systems and Technology10.1145/369645716:1(1-54)Online publication date: 23-Sep-2024
https://dl.acm.org/doi/10.1145/3696457
Emerson AHoughton PChen KBasheerabad VUbale RLeong C(2022)Predicting User Confidence in Video Recordings with Spatio-Temporal Multimodal AnalyticsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3558007(98-104)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3558007
Lepp HLeong CRoohr KMartin-Raugh MRamanarayanan VTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)Effect of Modality on Human and Machine Scoring of Presentation VideosProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3418880(630-634)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3382507.3418880

Recommendations

ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild
Computer Vision – ECCV 2024
Abstract
While previous years have seen great progress in the 3D reconstruction of humans from monocular videos, few of the state-of-the-art methods are able to handle loose garments that exhibit large non-rigid surface deformations during articulation. ...
Dyadic Behavior Analysis in Depression Severity Assessment Interviews
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Previous literature suggests that depression impacts vocal timing of both participants and clinical interviewers but is mixed with respect to acoustic features. To investigate further, 57 middle-aged adults (men and women) with Major Depression Disorder ...
Modeling human annotation errors to design bias-aware systems for social stream processing
ASONAM '19: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

High-quality human annotations are necessary to create effective machine learning systems for social media. Low-quality human annotations indirectly contribute to the creation of inaccurate or biased learning systems. We show that human annotation ...

Comments

Information & Contributors

Information

Published In

ICMI '19: Adjunct of the 2019 International Conference on Multimodal Interaction

October 2019

86 pages

ISBN:9781450369374

DOI:10.1145/3351529

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Check for updates

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fabris ABaranowska NDennis MGraus DHacker PSaldivar JZuiderveen Borgesius FBiega A(2024)Fairness and Bias in Algorithmic Hiring: A Multidisciplinary SurveyACM Transactions on Intelligent Systems and Technology10.1145/369645716:1(1-54)Online publication date: 23-Sep-2024
https://dl.acm.org/doi/10.1145/3696457
Emerson AHoughton PChen KBasheerabad VUbale RLeong C(2022)Predicting User Confidence in Video Recordings with Spatio-Temporal Multimodal AnalyticsCompanion Publication of the 2022 International Conference on Multimodal Interaction10.1145/3536220.3558007(98-104)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536220.3558007
Lepp HLeong CRoohr KMartin-Raugh MRamanarayanan VTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)Effect of Modality on Human and Machine Scoring of Presentation VideosProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3418880(630-634)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3382507.3418880

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

ReLoo: Reconstructing Humans Dressed in Loose Garments from Monocular Video in the Wild

Dyadic Behavior Analysis in Depression Severity Assessment Interviews

Modeling human annotation errors to design bias-aware systems for social stream processing

Comments

Information

Published In

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations