Towards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors
Authors: 
Qingxiao Zheng, 
Xi Lu, 
Qiao Jin, Jitesh Jain, Hedda Meadan-Kaplansky, Humphrey Shi, Jinjun Xiong, Yun HuangAuthors Info & Claims
Qingxiao Zheng, Xi Lu, Qiao Jin, Jitesh Jain, Hedda Meadan-Kaplansky, Humphrey Shi, Jinjun Xiong, Yun HuangAuthors Info & ClaimsPages 663 - 665
Abstract
This SIG is primarily designed for researchers and practitioners interested in exploring the utilization of multimodal artificial intelligence (AI) to understand and analyze human behaviors across various domains. It will particularly benefit professionals in UX design and research, cognitive science, education, psychology, and data science who are pursuing innovative ways to integrate multimodal data with AI to gain deeper behavioral insights. The session will focus on using AI to track, visualize, and generate insights around both verbal and non-verbal social interactions for various purposes, such as video question-answering. Discussions will cover practical applications in settings such as online learning environments, telecommunication, and social media platforms. This SIG will also provoke reflection on the ethical stewardship of AI, highlighting the importance of responsible usage to mitigate concerns such as algorithmic bias and data privacy.
References
[1]
Riku Arakawa, Kiyosu Maeda, and Hiromu Yakura. 2024. Supporting Experts with a Multimodal Machine-Learning-Based Tool for Human Behavior Analysis of Conversational Videos. arXiv preprint arXiv:2402.11145 (2024).
[2]
Chenhao Chen, Yutaka Arakawa, Ko Watanabe, and Shoya Ishimaru. 2022. Quan- titative Evaluation System for Online Meetings Based on Multimodal Microbe- havior Analysis. Sensors & Materials 34 (2022).
[3]
Xinyue Chen, Shuo Li, Shipeng Liu, Robin Fowler, and Xu Wang. 2023. MeetScript: Designing Transcript-based Interactions to Support Active Participation in Group Video Meetings. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1--32.
[4]
Bruno Carlos Dos Santos Melicio, Linyun Xiang, Emily Dillon, Latha Soorya, Mohamed Chetouani, Andras Sarkany, Peter Kun, Kristian Fenech, and Andras Lorincz. 2023. Composite AI for Behavior Analysis in Social Interactions. In Com- panion Publication of the 25th International Conference on Multimodal Interaction. 389--397.
[5]
Ankita Gandhi, Kinjal Adhvaryu, Soujanya Poria, Erik Cambria, and Amir Hus- sain. 2023. Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future di- rections. Information Fusion 91 (2023), 424--444.
[6]
Lin Guan, Yifan Zhou, Denis Liu, Yantian Zha, Heni Ben Amor, and Subbarao Kambhampati. 2024. " Task Success" is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors. arXiv preprint arXiv:2402.04210 (2024).
[7]
Matthew Jackson, Anne H Anderson, Rachel McEwan, and Jim Mullin. 2000. Impact of video frame rate on communicative behaviour in two and four party groups. In Proceedings of the 2000 ACM conference on Computer supported cooper- ative work. 11--20.
[8]
Jitesh Jain, Jianwei Yang, and Humphrey Shi. 2024. VCoder: Versatile Vision Encoders for Multimodal Large Language Models. In IEEE Conference on Computer Vision and Pattern Recognition.
[9]
Qiao Jin, Ishan Sodhi, Angie Chen, and Svetlana Yarosh. 2024. Interaction Forms of Collaborative VR Video Learning: An Exploratory Study. (2024).
[10]
ChunyuanLi,ZheGan,ZhengyuanYang,JianweiYang,LinjieLi,LijuanWang, Jianfeng Gao, et al. 2024. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision 16, 1--2 (2024), 1--214.
[11]
Tianyi Li and Francisco Iacobelli. 2023. Purposeful AI. In Companion Publica- tion of the 2023 Conference on Computer Supported Cooperative Work and Social Computing (Minneapolis, MN, USA) (CSCW '23 Companion). Association for Computing Machinery, New York, NY, USA, 563--565. https://doi.org/10.1145/ 3584931.3606954
[12]
HaotianLiu,ChunyuanLi,YuhengLi,andYongJaeLee.2024.ImprovedBaselines with Visual Instruction Tuning. In IEEE Conference on Computer Vision and Pattern Recognition.
[13]
Xi Lu, Yunan Chen, and Daniel A. Epstein. 2021. A Model of Socially Sustained Self-Tracking for Food and Diet. Proceedings of the ACM on Human-Computer Interaction 5 (10 2021), 451. Issue CSCW2. https://doi.org/10.1145/3479595
[14]
HeddaMeadan,JamesDLee,MichelleMSands,MoonYChung,andPauGarcía- Grau. 2023. The Coaching Fidelity Scale (CFS): Development and Evaluation of an Observational Measure of Coaching Fidelity. Infants & Young Children 36, 1 (2023), 37--52.
[15]
Arshi Naim. 2023. Consumer behavior in marketing patterns, types, segmentation. European Journal of Economics, Finance and Business Development 1, 1 (2023), 1--18.
[16]
Hong Shen, Tianshi Li, Toby Jia-Jun Li, Joon Sung Park, and Diyi Yang. 2023. Shap- ing the Emerging Norms of Using Large Language Models in Social Computing Research. In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing (Minneapolis, MN, USA) (CSCW '23 Com- panion). Association for Computing Machinery, New York, NY, USA, 569--571. https://doi.org/10.1145/3584931.3606955
[17]
Ignacio Villagrán. 2021. Exploring the Effects of Applying Learning Analytics for Teaching Procedural Skills in Health Sciences Education. In Companion Publi- cation of the 2021 Conference on Computer Supported Cooperative Work and Social Computing. 299--302.
[18]
Ridwan Whitehead, Andy Nguyen, and Sanna Järvelä. 2024. The Generative Mul- timodal Analysis (GMA) Methodology for Studying Socially Shared Regulation in Collaborative Learning. In The International Conference on Learning Analytics & Knowledge (LAK24).
[19]
Christy D Yoon, Adriana Kaori Terol, Hedda Meadan, and James D Lee. 2024. Gaze Behaviors and Social Communication Skills of Young Autistic Children: A Scoping Review. Review Journal of Autism and Developmental Disorders (2024), 1--15.
[20]
QingxiaoZheng,ParisaRabbani,Yu-RouLin,DaanMansour,andYunHuang. 2024. SOAP.AI: A Collaborative Tool for Documenting Human Behavior in Videos through Multimodal Generative AI. In Companion of the 2024 Computer- Supported Cooperative Work and Social Computing (CSCW Companion '24) (San Jose,CostaRica,2024--11-09/2024--11--13).ACM,NewYork,NY,USA,4. https: //doi.org/10.1145/3678884.3681819
[21]
Qingxiao Zheng, Yiliu Tang, Yiren Liu, Weizi Liu, and Yun Huang. 2022. UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Asso- ciation for Computing Machinery, New York, NY, USA, Article 570, 24 pages. https://doi.org/10.1145/3491102.3501855
[22]
QingxiaoZheng,ShengyangXu,LingqingWang,YiliuTang,RohanCSalvi,Guo Freeman, and Yun Huang. 2023. Understanding safety risks and safety design in social VR environments. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1--37.
Index Terms
- Towards Responsible Use of Large Multi-modal AI to Analyze Human Social Behaviors
Recommendations
Beyond Words: Infusing Conversational Agents with Human-like Typing Behaviors
CUI '24: Proceedings of the 6th ACM Conference on Conversational User InterfacesRecently, large language models have facilitated the emergence of highly intelligent conversational AI capable of engaging in human-like dialogues. However, a notable distinction lies in the fact that these AI models predominantly generate responses ...
Levels of Multimodal Interaction
ICMI Companion '24: Companion Proceedings of the 26th International Conference on Multimodal InteractionLarge Multimodal Models (LMMs) like OpenAI's GPT4o and Google's Gemini, introduced in 2024, process multiple modalities, enabling significant advances in multimodal interaction. Inspired by frameworks for self-driving cars and AGI, this paper proposes "...
Comments
Information & Contributors
Information
Published In
November 2024
755 pages
ISBN:9798400711145
DOI:10.1145/3678884
- General Chairs:
- Rosta Farzan,
- Claudia López,
- Program Chairs:
- Daniel Cardoso Llach,
- Daniele Quercia,
- Maryam Mustafa,
- Shuo Niu,
- Marisol Wong-Villacrés
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 November 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- National Science Foundation and the Institute of Education Sciences
Conference
CSCW '24
Sponsor:
CSCW '24: The 27th ACM Conference on Computer-Supported Cooperative Work and Social Computing
November 9 - 13, 2024
San Jose, Costa Rica
Acceptance Rates
Overall Acceptance Rate 2,235 of 8,521 submissions, 26%
Upcoming Conference
CSCW '25
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 178Total Downloads
- Downloads (Last 12 months)178
- Downloads (Last 6 weeks)71
Reflects downloads up to 25 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in