short-paper

Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities

Authors:

Danula Hettiachchi,

Flora Dilys Salim,

Falk ScholerAuthors Info & Claims

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1971 - 1975

https://doi.org/10.1145/3539618.3591981

Published: 18 July 2023 Publication History

Abstract

Physiological signals can potentially be applied as objective measures to understand the behavior and engagement of users interacting with information access systems. However, the signals are highly sensitive, and many controls are required in laboratory user studies. To investigate the extent to which controlled or uncontrolled (i.e., confounding) variables such as task sequence or duration influence the observed signals, we conducted a pilot study where each participant completed four types of information-processing activities (READ, LISTEN, SPEAK, and WRITE). Meanwhile, we collected data on blood volume pulse, electrodermal activity, and pupil responses. We then used machine learning approaches as a mechanism to examine the influence of controlled and uncontrolled variables that commonly arise in user studies. Task duration was found to have a substantial effect on the model performance, suggesting it represents individual differences rather than giving insight into the target variables. This work contributes to our understanding of such variables in using physiological signals in information retrieval user studies.

Supplemental Material

MP4 File

Presentation Video

Download
658.47 MB

References

[1]

Ioannis Arapakis, Ioannis Konstas, and Joemon M. Jose. 2009. Using Facial Expressions and Peripheral Physiological Signals as Implicit Indicators of Topical Relevance. In Proceedings of the 17th ACM International Conference on Multimedia (Beijing, China) (MM '09). Association for Computing Machinery, New York, NY, USA, 461--470. https://doi.org/10.1145/1631272.1631336

Digital Library

[2]

Ebrahim Babaei, Benjamin Tag, Tilman Dingler, and Eduardo Velloso. 2021. A Critique of Electrodermal Activity Practices at CHI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 177, 14 pages. https://doi.org/10.1145/3411764.3445370

Digital Library

[3]

Ashwin Ramesh Babu, Akilesh Rajavenkatanarayanan, James Robert Brady, and Fillia Makedon. 2018. Multimodal Approach for Cognitive Task Performance Prediction from Body Postures, Facial Expressions and EEG Signal. In Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data (Boulder, Colorado) (MCPMD '18). Association for Computing Machinery, New York, NY, USA, Article 14, 7 pages. https://doi.org/10.1145/3279810.3279849

Digital Library

[4]

VS Bakkialakshmi and T Sudalaimuthu. 2021. A Survey on Affective Computing for Psychological Emotion Recognition. In 2021 5th International Conference on Electrical, Electronics, Communication, Computer Technologies and Optimization Techniques (ICEECCOT) (Mysuru, India). IEEE, 480--486. https://doi.org/10.1109/ ICEECCOT52851.2021.9707947

[5]

Nicholas J Belkin. 1980. Anomalous States of Knowledge as a Basis for Information Retrieval. The Canadian Journal of Information Science 5, 1 (1980), 133--143.

[6]

Patricia J. Bota, Chen Wang, Ana L.N. Fred, and Hugo Placido Da Silva. 2019. A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals. IEEE Access 7 (2019), 140990-- 141020. https://doi.org/10.1109/ACCESS.2019.2944001

[7]

Jason J. Braithwaite, Derrick G. Watson, Robert Jones, and Mickey Rowe. 2013. A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs) for Psychological Experiments. Psychophysiology 49, 1 (2013), 1017--1034.

[8]

Georg Buscher, Andreas Dengel, Ralf Biedert, and Ludger V. Elst. 2012. Attentive Documents: Eye Tracking as Implicit Feedback for Information Retrieval and Beyond. ACM Trans. Interact. Intell. Syst. 1, 2, Article 9 (jan 2012), 30 pages. https://doi.org/10.1145/2070719.2070722

Digital Library

[9]

Michael J. Cole, Chathra Hendahewa, Nicholas J. Belkin, and Chirag Shah. 2014. Discrimination between Tasks with User Activity Patterns during Information Search. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval (Gold Coast, Queensland, Australia) (SIGIR '14). Association for Computing Machinery, New York, NY, USA, 567--576. https://doi.org/10.1145/2600428.2609591

Digital Library

[10]

Elena Di Lascio, Shkurta Gashi, and Silvia Santini. 2018. Unobtrusive Assessment of Students' Emotional Engagement during Lectures Using Electrodermal Activity Sensors. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3, Article 103 (sep 2018), 21 pages. https://doi.org/10.1145/3264913

Digital Library

[11]

Huanghao Feng, HoseinMGolshan, and Mohammad H Mahoor. 2018. AWavelet- Based Approach to Emotion Classification Using EDA Signals. Expert Systems with Applications 112 (2018), 77--86. https://doi.org/10.1016/j.eswa.2018.06.014

[12]

Nagarajan Ganapathy, Yedukondala Rao Veeranki, and Ramakrishnan Swaminathan. 2020. Convolutional Neural Network Based Emotion Classification Using Electrodermal Activity Signals and Time-Frequency Features. Expert Systems with Applications 159 (2020), 113571. https://doi.org/10.1016/j.eswa.2020.113571

[13]

Nan Gao, Wei Shao, Mohammad Saiedur Rahaman, and Flora D. Salim. 2020. N-Gage: Predicting in-Class Emotional, Behavioural and Cognitive Engagement in the Wild. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 79 (Sept. 2020), 26 pages. https://doi.org/10.1145/3411813

Digital Library

[14]

Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-Tracking Analysis of User Behavior inWWWSearch. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Sheffield, United Kingdom) (SIGIR '04). Association for Computing Machinery, New York, NY, USA, 478--479. https://doi.org/10.1145/1008992.1009079

Digital Library

[15]

Marti A. Hearst. 2009. Search User Interfaces. Cambridge University Press. https://doi.org/10.1017/CBO9781139644082

[16]

Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A Survey on Conversational Recommender Systems. ACM Comput. Surv. 54, 5, Article 105 (may 2021), 36 pages. https://doi.org/10.1145/3453154

Digital Library

[17]

Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends® in Information Retrieval 3, 1--2 (2009), 1--224. https://doi.org/10.1561/1500000012

Digital Library

[18]

Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Transactions on Affective Computing 3, 1 (2011), 18--31. https://doi.org/10.1109/TAFFC. 2011.15

Digital Library

[19]

Mariska E. Kret and Elio E. Sjak-Shie. 2019. Preprocessing Pupil Size Data: Guidelines and Code. Behavior Research Methods 51 (2019), 1336--1342. https: //doi.org/10.3758/s13428-018--1075-y

[20]

Yuan-Pin Lin, Chi-Hong Wang, Tzyy-Ping Jung, Tien-Lin Wu, Shyh-Kang Jeng, Jeng-Ren Duann, and Jyh-Horng Chen. 2010. EEG-Based Emotion Recognition in Music Listening. IEEE Transactions on Biomedical Engineering 57, 7 (2010), 1798--1806. https://doi.org/10.1109/TBME.2010.2048568

[21]

Yuan-Pin Lin, Chi-Hong Wang, Tien-Lin Wu, Shyh-Kang Jeng, and Jyh-Horng Chen. 2007. Multilayer Perceptron for EEG Signal Classification during Listening to Emotional Music. In TENCON 2007 - 2007 IEEE Region 10 Conference (Taipei, Taiwan). IEEE, 1--3. https://doi.org/10.1109/TENCON.2007.4428831

[22]

M. Maithri, U. Raghavendra, Anjan Gudigar, Jyothi Samanth, Prabal Datta Barua, Murugappan Murugappan, Yashas Chakole, and U. Rajendra Acharya. 2022. Automated Emotion Recognition: Current Trends and Future Perspectives. Computer Methods and Programs in Biomedicine 215 (2022), 106646. https://doi.org/10.1016/j.cmpb.2022.106646

Digital Library

[23]

Gary Marchionini. 1995. Information Seeking in Electronic Environments. Cambridge University Press. https://doi.org/10.1017/CBO9780511626388

[24]

Daniel McDuff, Paul Thomas, Nick Craswell, Kael Rowan, and Mary Czerwinski. 2021. Do Affective Cues Validate Behavioural Metrics for Search?. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 1544--1553. https://doi.org/10.1145/ 3404835.3462894

Digital Library

[25]

Yashar Moshfeghi and Joemon M. Jose. 2013. An Effective Implicit Relevance Feedback Technique Using Affective, Physiological and Behavioural Features. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (Dublin, Ireland) (SIGIR '13). Association for Computing Machinery, New York, NY, USA, 133--142. https://doi.org/10.1145/ 2484028.2484074

[26]

Yashar Moshfeghi, Peter Triantafillou, and Frank E. Pollick. 2016. Understanding Information Need: An FMRI Study. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR '16). Association for Computing Machinery, New York, NY, USA, 335--344. https://doi.org/10.1145/2911451.2911534

Digital Library

[27]

Javed Mostafa and Jacek Gwizdka. 2016. Deepening the Role of the User: Neuro- Physiological Evidence as a Basis for Studying and Improving Search. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (Carrboro, North Carolina, USA) (CHIIR '16). Association for Computing Machinery, New York, NY, USA, 63--70. https://doi.org/10.1145/2854946.2854979

Digital Library

[28]

Heather L. O'Brien, Paul Cairns, and Mark Hall. 2018. A Practical Approach to Measuring User Engagement with the Refined User Engagement Scale (UES) and New UES Short Form. International Journal of Human-Computer Studies 112 (2018), 28--39. https://doi.org/10.1016/j.ijhcs.2018.01.004

[29]

Bashar Rajoub. 2020. Chapter 3 -- Supervised and Unsupervised Learning. In Biomedical Signal Processing and Artificial Intelligence in Healthcare,Walid Zgallai (Ed.). Academic Press, 51--89. https://doi.org/10.1016/B978-0--12--818946--7.00003- 2

[30]

Johanne R. Trippas, Damiano Spina, Paul Thomas, Mark Sanderson, Hideo Joho, and Lawrence Cavedon. 2020. Towards a Model for Spoken Conversational Search. Information Processing & Management 57, 2 (2020), 102162. https: //doi.org/10.1016/j.ipm.2019.102162

Digital Library

[31]

Gyanendra K. Verma and Uma Shanker Tiwary. 2014. Multimodal Fusion Framework: A Multiresolution Approach for Emotion Classification and Recognition from Physiological Signals. NeuroImage 102 (2014), 162--172. https://doi.org/10.1016/j.neuroimage.2013.11.007

[32]

Ryen W. White. 2016. Interactions with Search Systems. Cambridge University Press. https://doi.org/10.1017/CBO9781139525305 [33] Ryen W. White and Ryan Ma. 2017. Improving Search Engines via Large-Scale Physiological Sensing. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR '17). Association for Computing Machinery, New York, NY, USA, 881--884. https://doi.org/10.1145/3077136.3080669

[33]

Robert T. Williams. 1972. A Table for Rapid Determination of Revised Dale-Chall Readability Scores. The Reading Teacher 26, 2 (1972), 158--165.

[34]

YingyingWu, Yiqun Liu, Ning Su, Shaoping Ma, andWenwu Ou. 2017. Predicting Online Shopping Search Satisfaction and User Behaviors with Electrodermal Activity. In Proceedings of the 26th International Conference on World Wide Web Companion (Perth, Australia) (WWW '17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 855--856. https://doi.org/10.1145/3041021.3054226

Digital Library

[35]

Tong Xue, Abdallah El Ali, Tianyi Zhang, Gangyi Ding, and Pablo Cesar. 2023. CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360? VR Videos. IEEE Transactions on Multimedia 25 (2023), 243--255. https://doi.org/10.1109/TMM.2021.3124080

[36]

Jing Zhai, A.B. Barreto, C. Chin, and Chao Li. 2005. Realization of Stress Detection using Psychophysiological Signals for Improvement of Human-Computer Interactions. In Proceedings. IEEE SoutheastCon, 2005. (Ft. Lauderdale, FL, USA). IEEE, 415--420. https://doi.org/10.1109/SECON.2005.1423280

Cited By

Frummet APapenmeier AFröbe MKiesel JAdlakha VBraunschweiler NDubiel MGhosh SGohsen MKreutz CMomeni MNilles MCherumanal SPirmoradi AThomas PTrippas JZelch IZendel O(2024)Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024ACM SIGIR Forum10.1145/3687273.368728258:1(1-12)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3687273.3687282
Ji KHettiachchi DSalim FScholer FSpina DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Characterizing Information Seeking Processes with Multiple Physiological SignalsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657793(1006-1017)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657793
Ji KSpina DHettiachchi DScholer FSalim F(2023)Towards Detecting Tonic Information Processing Activities with Physiological DataAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610679(1-5)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610679
Show More Cited By

Index Terms

Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval

Recommendations

Characterizing Information Seeking Processes with Multiple Physiological Signals
SIGIR '24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Information access systems are getting complex, and our understanding of user behavior during information seeking processes is mainly drawn from qualitative methods, such as observational studies or surveys. Leveraging the advances in sensing ...
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and Innnovation

In this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...
Manga content analysis using physiological signals
MANPU '16: Proceedings of the 1st International Workshop on coMics ANalysis, Processing and Understanding

Recently, the physiological signals have been analyzed more and more, especially in the context of everyday life activities such as watching video or looking at pictures. Tracking these signals gives access to the mental state of the user (interest, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2023

3567 pages

ISBN:9781450394086

DOI:10.1145/3539618

General Chairs:
Hsin-Hsi Chen
National Taiwan University
,
Wei-Jou (Edward) Duh
National Taiwan University
,
Hen-Hsen Huang
Academia Sinica
,
Program Chairs:
Makoto P. Kato
Spotify
,
Josiane Mothe
Universite de Toulouse
,
Barbara Poblete
University of Chile and Amazon Visiting Academic

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Australian Research Council
Australian Research Council Centre of Excellence for Automated Decision-Making and Society

Conference

SIGIR '23

Sponsor:

SIGIR

SIGIR '23: The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 23 - 27, 2023

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
132
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)7

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Frummet APapenmeier AFröbe MKiesel JAdlakha VBraunschweiler NDubiel MGhosh SGohsen MKreutz CMomeni MNilles MCherumanal SPirmoradi AThomas PTrippas JZelch IZendel O(2024)Report on the 8th Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI 2024) at CHIIR 2024ACM SIGIR Forum10.1145/3687273.368728258:1(1-12)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3687273.3687282
Ji KHettiachchi DSalim FScholer FSpina DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Characterizing Information Seeking Processes with Multiple Physiological SignalsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657793(1006-1017)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657793
Ji KSpina DHettiachchi DScholer FSalim F(2023)Towards Detecting Tonic Information Processing Activities with Physiological DataAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610679(1-5)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610679
Tosun Altınöz ŞDoğan UErcan Altınöz AÜzmez H(2023)Tolerance of Uncontrollability Questionnaire: Turkish Adaptation and Psychometric Evaluation in Clinical and Non-Clinical SamplesCognitive Therapy and Research10.1007/s10608-023-10450-048:3(526-536)Online publication date: 14-Nov-2023
https://doi.org/10.1007/s10608-023-10450-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten