research-article

Using Interaction Data to Predict Engagement with Interactive Media

Authors:

Jonathan Carlton,

John KeaneAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 1258 - 1266

https://doi.org/10.1145/3474085.3475631

Published: 17 October 2021 Publication History

Abstract

Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important in at least two ways: (1) a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experience and improve future productions; (2), this type of media has potential for real-time measures of engagement to be used to enhance the user experience by adapting content on-the-fly. Engagement is typically measured by asking samples of users to self-report, which is time consuming and expensive. In some domains, however, interaction data have been used to infer engagement. Fortuitously, the nature of interactive media facilitates a much richer set of interaction data than traditional media; our research aims to understand if these data can be used to infer audience engagement. In this paper, we report a study using data captured from audience interactions with an interactive TV show to model and predict engagement. We find that temporal metrics, including overall time spent on the experience and the interval between events, are predictive of engagement. The results demonstrate that interaction data can be used to infer users' engagement during and after an experience, and the proposed techniques are relevant to better understand audience preference and responses.

References

[1]

Ioannis Arapakis, Miguel Barreda-Angeles, and Alexandre Pereda-Banos. 2017. Interest as a proxy of engagement in news reading: Spectral and entropy analyses of EEG activity patterns. IEEE Transactions on Affective Computing, Vol. 10, 1 (2017), 100--114.

Digital Library

[2]

Ioannis Arapakis, Mounia Lalmas, and George Valkanas. 2014. Understanding within-content engagement through pattern analysis of mouse gestures. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. 1439--1448.

Digital Library

[3]

Ioannis Arapakis and Luis A Leiva. 2016. Predicting user engagement with direct displays using mouse cursor information. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 599--608.

Digital Library

[4]

Simon Attfield, Gabriella Kazai, Mounia Lalmas, and Benjamin Piwowarski. 2011. Towards a science of user engagement (position paper). In WSDM workshop on user modelling for Web applications. 9--12.

[5]

Nicola Barbieri, Fabrizio Silvestri, and Mounia Lalmas. 2016. Improving Post-click User Engagement on Native ads via Survival Analysis. In Proceedings of the 25th International Conference on World Wide Web. 761--770.

Digital Library

[6]

Ashwin Belle, Rosalyn Hobson, and Kayvan Najarian. 2011. A physiological signal processing system for optimal engagement and attention detection. In 2011 IEEE International Conference on Bioinformatics and Biomedicine Workshops (BIBMW). IEEE, 555--561.

Digital Library

[7]

Jonathan Carlton, Andy Brown, Caroline Jay, and John Keane. 2021. Using Interaction Data to Predict Engagement with Interactive Media (Multimedia'21 Paper Analysis Code). https://doi.org/10.5281/zenodo.5137806

Digital Library

[8]

Aleksandr Chuklin and Maarten de Rijke. 2016. Incorporating Clicks, Attention and Satisfaction into a Search Engine Result Page Evaluation Model. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. 175--184.

Digital Library

[9]

Marios Constantinides and John Dowell. 2018. A framework for Interaction-driven User Modeling of Mobile News Reading Behaviour. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization. 33--41.

Digital Library

[10]

Abdigani Diriye, Ryen White, Georg Buscher, and Susan Dumais. 2012. Leaving so soon? Understanding and predicting web search abandonment rationales. In Proceedings of the 21st ACM international conference on Information and knowledge management. 1025--1034.

Digital Library

[11]

Kevin Doherty and Gavin Doherty. 2018. Engagement in HCI: conception, theory and measurement. ACM Computing Surveys (CSUR), Vol. 51, 5 (2018), 1--39.

Digital Library

[12]

Nir Grinberg. 2018. Identifying modes of user engagement with online news and their relationship to information gain in text. In Proceedings of the 2018 World Wide Web Conference. 1745--1754.

Digital Library

[13]

Liangjie Hong and Mounia Lalmas. 2019. Tutorial on Online User Engagement: Metrics and Optimization. In Companion Proceedings of The 2019 World Wide Web Conference. 1303--1305.

Digital Library

[14]

Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni. 2014. Modeling Dwell Time to Predict Click-level Satisfaction. In Proceedings of the 7th ACM international conference on Web search and data mining. 193--202.

Digital Library

[15]

Dmitry Lagun and Mounia Lalmas. 2016. Understanding user attention and engagement in online news reading. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. 113--122.

Digital Library

[16]

Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of User Engagement. In International conference on user modeling, adaptation, and personalization. Springer, 164--175.

Digital Library

[17]

Jane Li, Scott Huffman, and Akihito Tokuda. 2009. Good abandonment in mobile and PC internet search. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 43--50.

Digital Library

[18]

Hongyu Lu, Min Zhang, and Shaoping Ma. 2018. Between clicks and satisfaction: Study on multi-phase user preferences and satisfaction for online news reading. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 435--444.

Digital Library

[19]

Hongyu Lu, Min Zhang, Weizhi Ma, Yunqiu Shao, Yiqun Liu, and Shaoping Ma. 2019. Quality effects on user preferences and behaviors in mobile news streaming. In The World Wide Web Conference. 1187--1197.

Digital Library

[20]

Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems. 4765--4774.

Digital Library

[21]

Wes McKinney et almbox. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference, Vol. 445. Austin, TX, 51--56.

[22]

Rishabh Mehrotra, Imed Zitouni, Ahmed Hassan Awadallah, Ahmed El Kholy, and Madian Khabsa. 2017. User Interaction Sequences for Search Satisfaction Prediction. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 165--174.

Digital Library

[23]

Ben Miroglio, David Zeber, Jofish Kaye, and Rebecca Weiss. 2018. The Effect of Ad Blocking on User Engagement with the Web. In Proceedings of the 2018 World Wide Web Conference. 813--821.

Digital Library

[24]

Yukiko I Nakano and Ryo Ishii. 2010. Estimating user's engagement from eye-gaze behaviors in human-agent conversations. In Proceedings of the 15th international conference on Intelligent user interfaces. 139--148.

Digital Library

[25]

Heather L O'Brien and Mahria Lebow. 2013. Mixed-methods approach to measuring user experience in online news interactions. Journal of the American Society for Information Science and Technology, Vol. 64, 8 (2013), 1543--1556.

[26]

Heather L O'Brien and Elaine G Toms. 2010. The development and evaluation of a survey to measure user engagement. Journal of the American Society for Information Science and Technology, Vol. 61, 1 (2010), 50--69.

Digital Library

[27]

Heather O'Brien. 2016. Theoretical perspectives on user engagement. In Why Engagement Matters. Springer, 1--26.

[28]

Heather L O'Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies, Vol. 112 (2018), 28--39.

[29]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, Vol. 12 (2011), 2825--2830.

Digital Library

[30]

Henrik Schoenau-Fog. 2011. Hooked!--evaluating engagement as continuation desire in interactive narratives. In International Conference on Interactive Digital Storytelling. Springer, 219--230.

Digital Library

[31]

Yang Song, Xiaolin Shi, Ryen White, and Ahmed Hassan Awadallah. 2014. Context-Aware Web Search Abandonment Prediction. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. 93--102.

Digital Library

[32]

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ?lhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, Vol. 17 (2020), 261--272. https://doi.org/10.1038/s41592-019-0686-2

[33]

Kyle Williams and Imed Zitouni. 2017. Does That Mean You're Happy? RNN-based Modeling of User Interaction Sequences to Detect Good Abandonment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 727--736.

Digital Library

[34]

Mengdie Zhuang, Gianluca Demartini, and Elaine G Toms. 2017. Understanding engagement through search behaviour. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1957--1966.

Digital Library

[35]

Mengdie Zhuang, Elaine G Toms, and Gianluca Demartini. 2018. Can User Behaviour Sequences Reflect Perceived Novelty?. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. 1507--1510.

Digital Library

Cited By

Katada SKomatani K(2024)Personalized Sentiment Estimation Based on Recall and Resting Ratio of Frontal EEGProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700248(1-7)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3696409.3700248
Gaspar-Figueiredo DFernández-Diego MNuredini RAbrahao SInsfran E(2024)Reinforcement Learning-Based Framework for the Intelligent Adaptation of User InterfacesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661329(40-48)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661329
He LBraggaar ABasar EKrahmer EAntheunis MWiers R(2024)Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot InteractionsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665536(1-14)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665536
Show More Cited By

Index Terms

Using Interaction Data to Predict Engagement with Interactive Media
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User models
      2. User studies
2. Information systems
  1. World Wide Web
    1. Web mining
      1. Web log analysis

Recommendations

Inferring User Engagement from Interaction Data
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

This paper presents preliminary results of a study designed to quantify users' engagement levels with interactive media content, through self-reported measures and interaction data. The broad hypothesis of the study is that interaction data can be used ...
Interactive documentaries: A Golden Age
SPECIAL ISSUE: TV and Video Entertainment Environments

This article is motivated by the opportunity presented by recent advances in information and communication technology—particularly by faster broadband connections and faster digital media processing capabilities—for interactive television to extend and ...
Using the Explicit User Profile to Predict User Engagement in Active Video Watching
UMAP '18: Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization

In this paper we leverage the explicit user profile (relating to experience, knowledge, and self-regulation) to predict user engagement in active video watching. Data from two user studies for informal learning of presentation skills in a Higher ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
279
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)13

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Katada SKomatani K(2024)Personalized Sentiment Estimation Based on Recall and Resting Ratio of Frontal EEGProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700248(1-7)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3696409.3700248
Gaspar-Figueiredo DFernández-Diego MNuredini RAbrahao SInsfran E(2024)Reinforcement Learning-Based Framework for the Intelligent Adaptation of User InterfacesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661329(40-48)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661329
He LBraggaar ABasar EKrahmer EAntheunis MWiers R(2024)Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot InteractionsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665536(1-14)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665536
Barker SGribble L(2024)Exploring student response systems for large group teaching: a tale of engagement at scaleJournal of Work-Applied Management10.1108/JWAM-10-2023-011516:2(316-328)Online publication date: 13-May-2024
https://doi.org/10.1108/JWAM-10-2023-0115
Abdullah MAlissa KAlzoubi O(2023)New User Engagement Prediction Using Machine Learning Algorithms2023 14th International Conference on Information and Communication Systems (ICICS)10.1109/ICICS60529.2023.10330488(01-06)Online publication date: 21-Nov-2023
https://doi.org/10.1109/ICICS60529.2023.10330488
Misiorek PCiesielczyk MRzycki B(2023)Digital Content Profiling Based on User Engagement FeaturesInformation Systems10.1007/978-3-031-30694-5_8(91-104)Online publication date: 20-Apr-2023
https://doi.org/10.1007/978-3-031-30694-5_8
Misiorek PWarmuz JKaczmarek DCiesielczyk M(2022)Modeling User Engagement Profiles for Detection of Digital Subscription PropensityInformation Systems10.1007/978-3-030-95947-0_5(55-68)Online publication date: 16-Feb-2022
https://doi.org/10.1007/978-3-030-95947-0_5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten