research-article

The Experience Sampling Method and its Tools: A Review for Developers, Study Administrators, and Participants

Authors:

Tony MalzhackerAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 7, Issue EICS

Article No.: 182, Pages 1 - 29

https://doi.org/10.1145/3593234

Published: 19 June 2023 Publication History

Abstract

The Experience Sampling Method (ESM) is a popular research method found in many fields to gather rich insights into participants' thoughts, emotions, and daily routines near the moment they happen. During the last decade, many technologically advanced ESM tools emerged that combine manually entered self-reports with automatically collected data from device sensors. In fact, it became difficult to keep track of them. We compiled a survey of ESM and its tools addressing technological capabilities for developers, study design opportunities for study administrators, and answering usability for study participants. It comprises data from 30 systems, applications, and toolkits, which are used for ESM studies. We present our results on the current state of the art from these main user perspectives, list general shortcomings, and give recommendations for future ESM tools.

References

[1]

ACM. ACM Digital Library. ACM Inc., https://dl.acm.org/, 2022. (Accessed 2/1/2023).

[2]

Apache. Apache Cordova. https://cordova.apache.org/, 2022. (Accessed 20/04/2021).

[3]

Apple. ResearchKit -- Create Powerful Apps for Medical Research. http://researchkit.org/, 2022. (Accessed 2/1/2023).

[4]

Arslan, Ruben C, Walther, Matthias P and Tata, Cyril S. (2020). form: A Study Framework Allowing for Automated Feedback Generation and Complex Longitudinal Experience-Sampling Studies using R. Behavior Research Methods 52, 1. pp. 376--387.

[5]

Bachmann, Anja, Zetzsche, Robert, Schankin, Andrea, Riedel, Till, Beigl, Michael, Reichert, Markus, Santangelo, Philip and Ebner-Priemer, Ulrich. (2015). ESMAC: A Web-Based Configurator for Context-Aware Experience Sampling Apps in Ambulatory Assessment. In Proceedings of the 5th EAI International Conference on Wireless Mobile Communication and Healthcare (London, Great Britain). ICST. pp. 15--18.

Digital Library

[6]

Batalas, Nikolaos, aan het Rot, Marije, Khan, Vassilis Javed and Markopoulos, Panos. (2018). Using TEMPEST: End-User Programming of Web-Based Ecological Momentary Assessment Protocols. Proceedings of the ACM on Human-Computer Interaction 2, EICS. pp. 1--24.

[7]

Batalas, Nikoloas and Markopoulos, Panos. (2012). Considerations for Computerised In Situ Data Collection Platforms. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2012 (June 25--26, Copenhagen, Denmark). ACM, N.Y. pp. 231--236.

[8]

Behavio. Funf | Open Sensing Framework. https://www.funf.org/about.html, 2022. (Accessed 2/1/2023).

[9]

BlueJayVentures. PIEL Survey | Experience Sampling Method (ESM) | Ecological Momentary Assessment (EMA) | Survey Software. https://pielsurvey.org/, 2022. (Accessed 2/1/2023).

[10]

Broda, Michael. (2017). Using Multilevel Models to Explore Predictors of High School Students' Nonresponse in Experience Sampling Method (ESM) Studies. Social Science Computer Review 35, 6. pp. 733--750.

Digital Library

[11]

Carter, Scott, Mankoff, Jennifer and Heer, Jeffrey. (2007). Momento: Support for Situated Ubicomp Experimentation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA). Association for Computing Machinery. pp. 125--134.

Digital Library

[12]

Clarivate. Web of Science. https://webofknowledge.com, 2022. (Accessed 2/1/2023).

[13]

Django. The Web Framework for Perfectionists with Deadlines | Django. https://www.djangoproject.com/, 2022. (Accessed 2/1/2023).

[14]

Enketo. Web Forms Evolved. Deploy and Conduct Surveys that Work without a Connection, on Any Device. https://enketo.org/, 2022. (Accessed 2/1/2023).

[15]

ESMCapture. Experience Sampling Made Simple. http://esmcapture.com/, 2022. (Accessed 2/1/2023).

[16]

EsperTech. Complex Event Processing, Streaming Analytics, Streaming SQL - EsperTech. https://www.espertech.com/, 2022. (Accessed 2/1/2023).

[17]

EU. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation). https://eur-lex.europa.eu/eli/reg/2016/679/oj, 2022.

[18]

Evans, Bob. (2016). Paco-Applying Computational Methods to Scale Qualitative Methods. In Ethnographic Praxis in Industry Conference Proceedings (Minneapolis, Minnesota, USA). Wiley Online Library. pp. 348--368.

[19]

Feldman Barrett, Lisa and Barrett, Daniel. (Summer 2001). An Introduction to Computerised Experience Sampling in Psychology. Social Science Computer Review 19, 2. pp. 175--185.

[20]

Ferreira, Denzil, Kostakos, Vassilis and Dey, Anind K. (2015). AWARE: Mobile Context Instrumentation Framework. Frontiers in ICT 2. pp. 6.

[21]

Fischer, Joel E. (2009). Experience-sampling tools: a critical review. Mobile living labs 09: Methods and tools for evaluation in the wild. pp. 35.

[22]

Froehlich, Jon, Chen, Mike Y, Consolvo, Sunny, Harrison, Beverly and Landay, James A. (2007). MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones. In Proceedings of the 5th international conference on Mobile systems, applications and services (San Juan, Puerto Rico). Association for Computing Machinery. pp. 57--70.

Digital Library

[23]

Funk, Matthias, Liang, Rong-Hao, Palanque, Philippe, Hu, Jun and Markopoulos, Panos. (June 2021). Engineering Interactive Computing Systems 2022: Editorial Introduction. Proceedings of the ACM on Human-Computer Interaction 5, EICS, 193. pp. 193:1--193:4.

[24]

Gamma, Erich, Helm, Richard, Johnson, Ralph and Vlissides, John. (1995). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman Publishing Co., Inc.,

Digital Library

[25]

Google. Flutter - Beautiful Native Apps in Record Time. https://flutter.dev/, 2022. (Accessed 2/1/2023).

[26]

Google. Google Scholar. https://scholar.google.com/, 2022. (Accessed 2/1/2023).

[27]

Google. Protocol Buffers | Google Developers. https://developers.google.com/protocol-buffers, 2022. (Accessed 2/1/2023).

[28]

Hall, Malcolm, Bell, Marek, Morrison, Alistair, Reeves, Stuart, Sherwood, Scott and Chalmers, Matthew. (2009). Adapting Ubicomp Software and its Evaluation. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2009 (July 15--17, Pittsburgh, PA). ACM, N.Y. pp. 141--148.

Digital Library

[29]

Harari, Gabriella M, Lane, Nicholas D, Wang, Rui, Crosier, Benjamin S, Campbell, Andrew T and Gosling, Samuel D. (2016). Using Smartphones to Collect Behavioral Data in Psychological Science: Opportunities, Practical Considerations, and Challenges. Perspectives on Psychological Science 11, 6. pp. 838--854.

[30]

Hektner, Joel M., Csikszentmihalyi, Mihaly and Schmidt, Jennifer A. (2006). Experience Sampling Method: Measuring the Quality of Everyday Life. Sage Publications Inc., Thousand Oaks, CA.

[31]

Hofmann, Wilhelm and Patel, Paresh V. (2015). SurveySignal: A Convenient Solution for Experience Sampling Research Using Participants' Own Smartphones Social Science Computer Review 33, 2. pp. 235--253.

[32]

iDialogPad. This is an EMA (Ecological Momentary Assessment) and General Questionaire Universal App for iOS. https://github.com/gemu2015/iDialogPad, 2022. (Accessed 2/1/2023).

[33]

IEEE. IEEE Xplore. https://ieeexplore.ieee.org, 2022. (Accessed 03/04/2021).

[34]

ilumivu. mEMA App : ilumivu. https://ilumivu.com/solutions/ecological-momentary-assessment-app/, 2022. (Accessed 2/1/2023).

[35]

Intille, Stephen S, Rondoni, John, Kukla, Charles, Ancona, Isabel and Bao, Ling. (2003). A Context-Aware Experience Sampling Tool. In CHI '03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA). Association for Computing Machinery. pp. 972--973.

Digital Library

[36]

Intille, Stephen S, Stone, AA and Shiffman, S. (2007). Technological Innovations Enabling Automatic, Context-Sensitive Ecological Momentary Assessment. The Science of Real-Time Data Capture: Self-Reports in Health Research. pp. 308--337.

[37]

ISO. Ergonomics of Human-System Interaction - Part 11: Usability: Definitions and Concepts. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en, 2018. (Accessed 20/04/2021).

[38]

Karr, Chris. Purple Robot is an Android Application for Creating Automated Experiences on the Android Platform with Sensing and Inference Capabilities. https://github.com/cbitstech/Purple-Robot, 2022. (Accessed 19/07/2021).

[39]

Keusch, Florian and Conrad, Frederick G. (2021). Using Smartphones to Capture and Combine Self-Reports and Passively Measured Behavior in Social Research. Journal of Survey Statistics and Methodology.

[40]

Koval, Peter, Hinton, J, Dozo, N, Gleeson, John, Alvarez, Mario, Harrison, A, Vu, D, Susanto, R, Jayaputera, G and Sinnott, Richard. SEMA3: Smartphone Ecological Momentary Assessment, Version 3. https://sema3.com/, 2019. (Accessed 2/1/2023).

[41]

Larson, Reed and Csikszentmihalyi, Mihaly. (2014). The Experience Sampling Method. In Csikszentmihalyi, Mihaly, ed. Flow and the Foundations of Positive Psychology. Springer-Verlag, Heidelberg. pp. 21--34.

[42]

Lathia, Neal, Rachuri, Kiran, Mascolo, Cecilia and Roussos, George. (2013). Open Source Smartphone Libraries for Computational Social Science. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication (Zurich, Switzerland). Association for Computing Machinery. pp. 911--920.

Digital Library

[43]

Lawes, Mario, Hetschko, Clemens, Sakshaug, Joseph W and Grießemer, Stephan. (2021). Contact Modes and Participation in App-Based Smartphone Surveys: Evidence From a Large-Scale Experiment. Social Science Computer Review. pp. 1--17.

[44]

LifeData. LifeData Experience Sampling App | LifeData Experience Sampling App for Research. https://www.lifedatacorp.com/, 2022. (Accessed 2/1/2023).

[45]

Ludwig, Thomas, Dax, Julian, Pipek, Volkmar and Randall, Dave. (2016). Work or Leisure? Designing a User-Centered Approach for Researching Activity ?in the Wild". Personal and Ubiquitous Computing 20, 4. pp. 487--515.

Digital Library

[46]

Luyten, Kris, Palanque, Philippe, Quigley, Aaron and Winckler, Marco. (June 2022). Engineering Interactive Computing Systems 2022: Editorial Introduction. Proceedings of the ACM on Human-Computer Interaction 6, EICS, 149. pp. 149:1--149:3.

Digital Library

[47]

Meers, Kristof, Dejonckheere, Egon, Kalokerinos, Elise K, Rummens, Koen and Kuppens, Peter. (2020). mobileQ: A Free User-Friendly Application for Collecting Experience Sampling Data. Behaviour Research Methods 52, 4. pp. 1510--1515.

[48]

Metricwire. Metricwire Inc. | Real-World Intelligence | Value Based-Decisions. https://metricwire.com/, 2022. (Accessed 19/07/2021).

[49]

Microsoft. GitHub: Where the World Builds Software · GitHub. https://github.com/, 2022. (Accessed 2/1/2023).

[50]

Microsoft. Xamarin | Open-Source Mobile App Platform for .NET. https://dotnet.microsoft.com/apps/xamarin, 2022. (Accessed 2/1/2023).

[51]

movisensXS. movisensXS -- Experience Sampling Solution. movisens GmbH. https://www.movisens.com/en/products/movisensxs/, 2022. (Accessed 2/1/2023).

[52]

Mozilla. MDN Web Docs. https://developer.mozilla.org/en-US/, 2022. (Accessed 20/04/2021).

[53]

Nielson, Jakob. 10 Usability Heuristics for User Interface Design. https://www.nngroup.com/articles/ten-usability-heuristics/, 1994. (Accessed 20/04/2021).

[54]

O'Reilly-Shah, Vikas and Mackey, Sean. (Apr-Jun 2016). Survalytics: An Open-Source Cloud-Integrated Experience Sampling, Survey, and Analytics and Metadata Collection Module for Android Operating System Apps. JMIR mHealth and uHealth 4, 2. pp. 17--26.

[55]

ODK-X. Build your Custom Data Management App. https://odk-x.org/, 2022. (Accessed 2/1/2023).

[56]

ODK. Collect Data Anywhere. https://getodk.org/index.html, 2022. (Accessed 2/1/2023).

[57]

OpenCPU. OpenCPU - Producing and Reproducing Results. https://www.opencpu.org/, 2022. (Accessed 2/1/2023).

[58]

Page, Matthew J., McKenzie, Joanne E., Bossuyt, Patrick M., Boutron, Isabelle, Hoffmann, Tammy C., Mulrow, Cynthia D., Shamseer, Larissa, Tetzlaff, Jennifer M., Akl, Elie A., Brennan, Sue E., Chou, Roger, Glanville, Julie, Grimshaw, Jeremy M., Hróbjartsson, Asbjørn, Lalu, Manoj M., Li, Tianjing, Loder, Elizabeth W., Mayo-Wilson, Evan, McDonald, Steve, McGuinness, Luke A., Stewart, Lesley A., Thomas, James, Tricco, Andrea C., Welch, Vivian A., Whiting, Penny and Moher, David (2021). The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. Systematic Reviews 10, 89. pp. 2--11.

[59]

Pejovic, Veliko, Lathia, Neal, Mascolo, Cecilia and Musolesi, Mirco. (2016). Mobile-Based Experience Sampling for Behaviour Research. In Tkalcic, Marko, De Carolis, Berardina, de Gemmis, Marco, Odic, Ante and Kosir, Andrej, eds. Emotions and Personality in Personalised Services. Springer-Verlag, Heidelberg. pp. 141--161.

[60]

Pejovic, Veljko, Lathia, Neal, Mascolo, Cecilia and Musolesi, Mirco. (2016). Mobile-Based Experience Sampling for Behaviour Research. In Emotions and personality in personalised services. Springer. pp. 141--161.

[61]

Qualtrics. Online Survey Software - 1B+ Surveys Annually | Qualtrics. https://www.qualtrics.com/uk/core-xm/survey-software/, 2022. (Accessed 2/1/2023).

[62]

Rough, Daniel J and Quigley, Aaron. (2020). End-User Development of Experience Sampling Smartphone Apps-Recommendations and Requirements. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2. pp. 1--19.

Digital Library

[63]

Rough, Daniel John and Quigley, Aaron John. (2017). Jeeves - An Experience Sampling Study Creation Tool. In BCS Health Informatics Scotland (HIS) (Edinburgh, United Kingdom). BCS Electronic Workshops in Computing.

[64]

Salganik, Matthew J. (2019). Bit by Bit: Social Research in the Digital Age. Princeton University Press,

[65]

Silvia, Paul J, Kwapil, Thomas R, Eddington, Kari M and Brown, Leslie H. (2013). Missed Beeps and Missing Data: Dispositional and Situational Predictors of Nonresponse in Experience Sampling Research. Social Science Computer Review 31, 4. pp. 471--481.

[66]

Stier, Sebastian, Breuer, Johannes, Siegers, Pascal and Thorson, Kjerstin. (2020). Integrating Survey Data and Digital Trace Data: Key Issues in Developing an Emerging Field. Social Science Computer Review 38, 5. pp. 503--516.

Digital Library

[67]

Struminskaya, Bella, Lugtig, Peter, Keusch, Florian and Höhne, Jan Karem. (2020). Augmenting Surveys With Data From Sensors and Apps: Opportunities and Challenges. Social Science Computer Review. pp. 1--13.

[68]

Tangmunarunkit, Hongsuda, Hsieh, Cheng-Kang, Longstaff, Brent, Nolen, S, Jenkins, John, Ketcham, Cameron, Selsky, Joshua, Alquaddoomi, Faisal, George, Dony and Kang, Jinha. (2015). Ohmage: A General and Extensible End-to-End Participatory Sensing Platform. ACM Transactions on Intelligent Systems and Technology (TIST) 6, 3. pp. Article 38, 01--21.

[69]

Taylor, John T and Taylor, Wayne T. (2021). Software Architecture: Foundations, Theory, and Practice. Springer,

[70]

Teamscope. Teamscope: Data Collection App for Research | Secure, Offline. https://www.teamscopeapp.com/, 2022. (Accessed 2/1/2023).

[71]

Thai, Sabrina and Page-Gould, Elizabeth. (Dec 2018). ExperienceSampler: An Open-Source Scaffold for Building Smartphone Apps for Experience Sampling. Psychological Methods 23, 4. pp. 729--739.

[72]

Urbandroid Team. Don't Kill My App. https://dontkillmyapp.com/, 2022. (Accessed 3/1/2023).

[73]

Van Berkel, Niels, Ferreira, Denzil and Kostakos, Vassilis. (2017). The Experience Sampling Method on Mobile Devices. ACM Computing Surveys (CSUR) 50, 6. pp. Article 93, 01--40.

[74]

Van Berkel, Niels, Ferreira, Denzil and Kostakos, Vassilis. (Dec. 2017). The Experience Sampling Method on Mobile Devices. ACM Computing Surveys 50, 6. pp. 93:1--93:40.

[75]

W3C. Geolocation API. https://www.w3.org/TR/geolocation/, 2022. (Accessed 2/1/2023).

[76]

Wenz, Alexander, Jäckle, Annette, Burton, Jonathan and Couper, Mick P. (2022). The Effects of Personalized Feedback on Participation and Reporting in Mobile App Data Collection. Social Science Computer Review 40, 1. pp. 165--178.

Digital Library

[77]

Xiong, Haoyi, Huang, Yu, Barnes, Laura E and Gerber, Matthew S. (2016). Sensus: A Cross-Platform, General-Purpose System for Mobile Crowdsensing in Human-Subject Studies. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany). Association for Computing Machinery. pp. 415--426.

Digital Library

Cited By

Haucke MHeinzel SLiu S(2024)Social mobile sensing and problematic alcohol consumption: Insights from smartphone metadataInternational Journal of Medical Informatics10.1016/j.ijmedinf.2024.105486188(105486)Online publication date: Aug-2024
https://doi.org/10.1016/j.ijmedinf.2024.105486

Index Terms

The Experience Sampling Method and its Tools: A Review for Developers, Study Administrators, and Participants
1. General and reference
  1. Document types
    1. Surveys and overviews
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices

Recommendations

The Experience Sampling Method on Mobile Devices

The Experience Sampling Method (ESM) is used by scientists from various disciplines to gather insights into the intra-psychic elements of human life. Researchers have used the ESM in a wide variety of studies, with the method seeing increased ...
Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports
Research highlights
- We investigate the effect of random, interval, and smartphone-unlock based questionnaires
Abstract
The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of ...
End-User Development of Experience Sampling Smartphone Apps -Recommendations and Requirements

Professional programmers are significantly outnumbered by end-users of software, making it problematic to predict the diverse, dynamic needs of these users in advance. An end-user development (EUD) approach, supporting the creation and modification of ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 7, Issue EICS

EICS

June 2023

568 pages

EISSN:2573-0142

DOI:10.1145/3605541

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 June 2023

Published in PACMHCI Volume 7, Issue EICS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
225
Total Downloads

Downloads (Last 12 months)174
Downloads (Last 6 weeks)5

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Haucke MHeinzel SLiu S(2024)Social mobile sensing and problematic alcohol consumption: Insights from smartphone metadataInternational Journal of Medical Informatics10.1016/j.ijmedinf.2024.105486188(105486)Online publication date: Aug-2024
https://doi.org/10.1016/j.ijmedinf.2024.105486

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents