research-article

Open access

The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users

Authors:

Asbjørn Følstad,

Petter Bae BrandtzaegAuthors Info & Claims

CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/3571884.3597144

Published: 19 July 2023 Publication History

All formats PDF

Abstract

The launch of ChatGPT has attracted significant attention and showcased the potentially game-changing capabilities of conversational AI. These capabilities, and lack of user research, highlight the need to investigate how users experience interactions with conversational AIs like ChatGPT. Therefore, we conducted a questionnaire study with ChatGPT users (N=194), inquiring about their good and poor experiences with ChatGPT. The user reports were analyzed by a thematic analysis and systematized through a pragmatic-hedonic framework. Our results demonstrate how user experience is influenced by pragmatic attributes such as ChatGPT providing useful and detailed information and easing work- or school-related tasks. Additionally, user experience is impacted by hedonic attributes, such as entertainment and creative interactions, and interactions leaving the user impressed or surprised. Our study underscores that user experience concerning conversational AI like ChatGPT is assessed by useful and productive interactions even in early phase of uptake, suggesting the importance of pragmatic attributes.

References

[1]

van Dis, E. A., Bollen, J., Zuidema, W., van Rooij, R. and Bockting, C. L. ChatGPT: five priorities for research. Nature, 614, 7947 (2023), 224-226.

[2]

Hu, K. ChatGPT sets record for fastest-growing user base - analyst note. 2023.

[3]

Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G. and Lu, Y. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 (2020).

[4]

Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Shuster, K. and Smith, E. M. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637 (2020).

[5]

Shrivastava, R. ChatGPT is coming to a customer service chatbot near you. (2023).

[6]

Mollick, E. ChatGPT is a tipping point for AI. Opinion piece, 2022.

[7]

Følstad, A. and Taylor, C. Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues. Quality and User Experience, 6, 1 (2021), 1-17.

[8]

ISO 9241-210:2019. Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems, 2019.

[9]

Law, E. L.-C. and Van Schaik, P. Modelling user experience–An agenda for research and practice. Interacting with computers, 22, 5 (2010), 313-322.

[10]

Bargas-Avila, J. A. and Hornbæk, K. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. CHI, 2011.

Digital Library

[11]

Kocaballi, A. B., Laranjo, L. and Coiera, E. Understanding and measuring user experience in conversational interfaces. Interacting with Computers, 31, 2 (2019), 192-207.

[12]

Haugeland, I. K. F., Følstad, A., Taylor, C. and Bjørkli, C. A. Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design. International Journal of Human-Computer Studies, 161 (2022), 102788.

Digital Library

[13]

Følstad, A. and Brandtzaeg, P. B. Users' experiences with chatbots: findings from a questionnaire study. Quality and User Experience, 5, 1 (2020), 3.

[14]

Hassenzahl, M. The thing and I: understanding the relationship between user and product. Funology 2: from usability to enjoyment (2018), 301-313.

[15]

Chen, H., Liu, X., Yin, D. and Tang, J. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19, 2 (2017), 25-35.

Digital Library

[16]

Følstad, A. and Skjuve, M. Chatbots for customer service: user experience and motivation. CUI, 2019.

Digital Library

[17]

Oesterreich, T. D., Anton, E., Schuir, J., Brehm, A. and Teuteberg, F. How can I help you? Design principles for task-oriented speech dialog systems in customer service. Information Systems and e-Business Management (2022), 1-43

[18]

Mogaji, E., Balakrishnan, J., Nwoba, A. C. and Nguyen, N. P. Emerging-market consumers’ interactions with banking chatbots. Telematics and Informatics, 65 (2021), 101711.

Digital Library

[19]

Melián-González, S., Gutiérrez-Taño, D. and Bulchand-Gidumal, J. Predicting the intentions to use chatbots for travel and tourism. Current Issues in Tourism, 24, 2 (2021), 192-210.

[20]

van der Goot, M. J., Hafkamp, L. and Dankfort, Z. Customer service chatbots: A qualitative interview study into the communication journey of customers. Springer, Conversations, 2020.

[21]

Luger, E. and Sellen, A. " Like Having a Really Bad PA" The Gulf between User Expectation and Experience of Conversational Agents. CHI, 2016.

[22]

Skjuve, M., Følstad, A., Fostervold, K. I. and Brandtzaeg, P. B. A longitudinal study of human–chatbot relationships. International Journal of Human-Computer Studies, 168 (2022), 102903.

[23]

Brandtzaeg, P. B., Skjuve, M., Dysthe, K. and Følstad, A. When the Social Becomes Non-Human: Young People's Perception of Social Support in Chatbots. ACM press, CHI, 2021

[24]

Ta, V., Griffith, C., Boatfield, C., Wang, X., Civitello, M., Bader, H., DeCero, E. and Loggarakis, A. User Experiences of Social Support From Companion Chatbots in Everyday Contexts: Thematic Analysis. Journal of medical Internet research, 22, 3 (2020), e16235.

[25]

Shum, H.-Y., He, X.-d. and Li, D. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19 (2018), 10-26.

[26]

Fitzpatrick, K. K., Darcy, A. and Vierhile, M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR mental health, 4, 2 (2017), e7785.

[27]

Chaves, A. P. and Gerosa, M. A. How should my chatbot interact? A survey on social characteristics in human–chatbot interaction design. International Journal of Human–Computer Interaction, 37, 8 (2021), 729-758.

[28]

Nordheim, C. B., Følstad, A. and Bjørkli, C. A. An initial model of trust in chatbots for customer service—findings from a questionnaire study. Interacting with Computers, 31, 3 (2019), 317-335.

[29]

Skjuve, M., Følstad, A., Fostervold, K. I. and Brandtzaeg, P. B. My Chatbot Companion-a Study of Human-Chatbot Relationships. International Journal of Human-Computer Studies, 149 (2021), 102601.

[30]

Go, E. and Sundar, S. S. Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior, 97 (2019), 304-316.

Digital Library

[31]

Palan, S. and Schitter, C. Prolific. ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17 (2018), 22-27.

[32]

Clarke, V., Braun, V. and Hayfield, N. Thematic analysis. Qualitative psychology: A practical guide to research methods, 3 (2015), 222-248.

[33]

Pinch, T. J. and Bijker, W. E. The social construction of facts and artefacts: Or how the sociology of science and the sociology of technology might benefit each other. Social studies of science, 14, 3 (1984), 399-441.

[34]

Lopatovska, I., Rink, K., Knight, I., Raines, K., Cosenza, K., Williams, H., Sorsche, P., Hirsch, D., Li, Q. and Martinez, A. Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science, 51, 4 (2019), 984-997.

[35]

Ta-Johnson, V. P., Boatfield, C., Wang, X., DeCero, E., Krupica, I. C., Rasof, S. D., Motzer, A. and Pedryc, W. M. Assessing the Topics and Motivating Factors Behind Human-Social Chatbot Interactions: Thematic Analysis of User Experiences. JMIR Human Factors, 9, 4 (2022), e38876.

[36]

AI, O. ChatGPT: Optimizing Language Models for Dialogue. (2023).

[37]

Mehdi, Y. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. (2023).

[38]

Dang, H., Mecke, L., Lehmann, F., Goller, S. and Buschek, D. How to Prompt? Opportunities and Challenges of Zero-and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models. arXiv preprint arXiv:2209.01390 (2022).

[39]

Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, March 2021.

Digital Library

[40]

Sedova, B. B. A. L. M. M. K. Truth, Lies, and Automation. Center for security and emerging technology, 2021.

[41]

Følstad, A., Skjuve, M., & Brandtzaeg, P. B. (2019). Different chatbots for different purposes: towards a typology of chatbots to understand interaction design. In Internet Science: INSCI 2018 International Workshops, St. Petersburg, Russia, October 24–26, 2018, Revised Selected Papers 5 (pp. 145-156). Springer International Publishing.

[42]

Haque, M. U., Dharmadasa, I., Sworna, Z. T., Rajapakse, R. N., & Ahmad, H. (2022). " I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data. arXiv preprint arXiv:2212.05856.

[43]

Hornbæk, K., & Hertzum, M. (2017). Technology acceptance and user experience: A review of the experiential component in HCI. ACM Transactions on Computer-Human Interaction (TOCHI), 24(5), 1-30.

Digital Library

Cited By

Kacperski CUlloa RBonnay DKulshrestha JSelb PSpitz A(2025)Characteristics of ChatGPT users from Germany: Implications for the digital divide from web tracking dataPLOS ONE10.1371/journal.pone.030904720:1(e0309047)Online publication date: 17-Jan-2025
https://doi.org/10.1371/journal.pone.0309047
Gessinger ISeaborn KSteeds MCowan B(2025)ChatGPT and meInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103400194:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103400
Aithal PAithal S(2024)User Invented Applications of AI-Based GPTs in Teaching-Learning and Research-PublicationsInternational Journal of Applied Engineering and Management Letters10.47992/IJAEML.2581.7000.0211(39-55)Online publication date: 25-Jan-2024
https://doi.org/10.47992/IJAEML.2581.7000.0211
Show More Cited By

Recommendations

Narrative agency and user experience in transmedia narratives: Brazilian telenovelas case
CLIHC '15: Proceedings of the Latin American Conference on Human Computer Interaction

This short paper proposes a reflection about the transmedia strategies in brazilian telenovelas, more specifically on how is the user experience in this kind of narratives. The user experience design in these narratives in multiple platforms is key to ...
How do users experience the interaction with an immersive screen?
Abstract
Ultra-high definition television (UHDTV) is designed using a variety of immersive technologies that are believed to promote user engagement. This study examines the immersion feature of UHDTV by focusing on user experience (UX) to ...
Highlights
- Examine the immersion feature of UHDTV.
- User experience and quality perception ...
Data Pattern for Allocating User Experience Meta-Data to User Experience Research Data
Proceedings of the Symposium on Human Interface 2009 on ConferenceUniversal Access in Human-Computer Interaction. Part I: Held as Part of HCI International 2009

The vision of user experience is making life of users of products as convenient as possible, especially during the interaction with a productor a service. An important aspect of perceived convenience is the user experience of a product. The visual ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces

July 2023

504 pages

ISBN:9798400700149

DOI:10.1145/3571884

Editors:
Minha Lee
Eindhoven University of Technology, Netherlands
,
Cosmin Munteanu
University of Waterloo, Canada
,
Martin Porcheron
Bold Insight, UK
,
Johanne Trippas
RMIT University, Australia
,
Sarah Theres Völkel
Google, Germany

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

The research council of Norway
The Norwegian Media Authority

Conference

CUI '23

Sponsor:

SIGCHI

CUI '23: ACM conference on Conversational User Interfaces

July 19 - 21, 2023

Eindhoven, Netherlands

Acceptance Rates

Overall Acceptance Rate 34 of 100 submissions, 34%

Upcoming Conference

CUI '25

Sponsor:
sigchi

ACM Conversational User Interfaces 2025

July 7 - 9, 2025

Waterloo , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
14,325
Total Downloads

Downloads (Last 12 months)10,510
Downloads (Last 6 weeks)631

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kacperski CUlloa RBonnay DKulshrestha JSelb PSpitz A(2025)Characteristics of ChatGPT users from Germany: Implications for the digital divide from web tracking dataPLOS ONE10.1371/journal.pone.030904720:1(e0309047)Online publication date: 17-Jan-2025
https://doi.org/10.1371/journal.pone.0309047
Gessinger ISeaborn KSteeds MCowan B(2025)ChatGPT and meInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103400194:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103400
Aithal PAithal S(2024)User Invented Applications of AI-Based GPTs in Teaching-Learning and Research-PublicationsInternational Journal of Applied Engineering and Management Letters10.47992/IJAEML.2581.7000.0211(39-55)Online publication date: 25-Jan-2024
https://doi.org/10.47992/IJAEML.2581.7000.0211
Petrič G(2024)Everyone Talks Everything With ChatGPTInternational Journal of Technology and Human Interaction10.4018/IJTHI.34922520:1(1-22)Online publication date: 17-Jul-2024
https://doi.org/10.4018/IJTHI.349225
Karthikeyan VKirubakaran GVarun Prakash RPalin Visu Y(2024)Trends and Research Potential With the Use of ChatGPT and Other AI ToolsApplications, Challenges, and the Future of ChatGPT10.4018/979-8-3693-6824-4.ch002(30-55)Online publication date: 28-May-2024
https://doi.org/10.4018/979-8-3693-6824-4.ch002
Talukder MKumar S(2024)The Development of ChatGPT and Its Implications for the Future of Customer Service in the Hospitality IndustryBlockchain Applications for Smart Contract Technologies10.4018/979-8-3693-1511-8.ch005(100-126)Online publication date: 14-Jun-2024
https://doi.org/10.4018/979-8-3693-1511-8.ch005
Casteleiro-Pitrez J(2024)Generative Artificial Intelligence Image Tools among Future Designers: A Usability, User Experience, and Emotional AnalysisDigital10.3390/digital40200164:2(316-332)Online publication date: 17-Apr-2024
https://doi.org/10.3390/digital4020016
Luther TKimmerle JCress U(2024)Teaming Up with an AI: Exploring Human–AI Collaboration in a Writing Scenario with ChatGPTAI10.3390/ai50300655:3(1357-1376)Online publication date: 5-Aug-2024
https://doi.org/10.3390/ai5030065
Whitman LMann KKanekar ABaker APidugu S(2024)A Pilot Study of the Use and Attitudes Toward Large Language Models Across Academic Disciplines2024 ASEE Annual Conference & Exposition Proceedings10.18260/1-2--46474Online publication date: Jul-2024
https://doi.org/10.18260/1-2--46474
Zhou TLi S(2024)Understanding user switch of information seeking: From search engines to generative AIJournal of Librarianship and Information Science10.1177/09610006241244800Online publication date: 10-Apr-2024
https://doi.org/10.1177/09610006241244800
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten