A Generative Artificial Intelligence Empowered Chatbot: System Usability and Student Teachers’ Experience
Pages 330 - 340
Abstract
Generative Artificial Intelligence empowered conversational agents (chatbots) seem to be increasingly used in various settings including education. While student teachers are key stakeholders in supporting and improving education, not many studies exist in student teachers’ views on the educational use of chatbots. The current study performs a usability evaluation and explores student teachers’ views on the academic use of the VIP-Bot, an advanced academic Discord chatbot, which leverages the OpenAI’s gpt-3.5-turbo-instruct model. Student teachers, within the context of the formative task of writing a literature review, interacted with the chatbot and self-reported their experiences through an online survey. The usability evaluation returned a relatively high SUS score (76.36) for the chatbot. Moreover, student teachers’ view on the chatbot acceptance, effectiveness and motivation were positive. The chatbot can be helpful in developing ideas and initiating further engagement with the literature. Academic misconduct concerns have been expressed if the chatbot is not used properly. The study, as a usability evaluation, is an essential step in further chatbot development and, as an investigation of student teachers’ views, it is an essential step on the chatbot employment in teaching and learning.
References
[1]
Luo, X., Tong, S., Fang, Z., Qu, Z.: Frontiers: machines Vs. humans: the impact of artificial intelligence chatbot disclosure on customer purchases. Market. Sci. 38(6), 913–1084. (2019).
[2]
Chiu, T.K.F.: Future research recommendations for transforming higher education with generative AI. Comput. Educ. Artif. Intell. 6 (2024)
[3]
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. : Improving language understanding by generative pre-training (2018). https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
[4]
Skjuve, M., Følstad, A., Brandtzaeg, P.B.: The user experience of ChatGPT: findings from a questionnaire study of early users. In: Proceedings of the 5th International Conference on Conversational User Interfaces (CUI 2023), pp. 1–10. Association for Computing Machinery, New York, NY, USA, Article 2 (2023).
[5]
Adeshola I and Adepoju AP The opportunities and challenges of ChatGPT in education Interact. Learn. Environ. 2023
[6]
Mariani MM, Hashemi N, and Wirtz J Artificial intelligence empowered conversational agents: a systematic literature review and research agenda J. Bus. Res. 2023 161
[7]
Nikou SA and Chang M Frasson C, Mylonas P, and Troussas C Learning by building chatbot: a system usability study and teachers’ views about the educational uses of chatbots ITS 2023 2023 Cham Springer 342-351
[8]
Hwang GJ and Chang CY A review of opportunities and challenges of chatbots in education Interact. Learn. Environ. 2023 31 7 4099-4112
[9]
Hassenzahl, M.: User experience and experience design. The Encyclopedia of Human-Computer Interaction. Interaction Design Foundation (2013) Accessed. https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/user-experience-and-experience-design
[10]
ISO.2018. 9241-11:2018: Ergonomics of Human-System Interaction–Part11: Usability: Definitions and concepts. International Standardization Organization (ISO) (2018)
[11]
Brooke, J.: SUS: a ‘quick and dirty’ usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (eds.) Usability Evaluation in Industry, pp. 189–194. Taylor and Francis, London/UK (1996)
[12]
Bangor A, Kortum P, and Miller J An empirical evaluation of the system usability scale Int. J. Hum.-Comput. Interact. 2008 24 6 574-594
[13]
Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13(3), 319–340 (1989). Accessed. http://www.jstor.org/stable/249008
[14]
McAuley E, Duncan T, and Tammen VV Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: a confirmatory factor analysis Res. Q. Exerc. Sport 1987 60 48-58
[15]
Ren R, Zapata M, Castro JW, Dieste O, and Acuña ST Experimentation for chatbot usability evaluation: a secondary study IEEE Access 2022 10 12430-12464
[16]
Casas, J., Tricot, M.-O., Khaled, O.A., Mugellini, E., Cudré-Mauroux, P.: Trends & methods in chatbot evaluation. In: Companion Publication of the 2020 International Conference on Multimodal Interaction (ICMI 2020 Companion), pp. 280–286. Association for Computing Machinery, New York, NY, USA (2021).
[17]
Sauro, J.: 5 Ways to Interpret a SUS Score. Measuring U (2018). Accessed. https://measuringu.com/interpret-sus-score/
[18]
Sauro, J., Lewis, J.: Quantifying the User Experience: Practical Statistics for User Research. Elsevier/Morgan Kaufmann, Amsterdam, Waltham, MA (2016)
[19]
Braun V and Clarke V Using thematic analysis in psychology Qual. Res. Psychol. 2006 3 2 77-101
[20]
Saldana, J.: The Coding Manual for Qualitative Researchers, 3rd edn. Sage Publications Ltd. (2015)
[21]
Lai CY, Cheung KY, and Chan CS Exploring the role of intrinsic motivation in ChatGPT adoption to support active learning: an extension of the technology acceptance model Comput. Educ. Artif. Intell. 2023 5
[22]
Lewis JR and Sauro J Kurosu M The factor structure of the system usability scale Human Centered Design 2009 Heidelberg Springer 94-103
[23]
Chiu TK, Moorhouse BL, Chai CS, and Ismailov M Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot Interact. Learn. Environ. 2023
[24]
Wollny S, Schneider J, Di Mitri D, Weidlich J, Rittberger M, and Drachsler H Are we there yet? - A systematic literature review on chatbots in education Front. Artif. Intell. 2021 4
[25]
Hwang W and Salvendy G Number of people required for usability evaluation: the 10±2 rule Commun. ACM 2010 53 5 130-133
Recommendations
Learning by Building Chatbot: A System Usability Study and Teachers’ Views About the Educational Uses of Chatbots
Augmented Intelligence and Intelligent Tutoring SystemsAbstractThis article describes an experiment to investigate the usability of the “Learning by Building Chatbot” environment and the views of in-service teachers about the potential educational uses of chatbots. Chatbots are text or voice-based ...
The Development and Evaluation of English Dialogue Companion System
ICALT '08: Proceedings of the 2008 Eighth IEEE International Conference on Advanced Learning TechnologiesIn this paper, we propose the English Dialogue Companion (EDC) system to help elementary school students learn second language, as English. Learners can practice conversation with the EDC system in English. In order to motivate the learners, we design ...
Chatbot with Touch and Graphics: An Interaction of Users for Emotional Expression and Turn-taking
CUI '20: Proceedings of the 2nd Conference on Conversational User InterfacesUse of chatbots for emotional exchange is recently increasing in various domains. However, as existing chatbots have been considered in terms of natural language processing techniques for interaction with text-based chatting, chatbot interaction with ...
Comments
Information & Contributors
Information
Published In
Jun 2024
449 pages
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 10 June 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 28 Jan 2025