research-article

Chatbots for active learning: : A case of phishing email identification

Authors:

Sebastian Hobert,

Asbjørn Følstad,

Effie Lai-Chong LawAuthors Info & Claims

Volume 179, Issue C

https://doi.org/10.1016/j.ijhcs.2023.103108

Published: 01 November 2023 Publication History

Highlights

•

Educational chatbots can be designed to support varying levels of cognitive engagement and active learning.

•

Chatbot interaction designs for enhanced cognitive engagement increase users' time spent within chatbot-based learning processes.

•

Users’ perceived subjective learning outcome can be increased by chatbot interaction designs for enhanced cognitive engagement.

•

Future research is required to determine the effects of chatbot interaction designs for cognitive engagement and active learning on user engagement.

Abstract

Chatbots represent a promising approach to provide instructional content and facilitate active learning processes. However, there is a lack of knowledge as how to design chatbot interactions for active learning. In response to this knowledge gap, we conducted an experimental study (n = 164) comparing four modes for providing instructional content in chatbots, with varying demands for cognitive engagement. The four modes – passive, active, constructive, and interactive – were based on the ICAP framework of active learning. The learning content concerned identification of phishing emails and the four modes were distinguished by how the participants were invited to engage with the content during their chatbot interaction. The ICAP modes of higher cognitive engagement required participants to spend more time on the interaction and led to perceptions of higher subjective learning outcome. However, the effects of the different ICAP modes were not found to be significantly different in terms of user engagement, social presence, intention to use, or objective learning outcomes. The study represents an important first step towards understanding the design of chatbots for active learning.

References

[1]

R.J. Amineh, H.D. Asl, Review of constructivism and social constructivism, J. So. Sci. Lit. Lang. 1 (1) (2015) 9–16.

[2]

T. Araujo, Living up to the chatbot hype: the influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions, Comput. Hum. Behav. 85 (2018) 183–189.

[3]

K. Bissell, R.M. LaSalle, P.D. Cin, Accenture’s ninth annual cost of cybercrime study: Unlocking the value of improved cybersecurity protection, Technical Report, Accenture (2019) https://www.accenture.com/us-en/insights/security/cost-cybercrime-study.

[4]

A. Brockbank, I. McGill, N. Beech, Reflective learning in practice, Reflective Learning in Practice, Routledge, 2017, pp. 18–28.

[5]

C.Y. Chang, G.J. Hwang, M.L. Gau, Promoting students’ learning achievement and self-efficacy: a mobile chatbot approach for nursing training, Br. J. Educ. Technol. 53 (1) (2021) 171–188.

[6]

M.T. Chi, R. Wylie, The ICAP framework: linking cognitive engagement to active learning outcomes, Educ. Psychol. 49 (4) (2014) 219–243.

[7]

L. Ciechanowski, A. Przegalinska, M. Magnuski, P. Gloor, In the shades of the uncanny valley: an experimental study of human–chatbot interaction, Future Gener. Comput. Syst. 92 (2019) 539–548.

[8]

R. Dale, The return of the chatbots, Nat. Lang. Eng. 22 (5) (2016) 811–817.

[9]

Feine, J., Morana, S., & Maedche, A. (2020). Designing interactive chatbot development systems. In Proceedings of ICIS 2020 (paper no. 1870).

[10]

S. Fergus, M. Botha, M. Ostovar, Evaluating academic answers generated using ChatGPT, J. Chem. Educ. 100 (4) (2023) 1672–1675.

[11]

M. Fidan, N. Gencel, Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: the effects on learning performance and intrinsic motivation, J. Educ. Comput. Res. (2022),.

[12]

K.K. Fitzpatrick, A. Darcy, M. Vierhile, Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial, JMIR Ment. Health 4 (2) (2017) e7785.

[13]

A. Følstad, P.B. Brandtzæg, Chatbots and the new world of HCI, ACM Interactions 24 (4) (2017) 38–42.

[14]

A. Følstad, P.B. Brandtzaeg, Users’ experiences with chatbots: findings from a questionnaire study, Qual. User Exp. 5 (1) (2020) 1–14.

[15]

Følstad, A., Skjuve, M., & Brandtzaeg, P.B. (2018). Different chatbots for different purposes: towards a typology of chatbots to understand interaction design. In Proceedings of the International Conference on Internet Science – INSCI 2018 (pp. 145–156). Cham, Switzerland: Springer.

[16]

A. Følstad, T. Araujo, E.L.C. Law, P.B. Brandtzaeg, S. Papadopoulos, L. Reis, E. Luger, Future directions for chatbot research: an interdisciplinary research agenda, Computing 103 (12) (2021) 2915–2942.

[17]

F.C. Fortenbaugh, J. DeGutis, L. Germine, J.B. Wilmer, M. Grosso, K. Russo, M. Esterman, Sustained attention across the life span in a sample of 10,000: dissociating ability and strategy, Psychol. Sci. 26 (9) (2015) 1497–1510.

[18]

L. Fryer, R. Carpenter, Bots as language learning tools, Lang. Learn. Technol. 10 (3) (2006) 8–14.

[19]

L.K. Fryer, M. Ainley, A. Thompson, A. Gibson, Z. Sherlock, Stimulating and sustaining interest in a language course: an experimental comparison of Chatbot and Human task partners, Comput. Hum. Behav. 75 (2017) 461–468.

[20]

L.K. Fryer, K. Nakao, A. Thompson, Chatbot learning partners: connecting learning experiences, interest and competence, Comput. Hum. Behav. 93 (2019) 279–289.

Digital Library

[21]

S. Gabrielli, S. Rizzi, G. Bassi, S. Carbone, R. Maimone, M. Marchesoni, S. Forti, Engagement and effectiveness of a healthy-coping intervention via chatbot for university students during the COVID-19 pandemic: mixed methods proof-of-concept study, JMIR Mhealth Uhealth 9 (5) (2021) e27965.

[22]

E. Go, S.S. Sundar, Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions, Comput. Hum. Behav. 97 (2019) 304–316.

Digital Library

[23]

A.K. Goel, L. Polepeddi, Jill Watson: a virtual teaching assistant for online education, In Learn. Eng. Online Educ. (2018) 120–143. Routledge.

[24]

Grand View Research, Education Technology Market size, Share & Trends Analysis Report, Grand View Research, 2022, Technical report https://www.grandviewresearch.com/industry-analysis/education-technology-market.

[25]

Hansch, A., Hillers, L., McConachie, K., Newman, C., Schildhauer, T. & Schmidt, J.P. (2015). Video and online learning: critical reflections and findings from the field. HIIG Discussion Paper Series No. 2015-02.

[26]

M. Hassenzahl, The interplay of beauty, goodness, and usability in interactive products, Hum.–Comput. Inter. 19 (4) (2004) 319–349.

[27]

I.K.F. Haugeland, A. Følstad, C. Taylor, C.A. Bjørkli, Understanding the user experience of customer service chatbots: an experimental study of chatbot interaction design, Int. J. Hum. Comput. Stud. 161 (2022).

[28]

L. He, E. Basar, R.W. Wiers, M.L. Antheunis, E. Krahmer, Can chatbots help to motivate smoking cessation? A study on the effectiveness of motivational interviewing on engagement and therapeutic alliance, BMC Public Health 22 (1) (2022) 1–14.

[29]

B. Heller, M. Proctor, D. Mah, L. Jewell, B. Cheung, Freudbot: an investigation of chatbot technology in distance education, In EdMedia+ Innovate Learning (2005) 3913–3918. Association for the Advancement of Computing in Education (AACE).

[30]

Hobert, S. (2019). Say hello to ‘Coding Tutor’! Design and evaluation of a chatbot-based learning system supporting students to learn to program. In Proceedings of the International Conference on Information Systems – ICIS 2019 (paper no. 2661).

[31]

Hobert, S. (2021). Individualized learning patterns require individualized conversations–Data-driven insights from the field on how chatbots instruct students in solving exercises. In Proceedings of CONVERSATIONS 2021 - International Workshop on Chatbot Research and Design (pp. 55–69). Cham, Switzerland: Springer.

[32]

Hobert, S. (2023). Fostering skills with chatbot-based digital tutors – training programming skills in a field study. i-com (Ahead-of-Print).

[33]

W. Huang, K.F. Hew, L.K. Fryer, Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning, J. Comput. Assisted Learn. 38 (1) (2022) 237–257.

[34]

Jain, M., Kumar, P., Kota, R., & Patel, S.N. (2018). Evaluating and informing the design of chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference – DIS 2018 (pp. 895–906). New York, NY: ACM.

[35]

D. Jampen, G. Gür, T. Sutter, B. Tellenbach, Don't click: towards an effective anti-phishing training. A comparative literature review, Human-centric Comput. Inf. Sci. 10 (1) (2020) 1–41.

[36]

E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, …., G. Kasneci, ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ. Differ 103 (2023).

[37]

M.A. Kuhail, N. Alturki, S. Alramlawi, K. Alhejori, Interacting with educational chatbots: a systematic review, Educ. Inf. Technol. 28 (1) (2023) 973–1018.

Digital Library

[38]

Laban, G., Araujo, T. (2020). Working together with conversational agents: the relationship of perceived cooperation with service performance evaluations. In Proceedings of Conversations 2019 -International Workshop on Chatbot Research and Design (pp. 215–228). Cham, Switzerland: Springer.

[39]

J. Lim, H. Ko, J.W. Yang, S. Kim, S. Lee, M.S. Chun, J. Park, Active learning through discussion: ICAP framework for education in health professions, BMC Med. Educ. 19 (1) (2019) 1–8.

[40]

G. Lindgaard, G. Fernandes, C. Dudek, J. Brown, Attention web designers: you have 50 milliseconds to make a good first impression!, Behav. Inf. Technol. 25 (2) (2006) 115–126.

[41]

LinkedIn Learning, 2020 Workplace learning report. Technical report, LinkedIn Learning, 2020, https://learning.linkedin.com/content/dam/me/learning/resources/pdfs/LinkedIn-Learning-2020-Workplace-Learning-Report.pdf.

[42]

LinkedIn Learning, 2022 workplace learning report, 2022, https://learning.linkedin.com/content/dam/me/learning/en-us/pdfs/workplace-learning-report/LinkedIn-Learning_Workplace-Learning-Report-2022-EN.pdf.

[43]

J. Litman, Curiosity and the pleasures of learning: wanting and liking new information, Cogn. Emot. 19 (6) (2005) 793–814.

[44]

M. McTear, Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots, Morgan & Claypool, 2021.

[45]

M. Mori, K.F. MacDorman, N. Kageki, The uncanny valley, [from the field] IEEE Robotics Autom. Magazine 19 (2) (2012) 98–100.

[46]

J. Morris, M.T. Chi, Improving teacher questioning in science using ICAP theory, J. Educ. Res. 113 (1) (2020) 1–12.

[47]

H.L. O'Brien, J. Arguello, R. Capra, An empirical study of interest, task complexity, and search behaviour on user engagement, Inf. Process Manag. 57 (3) (2020).

[48]

H.L. O'Brien, P. Cairns, M. Hall, A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form, Int. J. Hum. Comput. Stud. 112 (2018) 28–39.

[49]

E.G. Peterson, S. Hidi, Curiosity and interest: current perspectives, Educ. Psychol. Rev. 31 (4) (2019) 781–788.

[50]

J. Rudolph, S. Tan, S. Tan, ChatGPT: bullshit spewer or the end of traditional assessments in higher education?, J. Appl. Learn. Teach. 6 (1) (2023) 1–22. Shevat, A. (2017). Designing Bots: Creating Conversational Experiences. Boston, MA: O'Reilly Media.

[51]

M. Sarosa, M. Kusumawadhani, A. Suyono, Y.M. Azis, The effectiveness of chatbot as an online learning method on active and reflective learning styles, Multicultural Educ. 7 (9) (2021) 92–100.

[52]

Schöbel, S., Janson, A., & Mishra, A. (2019) A configurational view on avatar design–the role of emotional attachment, satisfaction, and cognitive load in digital learning. In Fortieth International Conference on Information Systems – ICIS 2019 (paper no. 1909).

[53]

T. Sitzmann, K. Ely, K.G. Brown, K.N. Bauer, Self-assessment of knowledge: a cognitive learning or affective measure?, Acad. Manag. Learn. Educ. 9 (2) (2010) 169–191.

[54]

A.D. Taylor, W. Hung, The effects of microlearning: a scoping review, Educ. Technol. Res. Dev. (2022) 1–33.

[55]

N. Tractinsky, A.S. Katz, D. Ikar, What is beautiful is usable, Interact. Comput. 13 (2) (2000) 127–145.

[56]

V. Venkatesh, F.D. Davis, A theoretical extension of the technology acceptance model: four longitudinal field studies, Manage. Sci. 46 (2) (2000) 186–204.

[57]

C. Wekerle, M. Daumiller, I. Kollar, Using digital technology to promote higher education learning: the importance of different learning activities and their relations to learning outcomes, J. Res. Technol. Educ. 54 (1) (2022) 1–17.

[58]

B.L. Wiggins, S.L. Eddy, D.Z. Grunspan, A.J. Crowe, The ICAP active learning framework predicts the learning gains observed in intensely active classroom experiences, AERA Open 3 (2) (2017),.

[59]

Winkler, R., Hobert, S., Salovaara, A., Söllner, M., & Leimeister, J.M. (2020). Sara, the lecturer: improving learning in online education with a scaffolding-based conversational agent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (paper no. 652). New York, NY, ACM.

[60]

Winkler, R., Weingart, P., & Söllner, M. (2020). Using smart personal assistants for online learning activities: what benefits can we expect? In Proceedings der 15. Internationalen Tagung Wirtschaftsinformatik 2020 (pp. 465–479). Berlin, Germany: GITO Verlag, 10.30844/wi_2020_d7-winkler.

[61]

Xiao, Z., Zhou, M.X., Chen, W., Yang, H., & Chi, C. (2020). If I hear you correctly: building and evaluating interview chatbots with active listening skills. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (paper no. 4). New York, NY: ACM.

[62]

J. Yin, T.T. Goh, B. Yang, Y. Xiaobin, Conversation technology with micro-learning: the impact of chatbot-based learning on students’ learning motivation and performance, J. Educ. Comput. Res. 59 (1) (2021) 154–177.

Cited By

Følstad ALaw Evan As N(2024)Conversational Breakdown in a Customer Service Chatbot: Impact of Task Order and Criticality on User Trust and EmotionACM Transactions on Computer-Human Interaction10.1145/369038331:5(1-52)Online publication date: 3-Sep-2024
https://dl.acm.org/doi/10.1145/3690383
Lei CDickson-Deane C(2024)Interface Design for Educational Chatbot to Increase Engagement for Online Learning: A Conceptual DesignLearning and Collaboration Technologies10.1007/978-3-031-61672-3_3(38-52)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61672-3_3

Index Terms

Chatbots for active learning: A case of phishing email identification
1. Applied computing
  1. Education
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Envisioned Pedagogical Uses of Chatbots in Higher Education and Perceived Benefits and Challenges
Learning and Collaboration Technologies: Games and Virtual Environments for Learning
Abstract
The widespread use of chatbots is a reality and their application in higher education is promising. Understanding higher education users’ expectations for the use of chatbots in education is important for the design and development of new ...
Individualized Learning Patterns Require Individualized Conversations – Data-Driven Insights from the Field on How Chatbots Instruct Students in Solving Exercises
Chatbot Research and Design
Abstract
Chatbots can foster the learning success of students in educational settings. This has been shown in prior research studies, e.g., using laboratory studies in online learning settings. Actual evaluations of using educational chatbots in the field ...
Prompting Large Language Models to Power Educational Chatbots
Advances in Web-Based Learning – ICWL 2023
Abstract
The recent rise in both popularity and performance of large language models has garnered considerable interest regarding their applicability to education. Technologies like ChatGPT, which can engage in human-like dialog, have already disrupted ...

Comments

Information & Contributors

Information

Published In

cover image International Journal of Human-Computer Studies

International Journal of Human-Computer Studies Volume 179, Issue C

Nov 2023

235 pages

ISSN:1071-5819

Issue’s Table of Contents

Copyright © 2023.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 November 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Følstad ALaw Evan As N(2024)Conversational Breakdown in a Customer Service Chatbot: Impact of Task Order and Criticality on User Trust and EmotionACM Transactions on Computer-Human Interaction10.1145/369038331:5(1-52)Online publication date: 3-Sep-2024
https://dl.acm.org/doi/10.1145/3690383
Lei CDickson-Deane C(2024)Interface Design for Educational Chatbot to Increase Engagement for Online Learning: A Conceptual DesignLearning and Collaboration Technologies10.1007/978-3-031-61672-3_3(38-52)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61672-3_3

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents