AI-experiments in education: An AI-driven randomized controlled trial for higher education research
Pages 19649 - 19677
Abstract
This study presents a novel approach contributing to our understanding of the design, development, and implementation AI-based systems for conducting double-blind online randomized controlled trials (RCTs) for higher education research. The process of the entire interaction with the participants (n = 1193) and their allocation to test and control groups was executed seamlessly by our AI system, without human intervention. In this fully automated experiment, we systematically examined eight hypotheses. The AI-experiment strengthened five of these hypotheses, while not accepting three of the factors previously acknowledged in the literature as influential in students’ choices of universities. We showcased how AI can efficiently interview participants and collect their input, offering robust evidence through an RCT (Gold standard) to establish causal relationships between interventions and their outcomes. This approach may enable researchers and industry practitioners to collect data from large samples on which such experiments can be conducted with and by AI to produce statistically reproducible, reliable, and generalizable results in an efficient, rigorous and ethical way.
References
[1]
Androutsopoulou A, Karacapilidis N, Loukis E, and Charalabidis Y Transforming the communication between citizens and government through AI-guided chatbots Government Information Quarterly 2019 36 2 358-367
[2]
Arsenijevic, U., & Jovic, M. (2019). Artificial intelligence marketing: Chatbots. In International Conference on Artificial Intelligence - Applications and Innovations (IC-AIAI) (pp. 19–22). IEEE.
[3]
Boyd D and Crawford K Critical questions for big data information Communication & Society 2012 15 5 662-679
[4]
Chaves AP, Egbert J, Hocking T, Doerry E, and Gerosa MA Chatbots language design: The influence of language variation on user experience with tourist assistant chatbots ACM Transactions on Computer-Human Interaction (TOCHI) 2022 29 2 1-38
[5]
Chocarro R, Cortiñas M, and Marcos-Matás G Teachers’ attitudes towards chatbots in education: A technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics Educational Studies 2023 49 2 295-313
[6]
Christian LM, Parsons NL, and Dillman DA Designing Scalar questions for web surveys Sociological Methods & Research 2009 37 3 393-425
[7]
Cingillioglu, I., Gal, U., & Prokhorov, A. (2023). Social media marketing for student recruitment: An algorithmically sequenced literature review. Journal of Marketing for Higher Education, 1–23.
[8]
Cingillioglu I, Gal U, and Prokhorov A Running a double-blind true social experiment with a goal oriented adaptive AI-based conversational agent in educational research International Journal of Educational Research 2024 124 102323
[9]
Couper MP New Developments in Survey Data Collection Annual Review of Sociology 2017 43 121-145
[10]
CX Central (2019). How the Australian Tax Office is using a virtual assistant to improve self-service. Retrieved February 15, 2023, from https://cxcentral.com.au/advanced-technology/virtual-assistant-to-improve-self-service/.
[11]
Dowling-Hetherington L Transnational higher education and the factors influencing student decision-making: The experience of an Irish university Journal of Studies in International Education 2020 24 3 291-313
[12]
Gimhani RMDG, Kumari S, and Swarnakantha NRS Student Learning and Assessment Support System International Research Journal of Innovations in Engineering and Technology 2023 7 10 123
[13]
Goda Y, Masanori Y, Matsukawa H, and Hata K Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking The Journal of Information and Systems in Education 2014 13 1 1-7
[14]
González CS, Muñoz-Cruz V, Toledo-Delgado PA, and Nacimiento-García E Personalized gamification for learning: A reactive Chatbot Architecture proposal Sensors (Basel, Switzerland) 2023 23 1 545
[15]
Heathcote D, Savage S, and Hosseinian-Far A Factors affecting university choice behaviour in the UK higher education Education Sciences 2020 10 8 199
[16]
Huete-Alcocer, N. (2017). A literature review of Word of Mouth and Electronic Word of Mouth: Implications for consumer behavior. Frontiers in Psychology, 8.
[17]
Hwang GJ and Chang CY A review of opportunities and challenges of chatbots in education Interactive Learning Environments 2023 31 7 4099-4112
[18]
Jalilvand MR and Samiei N The impact of electronic word of mouth on a tourism destination choice: Testing the theory of planned behavior Internet Research 2012 22 5 591-612
[19]
Kim, S., Lee, J., & Gweon, G. (2019). Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–12. ACM Press, NY.
[20]
Kizilcec RF and Schneider E Motivation as a lens to understand online learners: Toward data-driven design with the OLEI scale ACM Transactions on Computer-Human Interaction (TOCHI) 2015 22 2 1-24
[21]
Kooli C Chatbots in education and research: A critical examination of ethical implications and solutions Sustainability 2023 15 7 5614
[22]
Kuhail MA, Alturki N, Alramlawi S, and Alhejori K Interacting with educational chatbots: A systematic review Education and Information Technologies 2023 28 1 973-1018
[23]
Lopez M and Sicilia M eWOM as source of influence: The impact of participation in eWOM and perceived source trustworthiness on decision making Journal of Interactive Advertising 2014 14 2 86-97
[24]
Massoud K and Ayoubi RM Do flexible admission systems affect student enrollment? Evidence from UK universities Journal of Marketing for Higher Education 2019 29 1 84-101
[25]
Meng J, Rheu M, Zhang Y, Dai Y, and Peng W Mediated Social Support for Distress reduction: AI Chatbots vs. Human Proceedings of the ACM on Human-Computer Interaction 2023 7 CSCW1 1-25
[26]
Murray KB A test of services marketing theory: Consumer information acquisition activities Journal of Marketing 1991 55 1 10-25
[27]
Nishant, R., Schneckenberg, D., & Ravishankar, M. N. (2023). The formal rationality of artificial intelligence-based algorithms and the problem of bias. Journal of Information Technology.
[28]
Okonkwo CW and Ade-Ibijola A Chatbots applications in education: A systematic review Computers and Education: Artificial Intelligence 2021 2 100033
[29]
Pandey S, Sharma S, and Wazir S Mental healthcare chatbot based on natural language processing and deep learning approaches: Ted the therapist International Journal of Information Technology 2022 14 7 3757-3766
[30]
Park S, Choi J, Lee S, Oh C, Kim C, La S, Lee J, and Suh B Designing a Chatbot for a brief motivational interview on stress management: Qualitative case study Journal of Medical Internet Research 2019 21 4 e12231
[31]
Park D-M, Jeong SS, and Seo YS Systematic review on Chatbot techniques and applications Journal of Information Processing Systems 2022 18 1 26-47
[32]
Pérez JQ, Daradoumis T, and Puig JMM Rediscovering the use of chatbots in education: A systematic literature review Computer Applications in Engineering Education 2020 28 6 1549-1565
[33]
Prolific (2023). Prolific: A higher standard of online research. Retrieved February 22, 2023, from https://www.prolific.co.
[34]
Reicherts L, Rogers Y, Capra L, Wood E, Duong TD, and Sebire N It’s good to talk: A comparison of using voice versus screen-based interactions for agent-assisted tasks ACM Transactions on Computer-Human Interaction (TOCHI) 2022 29 3 1-41
[35]
Rhim J, Kwak M, Gong Y, and Gweon G Application of humanization to survey chatbots: Change in chatbot perception, interaction experience, and survey data quality Computers in Human Behavior 2022 126 107034
[36]
Sarkar, D. (2016). Semantic and Sentiment Analysis. In Text Analytics with Python: A Practical Real-World Approach to Gaining Actionable Insights from your Data (2016): 319–376. Apress, Berkeley, CA.
[37]
Shanka T, Quintal V, and Taylor R Factors influencing international students’ choice of an education destination–A correspondence analysis Journal of Marketing for Higher Education 2006 15 2 31-46
[38]
Sidaoui K, Jaakkola M, and Burton J AI feel you: Customer experience assessment via chatbot interviews Journal of Service Management 2020 31 4 745-766
[39]
Steffes EM and Burgee LE Social ties and online word of mouth Internet Research 2009 19 1 42-59
[40]
Suresh KP An overview of randomization techniques: An unbiased assessment of outcome in clinical research Journal of Human Reproductive Sciences 2011 4 1 8
[41]
Sweeney C, Potts C, Ennis E, Bond R, Mulvenna MD, O’neill S, Malcolm M, et al. Can Chatbots help support a person’s mental health? Perceptions and views from mental healthcare professionals and experts ACM Transactions on Computing for Healthcare 2021 2 3 1-15
[42]
Te Pas ME, Rutten WG, Bouwman AR, and Buise MP User experience of a Chatbot Questionnaire Versus a regular computer questionnaire: Prospective comparative study JMIR Medical Informatics 2020 8 12 e21982
[43]
Vannuccini, S., & Prytkova, E. (2023). Artificial Intelligence’s new clothes? A system technology perspective. Journal of Information Technology, 02683962231197824.
[44]
Villanueva J, Yoo S, and Hanssens DM The Impact of Marketing-Induced versus Word-of-mouth customer Acquisition on customer equity growth Journal of Marketing Research 2008 45 1 48-59
[45]
Walsh G and Mitchell V-W The effect of consumer confusion proneness on word of mouth, trust, and customer satisfaction European Journal of Marketing 2010 44 6 838-859
[46]
Westbrook RA Product/consumption-based affective responses and post-purchase processes Journal of Marketing Research 1987 24 3 258-270
[47]
Wollny S, Schneider J, Di Mitri D, Weidlich J, Rittberger M, and Drachsler H Are we there yet?-a systematic literature review on chatbots in education Frontiers in Artificial Intelligence 2021 4 654924
[48]
Yan Q and Wu S How differences in eWOM platforms impact consumers’ perceptions and decision-making Journal of Organizational Computing and Electronic Commerce 2018 28 4 315-333
[49]
Yet ML, Ching SY, and Teck HL Destination choice, service quality, satisfaction, and consumerism: International students in Malaysian institutions of higher education African Journal of Business Management 2011 5 5 1691-1702
[50]
Ziang X, Zhou MX, Liao VQ, Mark G, Chi C, Chen W, and Yang H Tell me about yourself: Using an AI-Powered chatbot to Conduct conversational surveys with Open-ended questions ACM Transactions on Computer-Human Interaction (TOCHI) 2020 27 3 1-37
Index Terms
- AI-experiments in education: An AI-driven randomized controlled trial for higher education research
Index terms have been assigned to the content through auto-classification.
Recommendations
Learning what works in ITS from non-traditional randomized controlled trial data
ITS'10: Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part IIThe traditional, well established approach to finding out what works in education research is to run a randomized controlled trial (RCT) using a standard pretest and posttest design RCTs have been used in the intelligent tutoring community for decades ...
Online controlled experiments: introduction, learnings, and humbling statistics
RecSys '12: Proceedings of the sixth ACM conference on Recommender systemsThe web provides an unprecedented opportunity to accelerate innovation by evaluating ideas quickly and accurately using controlled experiments (e.g., A/B tests and their generalizations). Whether for front-end user-interface changes, or backend ...
Enhancing Tutoring Effectiveness Through Automated Feedback: Preliminary Findings from a Pilot Randomized Controlled Trial on SAT Tutoring
L@S '24: Proceedings of the Eleventh ACM Conference on Learning @ ScaleTo address educational inequities, high-quality SAT tutoring is crucial for students who need support. Many tutors, however, are novices and require coaching to be effective. To examine this issue, we conducted a pilot randomized controlled trial (RCT) ...
Comments
Information & Contributors
Information
Published In
© The Author(s) 2024.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 26 March 2024
Accepted: 14 March 2024
Received: 22 November 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
- The University of Adelaide
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 26 Jan 2025