research-article

Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study

Author:

Abdulhadi ShoufanAuthors Info & Claims

ACM Transactions on Computing Education, Volume 23, Issue 4

Article No.: 45, Pages 1 - 29

https://doi.org/10.1145/3628162

Published: 11 December 2023 Publication History

Abstract

With the immense interest in ChatGPT worldwide, education has seen a mix of both excitement and skepticism. To properly evaluate its impact on education, it is crucial to understand how far it can help students without prior knowledge answer assessment questions. This study aims to address this question as well as the impact of the question type. We conducted multiple experiments with computer engineering students (experiment group: n=41 to 56), who were asked to use ChatGPT to answer previous test questions before learning about the related topics. Their scores were then compared with the scores of previous-term students who answered the same questions in a quiz or exam setting (control group: n=24 to 61). The results showed a wide range of effect sizes, from -2.55 to 1.23, depending on the question type and content. The experiment group performed best answering code analysis and conceptual questions but struggled with code completion and questions that involved images. However, the performance in code generation tasks was inconsistent. Overall, the ChatGPT group’s answers lagged slightly behind the control group’s answers with an effect size of -0.16. We conclude that ChatGPT, at least in the field of this study, is not yet ready to rely on by students who do not have sufficient background to evaluate generated answers. We suggest that educators try using ChatGPT and educate students on effective questioning techniques and how to assess the generated responses. This study provides insights into the capabilities and limitations of ChatGPT in education and informs future research and development.

References

[1]

Eleni Adamopoulou and Lefteris Moussiades. 2020. Chatbots: History, technology, and applications. Mach. Learn. Appl. 2 (2020), 100006.

[2]

Mohammed A. Al-Sharafi, Mostafa Al-Emran, Mohammad Iranmanesh, Noor Al-Qaysi, Noorminshah A. Iahad, and Ibrahim Arpaci. 2022. Understanding the impact of knowledge management factors on the sustainable use of AI-based chatbots for educational purposes using a hybrid SEM-ANN approach. Interact. Learn. Environ. (2022), 1–20.

[3]

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung et al. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. Retrieved from https://arXiv:2302.04023

[4]

Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming is hard-or at least it used to be: Educational opportunities and challenges of AI code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 500–506.

Digital Library

[5]

Luciana Benotti, Mara Cecilia Martnez, and Fernando Schapachnik. 2017. A tool for introducing computer science with automatic formative assessment. IEEE Trans. Learn. Technol. 11, 2 (2017), 179–192.

[6]

Yu Chen, Scott Jensen, Leslie J. Albert, Sambhav Gupta, and Terri Lee. 2022. Artificial intelligence (AI) student assistants in the classroom: Designing chatbots to support student success. Info. Syst. Front. 25, 1 (2022), 1–22.

[7]

Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 1136–1142.

Digital Library

[8]

M. El-Abd. 2023. A review of embedded systems education in the arduino age: lessons learned and future directions. International Journal of Engineering Pedagogy 7, 2 (2023), 79–93. Retrieved November 2, 2023 from https://www.learntechlib.org/p/207404/

[9]

Harry Barton Essel, Dimitrios Vlachopoulos, Akosua Tachie-Menson, Esi Eduafua Johnson, and Papa Kwame Baah. 2022. The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. Int. J. Edu. Technol. Higher Edu. 19, 1 (2022), 1–19.

[10]

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the Australasian Computing Education Conference. 10–19.

Digital Library

[11]

James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI wants to know if this will be on the exam: testing OpenAI’s codex on CS2 programming exercises. In Proceedings of the 25th Australasian Computing Education Conference. 97–104.

Digital Library

[12]

Aidan Gilson, Conrad W. Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023. How does ChatGPT perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med. Edu. 9, 1 (2023), e45312.

[13]

Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots classification and design techniques. In Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications: Web, Artificial Intelligence, and Network Applications (WAINA’19) 33. Springer, 946–956.

[14]

Maria Karyotaki, Athanasios Drigas, and Charalabos Skianis. 2022. Chatbots as cognitive, educational, advisory & coaching systems. Technium Soc. Sci. J. 30 (2022), 109–126.

[15]

Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences 103 (2023), 102274. DOI:

[16]

Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI code generators on supporting novice learners in introductory programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–23.

Digital Library

[17]

Hari Kishan Kondaveeti, Nandeesh Kumar Kumaravelu, Sunny Dayal Vanambathina, Sudha Ellison Mathe, and Suseela Vappangi. 2021. A systematic literature review on prototyping with Arduino: Applications, challenges, advantages, and limitations. Comput. Sci. Rev. 40 (2021), 100364.

Digital Library

[18]

Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2022. Interacting with educational chatbots: A systematic review. Edu. Info. Technol. 28, 1 (2022), 1–46.

[19]

Harsh Kumar, Kunzhi Yu, Andrew Chung, Jiakai Shi, and Joseph Jay Williams. 2022. Exploring the potential of chatbots to provide mental well-being support for computer science students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 1339–1339.

Digital Library

[20]

Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Dig. Health 2, 2 (2023), e0000198.

[21]

Sam Lau and Philip Guo. 2023. From “Ban it till we understand it” to “Resistance is futile”: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, 106–121.

[22]

Keeheon Lee, Jeongwon Jo, Jinyoung Kim, and Younah Kang. 2019. Can chatbots help reduce the workload of administrative officers?-Implementing and deploying FAQ chatbot service in a university. In Proceedings of the 21st International Conference on Human-Computer Interaction (HCII’19). Springer, 348–354.

[23]

Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 563–569.

Digital Library

[24]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097.

[25]

Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education. 931–937.

Digital Library

[26]

Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the GPT-3 large language model. In Proceedings of the ACM Conference on International Computing Education Research. 37–39.

Digital Library

[27]

Beth McMurtrie. 2023. Teaching Experts Are Worried About ChatGPT, but Not for the Reasons You Think. Retrieved from https://www.chronicle.com/article/ai-and-the-future-of-undergraduate-writing

[28]

Ifeanyi G. Ndukwe, Ben K. Daniel, and Chukwudi E. Amadi. 2019. A machine learning grading system using chatbots. In Proceedings of the20th International Conference in Artificial Intelligence in Education (AIED’19). Springer, 365–368.

[29]

Ha Nguyen. 2023. Role design considerations of conversational agents to facilitate discussion and systems thinking. Comput. Edu. 192 (2023), 104661.

Digital Library

[30]

Trung Thanh Nguyen, Anh Duc Le, Ha Thanh Hoang, and Tuan Nguyen. 2021. NEU-chatbot: Chatbot for admission of national economics university. Comput. Edu.: Artific. Intell. 2 (2021), 100036.

[31]

Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. 2021. Chatbots applications in education: A systematic review. Comput. Edu.: Artific. Intell. 2 (2021), 100033.

[32]

Brian Owens. 2023. How Nature readers are using ChatGPT. Retrieved from https://www.nature.com/articles/d41586-023-00500-8

[33]

José Quiroga Pérez, Thanasis Daradoumis, and Joan Manuel Marquès Puig. 2020. Rediscovering the use of chatbots in education: A systematic literature review. Comput. Appl. Eng. Edu. 28, 6 (2020), 1549–1565.

[34]

Jürgen Rudolph, Samson Tan, and Shannon Tan. 2023. ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? J. Appl. Learn. Teach. 6, 1 (2023).

[35]

María Consuelo Sáiz-Manzanares, Raúl Marticorena-Sánchez, Luis Jorge Martín-Antón, Irene González Díez, and Leandro Almeida. 2023. Perceived satisfaction of university students with the use of chatbots as a tool for self-regulated learning. Heliyon 9, 1 (2023), e12843.

[36]

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the ACM Conference on International Computing Education Research. 27–43.

Digital Library

[37]

Shipra Sharma and Balwinder Sodhi. 2023. Calculating originality of LLM assisted source code. Retrieved from https://arXiv:2307.04492

[38]

Mike Sharples. 2022. Automated essay writing: An AIED opinion. Int. J. Artific. Intell. Edu. 32, 4 (2022), 1119–1126.

[39]

Abdulhadi Shoufan. 2020. Lecture-free classroom: Fully active learning on Moodle. IEEE Trans. Edu. 63, 4 (2020), 314–321.

[40]

Abdulhadi Shoufan. 2021. Active distance learning of embedded systems. IEEE Access 9 (2021), 41104–41122.

[41]

A. Shoufan. 2023. Exploring students’ perceptions of ChatGPT: Thematic analysis and follow-up survey. IEEE Access 11 (2023), 38805–38818. DOI:

[42]

Guttorm Sindre. 2020. Code writing vs code completion puzzles: Analyzing questions in an E-Exam. In Proceedings of the IEEE Frontiers in Education Conference (FIE’20). IEEE, 1–9.

Digital Library

[43]

Pavel Smutny and Petra Schreiberova. 2020. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Comput. Edu. 151 (2020), 103862.

[44]

Teo Susnjak. 2022. ChatGPT: The end of online exam integrity? Retrieved from https://arXiv:2212.09292

[45]

Andrew Tran, Linxuan Li, Egi Rama, Kenneth Angelikas, and Stephen MacNeil. 2023. Using large language models to automatically identify programming concepts in code snippets. Sci. Edu. 1 (2023), 563–569.

[46]

José Fidel Urquiza-Yllescas, Sonia Mendoza, José Rodríguez, and Luis Martín Sánchez-Adame. 2022. An approach to the classification of educational chatbots. Journal of Intelligent & Fuzzy Systems 43, 4 (2022), 5095–5107.

[47]

Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.

Digital Library

[48]

Eva A. M. van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L. Bockting. 2023. ChatGPT: Five priorities for research. Retrieved from https://www.nature.com/articles/d41586-023-00288-7

[49]

Michel Wermelinger. 2023. Using GitHub Copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 172–178.

Digital Library

[50]

Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. Retrieved from https://arXiv:2302.11382

[51]

Shanshan Yang and Chris Evans. 2019. Opportunities and challenges in using AI chatbots in higher education. In Proceedings of the 3rd International Conference on Education and E-Learning. 79–83.

Digital Library

Cited By

Prather JLeinonen JKiesler NGorson Benario JLau SMacNeil SNorouzi NOpel SPettit VPorter LReeves BSavelka JSmith DStrickroth SZingaro DMonga MLonati VBarendsen ESheard JPaterson J(2025)Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709614(300-338)Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1145/3689187.3709614
Mailach AGorgosch DSiegmund NSiegmund J(2025)“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbotEmpirical Software Engineering10.1007/s10664-024-10561-630:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10561-6
Güner HEr E(2025)AI in the classroom: Exploring students’ interaction with ChatGPT in programming learningEducation and Information Technologies10.1007/s10639-025-13337-7Online publication date: 15-Jan-2025
https://doi.org/10.1007/s10639-025-13337-7
Show More Cited By

Index Terms

Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study

Recommendations

The effectiveness of ChatGPT on pre-service teachers' STEM teaching literacy, learning performance, and cognitive load in a teacher training course
ICETM '23: Proceedings of the 2023 6th International Conference on Educational Technology Management

Within the realm of teacher education, effective tools as support can enhance pre-service teachers' comprehension of STEM concepts and their ability to tackle with complex problems. This research aims to empirically investigate the impact of ChatGPT-...
Beyond the Textbook: A Study of ChatGPT Patterns of Use Perceptions and Experiences Among Students in Higher Education
ICETC '24: Proceedings of the 2024 16th International Conference on Education Technology and Computers
This study investigates the utilization pattern, perception, and experience of Higher Education Institutes (HEIs) students towards the ChatGPT application in an academic context. It employs a quantitative approach utilizing a questionnaire as the research ...
ChatGPT Didn't Take Me Very Far, Did It?
CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 2

The effect of ChatGPT (CG) on teachers, assessment and other matters have been discussed in many works, but no one appears to have studied how students are actually using it. We study how students use CG for coursework, a semester long project, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computing Education

ACM Transactions on Computing Education Volume 23, Issue 4

December 2023

213 pages

EISSN:1946-6226

DOI:10.1145/3631944

Editor:
Amy J. Ko
University of Washington, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 December 2023

Online AM: 18 October 2023

Accepted: 12 October 2023

Revised: 24 July 2023

Received: 01 April 2023

Published in TOCE Volume 23, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
2,022
Total Downloads

Downloads (Last 12 months)1,542
Downloads (Last 6 weeks)92

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Prather JLeinonen JKiesler NGorson Benario JLau SMacNeil SNorouzi NOpel SPettit VPorter LReeves BSavelka JSmith DStrickroth SZingaro DMonga MLonati VBarendsen ESheard JPaterson J(2025)Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709614(300-338)Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1145/3689187.3709614
Mailach AGorgosch DSiegmund NSiegmund J(2025)“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbotEmpirical Software Engineering10.1007/s10664-024-10561-630:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10561-6
Güner HEr E(2025)AI in the classroom: Exploring students’ interaction with ChatGPT in programming learningEducation and Information Technologies10.1007/s10639-025-13337-7Online publication date: 15-Jan-2025
https://doi.org/10.1007/s10639-025-13337-7
Kosar TOstojić DLiu YMernik M(2024)Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice ProgrammersMathematics10.3390/math1205062912:5(629)Online publication date: 21-Feb-2024
https://doi.org/10.3390/math12050629
Alawida MAbu Shawar BAbiodun OMehmood AOmolara AAl Hwaitat A(2024)Unveiling the Dark Side of ChatGPT: Exploring Cyberattacks and Enhancing User AwarenessInformation10.3390/info1501002715:1(27)Online publication date: 2-Jan-2024
https://doi.org/10.3390/info15010027
Jelson ALee S(2024)An empirical study to understand how students use ChatGPT for writing essays and how it affects their ownershipProceedings of the Third Workshop on Intelligent and Interactive Writing Assistants10.1145/3690712.3690720(26-30)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3690712.3690720
Stone I(2024)Exploring Human-Centered Approaches in Generative AI and Introductory Programming Research: A Scoping ReviewProceedings of the 2024 Conference on United Kingdom & Ireland Computing Education Research10.1145/3689535.3689553(1-7)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3689535.3689553
Mahon JMac Namee BBecker BMonga MLonati VBarendsen ESheard JPaterson J(2024)Guidelines for the Evolving Role of Generative AI in Introductory Programming Based on Emerging PracticeProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653602(10-16)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653602
Quan SDu YDing Y(2024)Young Children and ChatGPT: Parents' Use of ChatGPT in ParentingExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650880(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650880
Shoufan H(2024)Beware the Hype Around Information Technology!2024 IEEE Integrated STEM Education Conference (ISEC)10.1109/ISEC61299.2024.10664676(1-2)Online publication date: 9-Mar-2024
https://doi.org/10.1109/ISEC61299.2024.10664676
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents