Abstract
Teacher educators use digital clinical simulations (DCS) to provide improvisation opportunities within low-stakes classroom environments. In this study, we experimented with GPT-3 and few-shot learning to examine if it could be used with open-text DCS responses. We found that GPT-3 performed substantially worse than traditional machine learning (ML) models even on the same-sized training sets. However, the performance of GPT-3 decreased only marginally compared to traditional ML models with a training set of 20 examples (−0.06). Traditional ML models generally performed well and in some cases had similar performance to the human baseline. Future research will examine whether changes to labeling procedures or fine-tuning with existing data can improve the performance of GPT-3 with DCSs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brown, T.B., et al.: Language models are few-shot learners. arXiv:2005.14165 [cs], July 2020. http://arxiv.org/abs/2005.14165
Bywater, J.P., Chiu, J.L., Hong, J., Sankaranarayanan, V.: The Teacher Responding Tool: Scaffolding the teacher practice of responding to student ideas in mathematics classrooms. Comput. Educ. 139, 16–30 (2019). https://doi.org/10.1016/j.compedu.2019.05.004, https://www.sciencedirect.com/science/article/pii/S0360131519301137
Clavié, B., Gal, K.: EduBER: pretrained deep language models for learning analytics. In: Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20), p. 4 (2020)
Floridi, L., Chiriatti, M.: GPT-3: its nature, scope, limits, and consequences. Minds Mach. 30(4), 681–694 (2020). https://doi.org/10.1007/s11023-020-09548-1
Hillaire, G., et al.: Teacher moments: a digital clinical simulation platform with extensible AI architecture. Technical report, EdArXiv, May 2021. https://doi.org/10.35542/osf.io/jf348, https://edarxiv.org/jf348/, type: article
Liu, O.L., Brew, C., Blackmore, J., Gerard, L., Madhok, J., Linn, M.C.: Automated scoring of constructed-response science items: prospects and obstacles. Educ. Meas. Issues Pract. 33(2), 19–28 (2014). https://doi.org/10.1111/emip.12028, https://onlinelibrary.wiley.com/doi/abs/10.1111/emip.12028, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/emip.12028
Mikeska, J., Howell, H., Dieker, L., Hynes, M.: Understanding the role of simulations in k-12 mathematics and science teacher education: outcomes from a teacher education simulation conference. Contemp. Issues Technol. Teach. Educ. 21(3) (2021). https://citejournal.org/volume-21/issue-3-21/general/understanding-the-role-of-simulations-in-k-12-mathematics-and-science-teacher-education-outcomes-from-a-teacher-education-simulation-conference
OpenAI: OpenAI Documentation (2022). https://beta.openai.com/docs/introduction
Pedregosa, F., et al.: Scikit-learn: machine Learning in Python. Mach. Learn. Python 6 (2011)
Wang, S., Liu, Y., Xu, Y., Zhu, C., Zeng, M.: Want to reduce labeling cost? GPT-3 can help. arXiv:2108.13487 [cs], August 2021. http://arxiv.org/abs/2108.13487
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Littenberg-Tobias, J., Marvez, G.R., Hillaire, G., Reich, J. (2022). Comparing Few-Shot Learning with GPT-3 to Traditional Machine Learning Approaches for Classifying Teacher Simulation Responses. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium. AIED 2022. Lecture Notes in Computer Science, vol 13356. Springer, Cham. https://doi.org/10.1007/978-3-031-11647-6_95
Download citation
DOI: https://doi.org/10.1007/978-3-031-11647-6_95
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-11646-9
Online ISBN: 978-3-031-11647-6
eBook Packages: Computer ScienceComputer Science (R0)