Understanding the Role of Temperature in Diverse Question Generation by GPT-4
Authors: 
Arav Agarwal, Karthik Mittal, Aidan Doyle, Pragnya Sridhar, Zipiao Wan, Jacob Arthur Doughty, Jaromir Savelka, 
Majd SakrAuthors Info & Claims
Arav Agarwal, Karthik Mittal, Aidan Doyle, Pragnya Sridhar, Zipiao Wan, Jacob Arthur Doughty, Jaromir Savelka, Majd SakrAuthors Info & ClaimsPages 1550 - 1551
Abstract
We conduct a preliminary study of the effect of GPT's temperature parameter on the diversity of GPT4-generated questions. We find that using higher temperature values leads to significantly higher diversity, with different temperatures exposing different types of similarity between generated sets of questions. We also demonstrate that diverse question generation is especially difficult for questions targeting lower levels of Bloom's Taxonomy.
References
[1]
Benjamin S Bloom, Max D Englehart, Edward J Furst, Walker H Hill, David R Krathwohl, et al. 1956. Taxonomy of educational objectives, handbook I: the cognitive domain. New York: David McKay Co.
[2]
Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. 2023. Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources. arXiv:2306.10509 [cs.HC]
[3]
Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, and Majd Sakr. 2024. A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education. In Proceedings of the 26th Australasian Computing Education Conference ACE '24). Association for Computing Machinery, New York, NY, USA, 114--123. https://doi.org/10.1145/3636243.3636256
[4]
J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174. http://www.jstor.org/stable/2529310
[5]
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Association for Computing Machinery, NewYork, NY, USA, 931--937. https://doi.org/10.1145/3545945.3569785
[6]
Pranjal Dilip Naringrekar, Ildar Akhmetov, and Eleni Stroulia. 2023. Generating CS1 Coding Questions Using OpenAI. In Proceedings of the 25th Western Canadian Conference on Computing Education (Vancouver, BC, Canada) (WCCCE '23). Association for Computing Machinery, New York, NY, USA, Article 11, 2 pages. https://doi.org/10.1145/3593342.3593348
Index Terms
- Understanding the Role of Temperature in Diverse Question Generation by GPT-4
Recommendations
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
ACE '24: Proceedings of the 26th Australasian Computing Education ConferenceThere is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding ...
Automating Question Generation From Educational Text
Artificial Intelligence XLAbstractThe use of question-based activities (QBAs) is wide-spread in education, traditionally forming an integral part of the learning and assessment process. In this paper, we design and evaluate an automated question generation tool for formative and ...
Automatic Generation of Usage Manuals for Open-Source Software
CISIS '12: Proceedings of the 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS)Recently, the Open Source Software (OSS) has become used at many places. Even the installation and initial configuration of an OSS can be difficult for novice users, because there are few documents and manuals. To address this problem, we have proposed ...
Comments
Information & Contributors
Information
Published In
March 2024
2007 pages
ISBN:9798400704246
DOI:10.1145/3626253
- General Chairs:
- Ben Stephenson,
- Jeffrey A. Stone,
- Program Chairs:
- Lina Battestilli,
- Samuel A. Rebelsky,
- Libby Shoop
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 15 March 2024
Check for updates
Author Tags
Qualifiers
- Poster
Conference
SIGCSE 2024
Sponsor:
SIGCSE 2024: The 55th ACM Technical Symposium on Computer Science Education
March 20 - 23, 2024
OR, Portland, USA
Acceptance Rates
Overall Acceptance Rate 1,595 of 4,542 submissions, 35%
Upcoming Conference
SIGCSE Virtual 2024
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 86Total Downloads
- Downloads (Last 12 months)86
- Downloads (Last 6 weeks)12
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in