Beyond Functional Correctness: An Exploratory Study on the Time Efficiency of Programming Assignments
Authors: 
Yida Tao, Wenyan Chen, Qingyang Ye, Yao ZhaoAuthors Info & Claims
Yida Tao, Wenyan Chen, Qingyang Ye, Yao ZhaoAuthors Info & ClaimsPages 320 - 330
Abstract
Practical programming assignments are critical parts of programming courses in Computer Science education. Students are expected to translate programming concepts learned from lectures into executable implementations that solve the tasks outlined in the assignments. These implementations are primarily assessed based on their functional correctness, ensuring that students' code produces the expected output when provided with specific inputs.
However, functional correctness is not the only metric that evaluates the quality of programs. Runtime efficiency is a metric that is less frequently evaluated in programming courses, yet it holds significant importance in the context of professional software development. To investigate this gap and its potential ramifications, we conducted a large-scale empirical study on the time efficiency of 250 programming assignments that are evaluated solely on functional correctness. The results demonstrate that students' programming assignments exhibit significant variance in terms of execution time. We further identified 27 recurring inefficient code patterns from these assignments, and observed that most of the inefficient patterns can be optimized by automated tools such as PMD, IntelliJ IDEA and ChatGPT. Our findings provide actionable guidelines for educators to enhance the organization and integration of code performance topics throughout the programming course curriculum.
References
[1]
2023. Assignment Dataset and Other Artifacts. https://drive.google.com/file/d/1mfaJQqva3QZfOsM8TBehSVwTINhEwM08/view?usp=sharing
[2]
2023. IntelliJ IDEA - the Leading Java and Kotlin IDE. https://www.jetbrains.com/idea/
[3]
2023. Java Microbenchmark Harness (JMH). https://github.com/openjdk/jmh
[4]
2023. List of Java inspections. https://www.jetbrains.com/help/idea/list-of-java-inspections.html
[5]
2023. PMD: An extensible cross-language static code analyzer. https://pmd.github.io/
[6]
2023. PMD Predefined Rules: Performance. https://docs.pmd-code.org/latest/pmd_rules_java_performance.html
[7]
LJ Anthony. 2003. The Cambridge dictionary of statistics. Reference Reviews 17, 1 (2003), 29--30.
[8]
Brenda Cheang, Andy Kurnia, Andrew Lim, and Wee-Chong Oon. 2003. On automated grading of programming assignments in an academic institution. Comput. Educ. 41 (2003), 121--131.
[9]
Benjamin Clegg, Maria-Cruz Villa-Uriol, Phil McMinn, and Gordon Fraser. 2021. Gradeer: An Open-Source Modular Hybrid Grader. (2021), 60--65.
[10]
Benjamin Simon Clegg. 2021. The application of mutation testing to enhance the automated assessment of introductory programming assignments. Ph. D. Dissertation. University of Sheffield.
[11]
Sébastien Combéfis. 2022. Automated code assessment for education: review, classification and perspectives on techniques and tools. Software 1, 1 (2022), 3--30.
[12]
Sébastien Combéfis and Vianney le CLÉMENT de SAINT-MARCQ. 2012. Teaching Programming and Algorithm Design with Pythia, a Web-Based Learning Platform. Olympiads in Informatics 6 (2012).
[13]
Gordon Fraser, Ute Heuer, Nina Körber, Florian Obermüller, and Ewald Wasmeier. 2021. Litterbox: A linter for scratch programs. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). IEEE, 183--188.
[14]
J. E. Gaffney. 1981. Metrics in Software Quality Assurance. In Proceedings of the ACM '81 Conference (ACM '81). Association for Computing Machinery, New York, NY, USA, 126--130.
[15]
Chuqin Geng, Zhang Yihan, Brigitte Pientka, and Xujie Si. 2023. Can ChatGPT Pass An Introductory Level Functional Language Programming Course? arXiv preprint arXiv:2305.02230 (2023).
[16]
Austin Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021. An Inquisitive Code Editor for Addressing Novice Programmers' Misconceptions of Program Behavior. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 165--170.
[17]
Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023. Chatgpt and software testing education: Promises & perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 4130--4137.
[18]
Ville Karavirta and Petri Ihantola. 2010. Automatic assessment of javascript exercises. Proceedings of the 1st Educators' Day on Web Engineering Curricula, Vienna, Austria (2010), 5--9.
[19]
David Kawrykow and Martin P Robillard. 2009. Detecting inefficient API usage. In 2009 31st International Conference on Software Engineering-Companion Volume. IEEE, 183--186.
[20]
Roope Luukkainen, Jussi Kasurinen, Uolevi Nikula, and Valentina Lenarduzzi. 2022. ASPA: A Static Analyser to Support Learning and Continuous Feedback on Programming Courses. An Empirical Validation. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 29--39.
[21]
Wellington Oliveira, Renato Oliveira, Fernando Castor, Benito Fernandes, and Gustavo Pinto. 2019. Recommending energy-efficient java collections. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 160--170.
[22]
OpenAI. 2023. ChatGPT. https://openai.com/blog/chatgpt
[23]
José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Trans. Comput. Educ. 22, 3, Article 34 (jun 2022), 40 pages.
[24]
Sagar Parihar, Ziyaan Dadachanji, Praveen Kumar Singh, Rajdeep Das, Amey Karkare, and Arnab Bhattacharya. 2017. Automatic Grading and Feedback Using Program Repair for Introductory Programming Courses. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). Association for Computing Machinery, 92--97.
[25]
Kai Presler-Marshall, Sarah Heckman, and Kathryn Stolee. 2021. SQLRepair: Identifying and Repairing Mistakes in Student-Authored SQL Queries. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 199--210.
[26]
Basit Qureshi. 2023. Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv preprint arXiv:2304.11214 (2023).
[27]
Jake Renzella and Andrew Cain. 2020. Enriching Programming Student Feedback with Audio Comments (ICSE-SEET '20). Association for Computing Machinery, New York, NY, USA, 173--183.
[28]
Marija Selakovic and Michael Pradel. 2016. Performance issues and optimizations in javascript: an empirical study. In Proceedings of the 38th International Conference on Software Engineering. 61--72.
[29]
Yida Tao, Shan Tang, Yepang Liu, Zhiwu Xu, and Shengchao Qin. 2021. Speeding up data manipulation tasks with alternative implementations: an exploratory study. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 4 (2021), 1--28.
[30]
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the Ultimate Programming Assistant-How far is it? arXiv preprint arXiv:2304.11938 (2023).
[31]
Tianjia Wang, Daniel Vargas-Diaz, Chris Brown, and Yan Chen. 2023. Towards Adapting Computer Science Courses to AI Assistants' Capabilities. arXiv preprint arXiv:2306.03289 (2023).
[32]
Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023).
[33]
Junwen Yang, Pranav Subramaniam, Shan Lu, Cong Yan, and Alvin Cheung. 2018. How not to structure your database-backed web applications: a study of performance bugs in the wild. In Proceedings of the 40th International Conference on Software Engineering. 800--810.
Index Terms
- Beyond Functional Correctness: An Exploratory Study on the Time Efficiency of Programming Assignments
Recommendations
Developing a common format for sharing programming assignments
Computer science educators spend a lot of effort designing programming assignments, and many are willing to share the results of this investment. However, sharing of programming assignments occurs primarily in an ad hoc manner through informal channels. ...
Developing real-world programming assignments for CS1
ITICSE '06: Proceedings of the 11th annual SIGCSE conference on Innovation and technology in computer science educationInstructors have struggled to generate good programming assignments for the CS1 course. In attempting to deal with this issue ourselves, we have generated two real-world programming assignments that can be solved by most students yet generate challenges ...
Using hypersafety verification for proving correctness of programming assignments
ICSE-NIER '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging ResultsWe look at the problem of deciding correctness of a programming assignment submitted by a student, with respect to a reference implementation provided by the teacher, the correctness property being output equivalence of the two programs. Typically, ...
Comments
Information & Contributors
Information
Published In
April 2024
417 pages
ISBN:9798400704987
DOI:10.1145/3639474
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Kiev Gama,
- Janet Siegmund,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- SUSTech Undergraduate Teaching Quality and Education Reform Project
Conference
ICSE-SEET '24
Sponsor:
ICSE-SEET '24: 46th International Conference on Software Engineering: Software Engineering Education and Training
April 14 - 20, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 61Total Downloads
- Downloads (Last 12 months)61
- Downloads (Last 6 weeks)10
Reflects downloads up to 20 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in