research-article

Feedback generation for performance problems in introductory programming assignments

Authors:

Florian ZulegerAuthors Info & Claims

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 41 - 51

https://doi.org/10.1145/2635868.2635912

Published: 11 November 2014 Publication History

Abstract

Providing feedback on programming assignments manually is a tedious, error prone, and time-consuming task. In this paper, we motivate and address the problem of generating feedback on performance aspects in introductory programming assignments. We studied a large number of functionally correct student solutions to introductory programming assignments and observed: (1) There are different algorithmic strategies, with varying levels of efficiency, for solving a given problem. These different strategies merit different feedback. (2) The same algorithmic strategy can be implemented in countless different ways, which are not relevant for reporting feedback on the student program. We propose a light-weight programming language extension that allows a teacher to define an algorithmic strategy by specifying certain key values that should occur during the execution of an implementation. We describe a dynamic analysis based approach to test whether a student's program matches a teacher's specification. Our experimental results illustrate the effectiveness of both our specification language and our dynamic analysis. On one of our benchmarks consisting of 2316 functionally correct implementations to 3 programming problems, we identified 16 strategies that we were able to describe using our specification language (in 95 minutes after inspecting 66, i.e., around 3%, implementations). Our dynamic analysis correctly matched each implementation with its corresponding specification, thereby automatically producing the intended feedback.

References

[1]

Making Programs Efficient. http://pexforfun.com/makingprogramsefficient.

[2]

Microsoft ”Roslyn” CTP. http: //msdn.microsoft.com/en-us/vstudio/roslyn.aspx.

[3]

Pex for fun. http://www.pexforfun.com/.

[4]

A. Adam and J.-P. H. Laurent. LAURA, a system to debug student programs. Artif. Intell., 15(1-2), 1980.

[5]

P. Bose, J. F. Buss, and A. Lubiw. Pattern matching for permutations. Inf. Process. Lett., 65(5):277–283, 1998.

Digital Library

[6]

J. Burnim, N. Jalbert, C. Stergiou, and K. Sen. Looper: Lightweight detection of infinite loops at runtime. In ASE, pages 161–169, 2009.

Digital Library

[7]

S. Goldsmith, A. Aiken, and D. S. Wilkerson. Measuring empirical computational complexity. In ESEC/SIGSOFT FSE, 2007.

Digital Library

[8]

S. Gulwani. Example-based learning in computer-aided STEM education. To appear in Commun. ACM, 2014.

Digital Library

[9]

S. Gulwani, K. K. Mehra, and T. M. Chilimbi. Speed: precise and efficient static estimation of program computational complexity. In POPL, pages 127–139, 2009.

Digital Library

[10]

S. Gulwani, I. Radiˇ cek, and F. Zuleger. Feedback generation for performance problems in introductory programming assignments. CoRR, abs/1403.4064, 2014.

Digital Library

[11]

S. Gulwani and F. Zuleger. The reachability-bound problem. In PLDI, pages 292–304, 2010.

Digital Library

[12]

A. Gupta, T. A. Henzinger, R. Majumdar, A. Rybalchenko, and R.-G. Xu. Proving non-termination. In POPL, pages 147–158, 2008.

Digital Library

[13]

P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppälä. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA, 2010. ACM.

Digital Library

[14]

D. Jackson and M. Usher. Grading student programs using ASSYST. In SIGCSE, pages 335–339, 1997.

Digital Library

[15]

K. Masters. A brief guide to understanding MOOCs. The Internet Journal of Medical Education, 1(2), 2011.

[16]

R. Milner. An algebraic definition of simulation between programs. Technical report, Stanford, CA, USA, 1971.

Digital Library

[17]

W. R. Murray. Automatic program debugging for intelligent tutoring systems. Computational Intelligence, 3, 1987.

[18]

K. Nguyen and G. Xu. Cachetor: Detecting cacheable data to remove bloat. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 268–278, New York, NY, USA, 2013. ACM.

Digital Library

[19]

A. Nistor, L. Song, D. Marinov, and S. Lu. Toddler: Detecting performance problems via similar memory-access patterns. In Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pages 562–571, Piscataway, NJ, USA, 2013. IEEE Press.

Digital Library

[20]

R. Saikkonen, L. Malmi, and A. Korhonen. Fully automatic assessment of programming exercises. In Proceedings of the 6th Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE ’01, pages 133–136, New York, NY, USA, 2001. ACM.

Digital Library

[21]

R. Singh, S. Gulwani, and A. Solar-Lezama. Automated feedback generation for introductory programming assignments. In PLDI, pages 15–26, 2013.

Digital Library

[22]

M. Striewe and M. Goedicke. Using run time traces in automated programming tutoring. In ITiCSE, pages 303–307, 2011.

Digital Library

[23]

M. Striewe and M. Goedicke. Trace alignment for automated tutoring. In CAA, 2013.

[24]

N. Tillmann and J. de Halleux. Pex-white box test generation for .NET. In TAP, pages 134–153, 2008.

Digital Library

[25]

N. Tillmann, J. de Halleux, T. Xie, S. Gulwani, and J. Bishop. Teaching and learning programming and software engineering via interactive gaming. In ICSE, 2013.

Digital Library

[26]

T. Uno. Algorithms for enumerating all perfect, maximum and maximal matchings in bipartite graphs. In ISAAC, pages 92–101, 1997.

Digital Library

[27]

F. Zuleger, S. Gulwani, M. Sinn, and H. Veith. Bound analysis of imperative programs with the size-change abstraction. In SAS, pages 280–297, 2011. Introduction Overview Motivation Specifying Algorithmic Strategies Specifications and Implementations The Language L Student Implementation Teacher Specification Matching Trace Embedding Partial Matching Full Matching Extensions Implementation and Experiments Experimental Setup Methodology Evaluation Threats to Validity Related Work Automated Feedback Performance Analysis References

Digital Library

Cited By

Agarwal N(2024)Finding and Investigating Buggy Codes to Make CS1 Learning EfficientProceedings of the 17th Innovations in Software Engineering Conference10.1145/3641399.3641442(1-3)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1145/3641399.3641442
Hull MPednekar VMurray HRoy NTung ERoutray SGuerin CChen JWang ZLee SRoozbahani MChau D(2024)VG: Automatic Grading of D3 VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332718130:1(617-627)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327181
Liu YYang ZFan LWu YChen XZhou X(2024)Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programsJournal of Software: Evolution and Process10.1002/smr.2679Online publication date: 27-May-2024
https://doi.org/10.1002/smr.2679
Show More Cited By

Index Terms

Feedback generation for performance problems in introductory programming assignments
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Automated feedback generation for introductory programming assignments
PLDI '13: Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that ...
Automated clustering and program repair for introductory programming assignments
PLDI '18

Providing feedback on programming assignments is a tedious task for the instructor, and even impossible in large Massive Open Online Courses with thousands of students. Previous research has suggested that program repair techniques can be used to ...
Automated feedback generation for introductory programming assignments
PLDI '13

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2014

856 pages

ISBN:9781450330565

DOI:10.1145/2635868

General Chair:
Shing-Chi Cheung
Hong Kong University of Science and Technology, China
,
Program Chairs:
Alessandro Orso
Georgia Institute of Technology, USA
,
Margaret-Anne Storey
University of Victoria, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGSOFT/FSE'14

Sponsor:

SIGSOFT

SIGSOFT/FSE'14: 22nd ACM SIGSOFT Symposium on the Foundations of Software Engineering

November 16 - 21, 2014

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
504
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)5

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Agarwal N(2024)Finding and Investigating Buggy Codes to Make CS1 Learning EfficientProceedings of the 17th Innovations in Software Engineering Conference10.1145/3641399.3641442(1-3)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1145/3641399.3641442
Hull MPednekar VMurray HRoy NTung ERoutray SGuerin CChen JWang ZLee SRoozbahani MChau D(2024)VG: Automatic Grading of D3 VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332718130:1(617-627)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3327181
Liu YYang ZFan LWu YChen XZhou X(2024)Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programsJournal of Software: Evolution and Process10.1002/smr.2679Online publication date: 27-May-2024
https://doi.org/10.1002/smr.2679
Li LLiu HLi KJiang YSun R(2023)Generating Concise Patches for Newly Released Programming AssignmentsIEEE Transactions on Software Engineering10.1109/TSE.2022.315352249:1(450-467)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3153522
Strijbol NVan Petegem CMaertens RSels BScholliers CDawyndt PMesuere B(2023)TESTed—An educational testing framework with language-agnostic test suites for programming exercisesSoftwareX10.1016/j.softx.2023.10140422(101404)Online publication date: May-2023
https://doi.org/10.1016/j.softx.2023.101404
Okonkwo CAde-Ibijola A(2023)Synthesis of nested loop exercises for practice in introductory programmingEgyptian Informatics Journal10.1016/j.eij.2023.03.00124:2(191-203)Online publication date: Jul-2023
https://doi.org/10.1016/j.eij.2023.03.001
Cavalcanti AMello RGašević DFreitas F(2023)Towards Explainable Prediction Feedback Messages Using BERTInternational Journal of Artificial Intelligence in Education10.1007/s40593-023-00375-wOnline publication date: 9-Nov-2023
https://doi.org/10.1007/s40593-023-00375-w
Marmo R(2022)Artificial Intelligence in E-Learning SystemsEncyclopedia of Data Science and Machine Learning10.4018/978-1-7998-9220-5.ch091(1531-1545)Online publication date: 14-Oct-2022
https://doi.org/10.4018/978-1-7998-9220-5.ch091
Ahmed UFan ZYi JAl-Bataineh ORoychoudhury A(2022)Verifix: Verified Repair of Programming AssignmentsACM Transactions on Software Engineering and Methodology10.1145/351041831:4(1-31)Online publication date: 12-Jul-2022
https://dl.acm.org/doi/10.1145/3510418
Agarwal NKarkare AKizilcec RDavis KOchoa X(2022)LEGenT: Localizing Errors and Generating Testcases for CS1Proceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528282(102-112)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528282
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents