research-article

Towards answering “Am I on the right track?” automatically using program synthesis

Authors:

Molly Q Feldman,

William E. Byrd,

François Guimbretière,

Erik AndersenAuthors Info & Claims

SPLASH-E 2019: Proceedings of the 2019 ACM SIGPLAN Symposium on SPLASH-E

Pages 13 - 24

https://doi.org/10.1145/3358711.3361626

Published: 25 October 2019 Publication History

Abstract

Students learning to program often need help completing assignments and understanding why their code does not work as they expect it to. One common place where they seek such help is at teaching assistant office hours. We found that teaching assistants in introductory programming (CS1) courses frequently answer some variant of the question ``Am I on the Right Track?''. The goal of this work is to develop an automated tool that provides similar feedback for students in real-time from within an IDE as they are writing their program. Existing automated tools lack the generality that we seek, often assuming a single approach to a problem, using hand-coded error models, or applying sample fixes from other students. In this paper, we explore the use of program synthesis to provide less constrained automated answers to ``Am I on the Right Track'' (AIORT) questions. We describe an observational study of TA-student interactions that supports targeting AIORT questions, as well as the development of and design considerations behind a prototype integrated development environment (IDE). The IDE uses an existing program synthesis engine to determine if a student is on the right track and we present pilot user studies of its use.

References

[1]

Harold Abelson, R. Kent Dybvig, Christopher T. Haynes, Guillermo Juan Rozas, NI Adams, Daniel P. Friedman, E Kohlbecker, GL Steele, David H Bartley, R Halstead, et al. 1998. Revised 5 report on the algorithmic language Scheme. Higher-Order and Symbolic Computation (1998).

[2]

John R Anderson, C Franklin Boyle, and Brian J Reiser. 1985. Intelligent tutoring systems. Science 228, 4698 (1985), 456–462.

[3]

John R Anderson, Frederick G Conrad, and Albert T Corbett. 1989. Skill acquisition and the LISP tutor. Cognitive Science 13, 4 (1989), 467–505.

[4]

Kristy Elizabeth Boyer, William Lahti, Robert Phillips, Michael D Wallis, Mladen A Vouk, and James C Lester. 2010. Principles of asking effective questions during student problem solving. In Proceedings of the 41st SIGCSE Technical Symposium. ACM, 460–464.

Digital Library

[5]

John Seely Brown and Richard R Burton. 1978. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive science 2, 2 (1978), 155–192.

[6]

William E. Byrd, Michael Ballantyne, Gregory Rosenblatt, and Matthew Might. 2017. A Unified Approach to Solving Seven Programming Problems (Functional Pearl). Proc. ACM Program. Lang. 1, ICFP (Aug. 2017).

Digital Library

[7]

Sarah E Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scraping Distributed Hierarchical Web Data. In The 31st Annual ACM Symposium on User Interface Software and Technology. ACM, 963–975.

Digital Library

[8]

Herbert H Clark, Susan E Brennan, et al. 1991. Grounding in communication. Perspectives on socially shared cognition 13, 1991 (1991), 127–149.

[9]

Luca de Alfaro and Michael Shavlovsky. 2014. CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. In Proceedings of the 45th ACM technical symposium on Computer science education. ACM, 415–420.

Digital Library

[10]

Molly Q Feldman, Ji Yong Cho, Monica Ong, Sumit Gulwani, Zoran Popović, and Erik Andersen. 2018. Automatic Diagnosis of Students’ Misconceptions in K-8 Mathematics. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM.

Digital Library

[11]

Matthias Felleisen, Robert Bruce Finder, Matthew Flatt, and Shriram Krishnamurthi. 2001. How to design programs: an introduction to programming and computing. MIT Press.

[12]

Matthias Felleisen, Robert Bruce Finder, Matthew Flatt, and Shriram Krishnamurthi. 2014. How to Design Programs, Second Edition. MIT Press.

[13]

Robert Bruce Findler. 2014. DrRacket: The Racket Programming Environment. (2014).

[14]

Alex Gerdes, Bastiaan Heeren, Johan Jeuring, and L Thomas van Binsbergen. 2017. Ask-Elle: an adaptable programming tutor for Haskell giving automated feedback. International Journal of Artificial Intelligence in Education 27, 1 (2017), 65–100.

[15]

Michael Glass, Jung Hee Kim, Martha W Evens, Joel A Michael, and Allen A Rovick. 1999. Novice vs. expert tutors: A comparison of style. In MAICS-99, Proceedings of the Tenth Midwest AI and Cognitive Science Conference. 43–49.

[16]

Elena L Glassman, Lyla Fischer, Jeremy Scott, and Robert C Miller. 2015. Foobaz: Variable name feedback for student code at scale. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 609–617.

Digital Library

[17]

Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016. Learnersourcing personalized hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 1626–1636.

Digital Library

[18]

Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip Guo, and Robert C. Miller. 2015. OverCode: visualizing variation in student solutions to programming problems at scale. Transactions on ComputerHuman Interaction (2015).

[19]

Sumit Gulwani, Alex Polozov, and Rishabh Singh. 2017. Program Synthesis. Vol. 4. NOW. 1–119 pages. https://www.microsoft.com/enus/research/publication/program- synthesis/

[20]

Philip J Guo. 2013. Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education. ACM, 579–584.

[21]

Philip J Guo. 2015. Codeopticon: Real-time, one-to-many human tutoring for computer programming. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 599– 608.

Digital Library

[22]

Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R Klemmer. 2010. What would other programmers do: suggesting solutions to error messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1019–1028.

Digital Library

[23]

Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017. Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. ACM, 89–98.

Digital Library

[24]

Jack Hollingsworth. 1960. Automatic graders for programming classes. Commun. ACM 3, 10 (1960), 528–529.

Digital Library

[25]

John F Hughes. 2018. CS: An Integrated Introduction. http://cs.brown. edu/courses/csci0170/

[26]

Hyeonsu Kang and Philip J. Guo. 2017. Omnicode: A Novice-Oriented Live Programming Environment with Always-On Run-Time Value Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST ’17). ACM, 737–745.

Digital Library

[27]

Michael Kölling and John Rosenberg. 1996. An object-oriented program development environment for the first programming course. In ACM SIGCSE Bulletin, Vol. 28. ACM, 83–87.

Digital Library

[28]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI) 20, 6 (2013), 33.

Digital Library

[29]

Tia Newhall, Lisa Meeden, Andrew Danner, Ameet Soni, Frances Ruiz, and Richard Wicentowski. 2014. A support program for introductory CS courses that improves student performance and retains students from underrepresented groups. In Proceedings of the 45th ACM SIGCSE. ACM, 433–438.

Digital Library

[30]

Nadia Polikarpova, Ivan Kuraj, and Armando Solar-Lezama. 2016. Program synthesis from polymorphic refinement types. ACM SIGPLAN Notices 51, 6 (2016), 522–538.

Digital Library

[31]

Joe Gibbs Politz, Joseph M Collard, Arjun Guha, Kathi Fisler, and Shriram Krishnamurthi. 2016. The Sweep: Essential Examples for InFlow Peer Review. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM, 243–248.

Digital Library

[32]

Oleksandr Polozov and Sumit Gulwani. 2015. FlashMeta: a framework for inductive program synthesis. In ACM SIGPLAN Notices, Vol. 50. ACM, 107–126.

Digital Library

[33]

Kelly Rivers and Kenneth R Koedinger. 2014. Automating hint generation with solution space path construction. In International Conference on Intelligent Tutoring Systems. Springer, 329–339.

Digital Library

[34]

Chung-chieh Shan and Robert Rose. 2019. C211/H211: Introduction to Computer Science. https://www.cs.indiana.edu/classes/c211/

[35]

Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013. Automated feedback generation for introductory programming assignments. ACM SIGPLAN Notices 48, 6 (2013), 15–26.

Digital Library

[36]

Armando Solar-Lezama, Gilad Arnold, Liviu Tancau, Rastislav Bodik, Vijay Saraswat, and Sanjit Seshia. 2007. Sketching stencils. ACM SIGPLAN Notices 42, 6 (2007), 167–178.

Digital Library

[37]

Armando Solar-Lezama, Rodric Rabbah, Rastislav Bodík, and Kemal Ebcioğlu. 2005. Programming by sketching for bit-streaming programs. In ACM SIGPLAN Notices, Vol. 40. ACM, 281–294.

Digital Library

[38]

Ryo Suzuki, Gustavo Soares, Andrew Head, Elena Glassman, Ruan Reis, Melina Mongiovi, and Björn Antoni, Loris D’and Hartman. 2017. TraceDiff: Debugging unexpected code behavior using trace divergences. In Visual Languages and Human-Centric Computing (VL/HCC), 2017 IEEE Symposium on. IEEE.

[39]

Emina Torlak and Rastislav Bodik. 2013. Growing solver-aided languages with rosette. In Proceedings of the 2013 ACM international symposium on New ideas, new paradigms, and reflections on programming & software. ACM, 135–152.

Digital Library

[40]

Arto Vihavainen, Thomas Vikberg, Matti Luukkainen, and Jaakko Kurhila. 2013. Massive increase in eager TAs: Experiences from extreme apprenticeship-based CS1. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. ACM, 123–128.

Digital Library

Cited By

Van Praet LHoobergs JSchrijvers THermans FBohrer R(2024)ASSIST: Automated Feedback Generation for Syntax and Logical Errors in Programming ExercisesProceedings of the 2024 ACM SIGPLAN International Symposium on SPLASH-E10.1145/3689493.3689981(66-76)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1145/3689493.3689981
Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Rocha HCosta ETedesco P(2023)Helping to provide adaptive feedback to novice programmers: a framework to assist the Teachers2023 18th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI58278.2023.10212000(1-6)Online publication date: 20-Jun-2023
https://doi.org/10.23919/CISTI58278.2023.10212000
Show More Cited By

Index Terms

Towards answering “Am I on the right track?” automatically using program synthesis
1. Applied computing
  1. Education
2. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Automatic programming

Recommendations

Assessing the Development of Computer Science Pedagogical Content Knowledge in the TEALS Program (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

One of the critical barriers to increasing pre-collegiate computer science course offerings in the U.S. is a lack of qualified computer science teachers. Programs such as TEALS, a teacher preparation program pairing high school teachers with computing ...
Algorithmic program synthesis: introduction

Program synthesis is a process of producing an executable program from a specification. Algorithmic synthesis produces the program automatically, without an intervention from an expert. While classical compilation falls under the definition of ...
Should Your College Computer Science Program Partner with a Coding Boot Camp? (Abstract Only)
SIGCSE '17: Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education

The rise of so-called "coding boot camps" as an alternative training ground for software development is prominent in the popular press, and these camps have caught the attention of colleges and universities. Administrators and faculty considering ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SPLASH-E 2019: Proceedings of the 2019 ACM SIGPLAN Symposium on SPLASH-E

October 2019

65 pages

ISBN:9781450369893

DOI:10.1145/3358711

Program Chair:
Elisa Baniassad
University of British Columbia, Canada

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Institutes of Health

Conference

SPLASH '19

Sponsor:

SIGPLAN

SPLASH '19: 2019 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity

October 25, 2019

Athens, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
241
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Van Praet LHoobergs JSchrijvers THermans FBohrer R(2024)ASSIST: Automated Feedback Generation for Syntax and Logical Errors in Programming ExercisesProceedings of the 2024 ACM SIGPLAN International Symposium on SPLASH-E10.1145/3689493.3689981(66-76)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1145/3689493.3689981
Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Rocha HCosta ETedesco P(2023)Helping to provide adaptive feedback to novice programmers: a framework to assist the Teachers2023 18th Iberian Conference on Information Systems and Technologies (CISTI)10.23919/CISTI58278.2023.10212000(1-6)Online publication date: 20-Jun-2023
https://doi.org/10.23919/CISTI58278.2023.10212000
Joyse Barbosa Rocha Hde Barros Costa ECabral de Azevedo Restelli Tedesco P(2023)A Taxonomy to Assist TAs in Providing Adaptive Feedback to Novice Programmers2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10343309(1-9)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10343309
Van Petegem CDeconinck LMourisse DMaertens RStrijbol NDhoedt BDe Wever BDawyndt PMesuere B(2022)Pass/Fail Prediction in Programming CoursesJournal of Educational Computing Research10.1177/07356331221085595(073563312210855)Online publication date: 6-Jun-2022
https://doi.org/10.1177/07356331221085595
Nose JCong YMasuhara HHenz MLerner B(2022)Mio: A Block-Based Environment for Program DesignProceedings of the 2022 ACM SIGPLAN International Symposium on SPLASH-E10.1145/3563767.3568127(62-69)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563767.3568127
Lau SGuo PHenz MLerner B(2022)Codehound: Helping Instructors Track Pedagogical Code Dependencies in Course MaterialsProceedings of the 2022 ACM SIGPLAN International Symposium on SPLASH-E10.1145/3563767.3568126(1-6)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3563767.3568126
D’Antoni LHu QKim JReps T(2021)Programmable Program SynthesisComputer Aided Verification10.1007/978-3-030-81685-8_4(84-109)Online publication date: 20-Jul-2021
https://dl.acm.org/doi/10.1007/978-3-030-81685-8_4
Peleg HGabay RItzhaky SYahav E(2020)Programming with a read-eval-synth loopProceedings of the ACM on Programming Languages10.1145/34282274:OOPSLA(1-30)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428227
Reichenbach CBaniassad ECurtsinger C(2020)The PL-detective revisitedProceedings of the 2020 ACM SIGPLAN Symposium on SPLASH-E10.1145/3426431.3428655(12-22)Online publication date: 20-Nov-2020
https://dl.acm.org/doi/10.1145/3426431.3428655

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten