Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3351287.3351289acmotherconferencesArticle/Chapter ViewAbstractPublication PagesukicerConference Proceedingsconference-collections
research-article

Designing Intentional Bugs for Learning

Published: 05 September 2019 Publication History

Abstract

There is general agreement that people can learn from their own mistakes, but the idea of intentionally incorporating errors into curricula is controversial. Some think it will introduce misconceptions, while others think it could help students learn concepts, avoid common errors and misconceptions, and reduce anxiety about errors. This paper develops a theoretical framework that supports this technique, and reviews the experiences and experiments of over a dozen research teams that have employed it for mathematics and computer science education. It also reports on a 14-week study that investigates this issue, where undergraduates learning to program with Python solved web-based practice problems during weekly labs. Approximately 80% of the problems were the same for all students. For the remaining 20%, the treatment group got problems with carefully designed bugs in supplied code. The control group got similar problems without bugs. According to two pre/post measures, the treatment group learned as much as the control group, with no statistical difference in learning gains. Instructional designs, design principles, and future research plans are discussed.

References

[1]
J. L. Booth, Karin E. Lange, Kenneth R. Koedinger, and Kristie J. Newton. 2013. Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction 25 (2013), 24--34.
[2]
R. Borasi. 1994. Capitalising on errors as "springboards for inquiry": A teaching experiment. Journal for Research in Mathematics Education 25, 2 (1994), 166--208.
[3]
Karen Brennan, Christan Balch, and Michelle Chung. 2011. Creative Computing. (2011). http://scratched.gse.harvard.edu/guide/files/CreativeComputing20141015.pdf
[4]
Ann L Brown. 1992. Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences 2, 2 (1992), 141--178.
[5]
Renee Bryce. 2011. Bug Wars: A Competitive Exercise to Find Bugs in Code. Journal of Computing Sciences in Colleges 27, 2 (2011), 43--50.
[6]
Xingliang Chen, Antonija Mitrovic, and Moffat Mathews. 2016. Do Erroneous Examples Improve Learning in Addition to Problem Solving and Worked Examples?. In Proceedings of International Conference on Intelligent Tutoring Systems (ITS '16). Springer, Cham, 13--22.
[7]
Xingliang Chen, Tanja Mitrovic, and Moffat Mathews. 2016. Do Novices and Advanced Students Benefit from Erroneous Examples Differently? Proceedings of 24th International Conference on Computers in Education (2016).
[8]
Michelene T. H. Chi and Miriam Bassok. 1989. Learning from Examples via Self-Explanations. In Learning and Research Development Center Technical Report No. 11. University of Pittsburgh, 1--37.
[9]
Ruth Clark, Frank Nguyen, and John Sweller. 2006. Efficiency in Learning, Evidence-Based Guidelines to Manage Cognitive Load. Wiley, San Francisco, CA.
[10]
Allan Collins. 1990. Towards a Design Science of Education, Technical Report No. 1. Center for Technology in Education. Center for Technology in Education, Center for Technology in Education, New York NY, 1--9.
[11]
Jere Confrey. 1990. What constructivism implies for teaching. In Journal for Research in Mathematics Education. Monograph Vol 4. NCTM, 107--122.
[12]
Kathryn Cunningham, Cherry Street, Sarah Blanchard, Barbara Ericson, and Mark Guzdial. 2017. Using Tracing and Sketching to Solve Programming Problems. Proceedings of the thirteenth International Workshop on Computing Education Research Workshop (ICER '17) (2017), 164--172.
[13]
Quintin Cutts. 2012. Quintin Cutts - Teaching Programming: Too much doing, not enough understanding. https://www.youtube.com/watch?v=Pim4aYfiZiY
[14]
Lionel E Deimel and J Fernando Naveda. 1990. Reading Computer Programs: Instructor's Guide and Exercises. Technical Report August.
[15]
Brian Dorn and M. Guzdial. 2006. Graphic designers who program as informal computer science learners. In Proceedings of the second International Workshop on Computing Education Research (ICER '06). ACM, New York, NY, 127--134.
[16]
Allen B. Downey. 2016. Think Python, 2nd Edition. O'Reilly Media Inc., Sebastopol CA.
[17]
B. du Boulay. 1986. Some Difficulties of Learning to Program. Journal of Educational Computing Research 2, 1 (1986), 57--73.
[18]
B. du Boulay, T. O'Shea, and J. Monk. 1981. The black box inside the glass box: presenting computing concepts to novices. International Journal of Man-Machine Studies 14 (1981), 237--249.
[19]
Kelley Durkin and Bethany Rittle-Johnson. 2012. The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction 22, 3 (jun 2012), 206--214.
[20]
Barbara J. Ericson, Steven Moore, Briana Morrison, and Mark Guzdial. 2015. Usability and Usage of Interactive Features in an Online Ebook for CS Teachers. Proceedings of the tenth Workshop in Primary and Secondary Computing Education (WiPSCE '15) (2015), 111--120.
[21]
L. Festinger. 1957. A theory of cognitive dissonance. Stanford University Press, Stanford, CA.
[22]
L. Festinger. 1962. A theory of cognitive dissonance (Vol. 2). Stanford University Press, Stanford CA.
[23]
Sally Fincher and Marian Petre (Eds.). 2004. Computer Science Education Research. Taylor & Francis, London; NY.
[24]
Sue Fitzgerald, Gary Lewandowski, Renée McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers. In Computer Science Education, Vol. 18. 93--116.
[25]
Michael Frese and Alexandra Altmann. 1989. The Treatment of Errors in Learning and Training. In Developing Skills With Information Technology. John Wiley, 65--86.
[26]
David Ginat. 2008. Learning from Wrong and Creative Algorithm Design. ACM SIGCSE Bulletin 40, 1 (2008), 26--30.
[27]
David Ginat and Ronit Shmallo. 2013. Constructive Use of Errors in Teaching CS1. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE 2013). ACM, 353--358.
[28]
A. C. Graesser. 2009. Inaugural Editorial for Journal of Educational Psychology. Journal of Educational Psychology 101, 2 (2009), 259--261.
[29]
Jean M. Griffin. 2019. Constructionism and De-Constructionism: Opposite yet Complementary Pedagogies. Constructivist Foundations 14, 3 (2019), 234--243. https://constructivist.info/14/3/234.griffin.pdf
[30]
Jean M. Griffin, Eliot Kaplan, and Q. Burke. 2012. Debug'ems and Other Deconstruction Kits for STEM Learning. In Proceedings of the Integrated STEM Education Conference (ISEC '12), IEEE 2nd. IEEE, 1--4.
[31]
Cornelia S Große and Alexander Renkl. 2004. Learning From Worked Examples: What Happens If Errors Are Included? Instructional Design for Effective and Enjoyable Computer-Supported Learning (2004), 356--364.
[32]
Cornelia S. Große and Alexander Renkl. 2007. Finding and fixing errors in worked examples: Can this foster learning outcomes? Learning and Instruction 17, 6 (dec 2007), 612--634.
[33]
Idit Harel. 1988. Software Design for Learning: Children's Construction of Meaning for Fractions and LOGO Programming (Doctoral Dissertation). Ph.D. Dissertation. MIT.
[34]
Brian Harrington and Nick Cheng. 2018. Tracing vs. Writing Code: Beyond the Learning Hierarchy. In Proceedings of the 49th Technical Symposium on Computer Science Education (SIGCSE '18). ACM, 423--428.
[35]
Tim Heemsoth and Aiso Heinze. 2013. The impact of incorrect examples on learning fractions: A field experiment with 6th grade students. Instructional Science 42, 4 (dec 2013), 639--657.
[36]
Daniel Hoffman, Ming Lu, and Tim Pelton. 2011. A web-based generation and delivery system for active code reading. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). 483--488.
[37]
P. Hubwieser, J. Magenheim, A. Muhling, and A. Ruf. 2013. Towards a conceptualization of pedagogical content knowledge for computer science. In Proceedings of the ninth International Conference on Computing Education Research (ICER '13). ACM, New York NY, USA, 1--8.
[38]
Patrick Hulin, Andy Davis, Rahul Sridhar, Andrew Fasano, Cody Gallagher, Tim Leek, and Brendan Dolan-Gavitt. 2017. AutoCTF: Creating Diverse Pwnables via Automated Bug Injection. In 11th {USENIX} Workshop on Offensive Technologies - WOOT '17.
[39]
Walter Kaess and David Zeaman. 1960. POSITIVE AND NEGATIVE KNOWLEDGE OF RESULTS ON A PRESSEY-TYPE PUNCHBOARD. Journal of Educational Psychology 60, 1 (1960), 12--17.
[40]
Irvin Katz and John R. Anderson. 1987. Debugging: An Analysis of Bug-Location Strategies. Human-Computer Interaction 3, 4 (1987), 351--399.
[41]
Päivi Kinnunen, Robert McCartney, Laurie Murphy, and Lynda Thomas. 2007. Through the eyes of instructors: a phenomenographic investigation of student success Categories and Subject Descriptors. In Proceeding of the third International Workshop on Computing Education Research - ICER '07. ACM, 61--73.
[42]
David Klahr and Sharon McCoy Carver. 1988. Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. In Cognitive Psychology, Vol. 20. 362--404.
[43]
Amruth N. Kumar. 2015. Solving Code-Tracing Problems and its Effect on CodeWriting Skills Pertaining to Program Semantics. In ITiCSE '15. 314--319.
[44]
Amruth N. Kumar and Neeraj Singhal. 2000. Using Java to Help Students Practice Problem-Solving. Proceedings of The Second International Conference and Exhibition on The Practical Application of Java (PA JAVA '00) (2000), 205--216.
[45]
D. M. Kurland, Roy D. Pea, C. Clement, and R. Mawby. 1986. A Study of the Development of Programming Ability and Thinking Skills in High School Students. Journal of Educational Computing Research 2, 4 (1986), 429--458.
[46]
Michael J Lee. 2015. Teaching and Engaging with Debugging Puzzles (Doctoral Dissertation). Ph.D. Dissertation. University of Washington.
[47]
Michael J. Lee, Faezeh Bahmani, Irwin Kwan, Jilian Laferte, Polina Charters, Amber Horvath, Fanny Luor, Jill Cao, Catherine Law, Michael Beswetherick, Sheridan Long, Margaret Burnett, and Andrew J. Ko. 2014. Principles of a Debugging-First Puzzle Game for Computing Education. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC. IEEE, 57--64.
[48]
Michael J Lee and Andrew J Ko. 2011. Personifying Programming Tool Feedback Improves Novice ProgrammersÊij Learning. In Proceedings of the seventh International Workshop on Computing Education Research (ICER '11). ACM, 109--116.
[49]
Alex Lishinski, Aman Yadav, and Richard Enbody. 2017. Students' Emotional Reactions to Programming Projects in Introduction to Programming: Measurement Approach and Influence on Learning Outcomes. In Proceedings of the thirteenth International Workshop on Computing Education Research Workshop (ICER '17). ACM, 30--38.
[50]
Raymond Lister, Elizabeth S. Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, Beth Simon, and Lynda Thomas. 2004. A Multi-National Study of Reading and Tracing Skills in Novice Programmers. ACM SIGCSE Bulletin 36, 4 (2004), 119.
[51]
Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further Evidence of a Relationship between Explaining, Tracing and Writing Skills in Introductory Programming. ACM SIGCSE Bulletin 41 (2009), 161.
[52]
M. Lopez, J. Whalley, P. Robbins, and R. Lister. 2008. Relationships Between Reading, Tracing and Writing Skills in Introductory Programming. In Proceedings of the 4th International Workshop on Computing Education Research (ICER '08). ACM, 101--112.
[53]
Linxiao Ma, John Ferguson, Marc Roper, Isla Ross, and Murray Wood. 2008. Using Cognitive Conflict and Visualisation to Improve Mental Models Held by Novice Programmers. In Proceedings of the 39th Technical Symposium on Computer Science Education - SIGCSE '08. ACM, NY, 342--346.
[54]
L. M. Mann, M. C. Linn, and M. Clancy. 1994. Can Tracing Tools Contribute to Programming Proficiency? The LISP Evaluation Modeler. Interactive Learning Environments 4, 1 (1994), 96--113.
[55]
Renée McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: a review of the literature from an educational perspective. Computer Science Education 18, April 2015 (2008), 67--92.
[56]
Bruce M. McLaren, Deanne M. Adams, and Richard E. Mayer. 2015. Delayed Learning Effects with Erroneous Examples: A Study of Learning Decimals with a Web-Based Tutor. International Journal of Artificial Intelligence in Education 25, 4 (2015), 520--542.
[57]
Erica Melis. 2005. Design of Erroneous Examples for ActiveMath. Artificial Intelligence in Education (2005), 451--458.
[58]
Erica Melis, Eric Andres, Jochen Budenbender, Adrian Frischauf, Paul Libbrecht, Martin Pollet, and Carsten Ullrich. 2001. ActiveMath: A Generic and Adaptive Web-Based Learning Environment. International Journal of Artificial Intelligence in Education (IJAIED) 12 (2001), pp.385--407.
[59]
Brad Miller and D. Ranum. 2014. Runestone Interactive: Tools for Creating Interactive Course Materials. In Proceedings of the First ACM Conference on Learning @ Scale (L@S '14). ACM, 213--214.
[60]
Marvin Minsky. 1994. Negative Expertise. International Journal of Expert Systems 7, 1 (1994), 13--19.
[61]
Andrés Moreno and Mike Joy. 2014. Defining and Evaluating Conflictive Animations for Programming Education: The Case of Jeliot ConAn. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education - SIGCSE '14. ACM, 629--634.
[62]
L A O'Hara and R J Sternberg. 1999. Creativity and Intelligence. In Handbook of Creativity, R J Sternberg (Ed.). Cambridge University Press.
[63]
Stellan Ohlsson. 1996. Learning from error and the design of task environments. International Journal of Educational Research 25, 5 (1996), 419--448.
[64]
F. Oser, C. Näpflin, C. Hofer, and P. Aerni. 2012. Towards a theory of Negative Knowledge (NK). Almost-mistakes as drivers of episodic memory amplification. Professional and Practice-based Learning 6 (2012), 53--70.
[65]
F. Oser and M. Spychinger. 2005. Lernen ist schmerzhaft: Zur Theorie des Negativen Wissens und zur Praxis der Fehlerkultur. Beltz, Weinheim.
[66]
Seymour Papert. 1980. Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, Inc.
[67]
Roy D. Pea, E. Soloway, and J. G. Spohrer. 1987. The buggy path to the development of programming expertise. Focus on Learning Problems in Mathematics 9, 1 (1987), 5--30.
[68]
D. Perkins and F. Martin. 1989. Fragile Knowledge and Neglected Strategies in Novice Programmers. In Studying the Novice Programmer, E. Soloway and J Spohrer (Eds.). Lawrence Erlbaum Associates, Hillsdale, NJ, 213--229.
[69]
Jannik Pewny and Thorstein Holz. 2016. EvilCoder: Automated Bug Insertion. Proceedings of the 32nd Annual Conference on Computer Security Applications (2016), 214--225.
[70]
Jean Piaget. 1980. Experiments in contradictions. University of Chicago Press, Chicago.
[71]
G Polya. 1957. How to Solve It. Doubleday, Garden City, NJ.
[72]
Brad Richards. 2000. Bugs as Features: Teaching Network Protocols Through Debugging. Proceedings of the thirty-first SIGCSE Technical Symposium on Computer Science Education (SIGCSE '00) 32, 1 (2000), 256--259.
[73]
Anthony Robins. 2010. Learning edge momentum: a new account of outcomes in CS1. Computer Science Education 20, 1 (2010), 37--71.
[74]
Robert S. Siegler. 2002. Microgenetic Studies On Self-Explanation. In Microdevelopment: Transition Processes in Development and Learning, Nira Granott and Jim Parziale (Eds.). Cambridge University Press, 31--58.
[75]
E. Soloway. 1986. Learning to program = learning to construct mechanisms and explanations., 850--858 pages.
[76]
E. Soloway and J. C. Spohrer (Eds.). 1989. Studying the novice programmer. L. Erlbaum Associates, Hillsdale, NJ.
[77]
Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A Review of Generic Program Visualization Systems for Introductory Programming Education. ACM Transactions on Computing Education (TOCE '13) 13, 4 (2013), 1--68.
[78]
J. Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12, 2 (1988), 257--285.
[79]
J. Sweller and Graham A. Cooper. 1985. The Use of Worked Examples as a Substitute for Problem Solving in Learning Algebra. Cognition and Instruction 2, 1 (1985), 59--89.
[80]
J Gregory Trafton and Brian J Reiser. 1993. The Contributions of Studying Examples and Solving Problems to Skill Acquisition. In Proceedings of the 15th Annual Conference of the Cognitive Science Society, M. Polson (Ed.). Lawrence Erlbaum, Hillsdale, NJ:, 1017--1022.
[81]
Dimitra Tsovaltzi, Erica Melis, Bruce M McLaren, Ann-Kristin Meyer, Michael Dietrich, and George Goguadze. 2010. Learning from Erroneous Examples: When and How do Students Benefit from them?. In Proceedings of the European Conference on Technology Enhanced Learning, LNCS (vol. 6383). Springer, Heidelberg.
[82]
Jeroen Van Merriënboer. 1990. Strategies for Programming Instruction in High School: Program Completion vs. Program Generation. Journal of Educational Computing Research 6, 3 (1990), 265--285.
[83]
Kurt VanLehn. 1999. Rule-learning events in the acquisition of a complex skill: An evaluation of CASCADE. Journal of the Learning Sciences 8, 1 (1999), 71--125.
[84]
Anne Venables, Grace Tan, and Raymond Lister. 2009. A Closer Look at Tracing, Explaining and Code Writing Skills in the Novice Programmer. Proceedings of the fifth International Workshop on Computing Education Research Workshop (ICER '09) (2009), 117--128.
[85]
J. L. Whalley, R. Lister, E. Thompson, T. Clear, P. Robbins, P. K. Kumar, and C. Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. Australian Computer Society, Inc., 243--252.
[86]
R. R. Wilcox. 2016. Understanding and Applying Basic Statistical Methods Using R. John Wiley & Sons, Hoboken NJ.
[87]
Benjamin Xie, Greg L Nelson, and Andrew J Ko. 2018. An Explicit Strategy to Scaffold Novice Program Tracing. 2 (2018).
[88]
Peng Zhang, Jules White, and C Schmidt. 2016. HoliCoW: Automatically Breaking Team-based Software Projects to Motivate Student Testing. In IEEE/ACM 38th International Conference on Software Engineering Companion. IEEE, 436--439.

Cited By

View all
  • (2022)A Review of Worked Examples in Programming ActivitiesACM Transactions on Computing Education10.1145/356026623:1(1-35)Online publication date: 29-Dec-2022
  • (2021)Can Students Learn from Grading Erroneous Computer Programs?2021 International Conference on Advanced Learning Technologies (ICALT)10.1109/ICALT52272.2021.00070(211-215)Online publication date: Jul-2021

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
UKICER '19: Proceedings of the 2019 Conference on United Kingdom & Ireland Computing Education Research
September 2019
81 pages
ISBN:9781450372572
DOI:10.1145/3351287
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • Univ of Kent at Canterbury: University of Kent at Canterbury

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 September 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. CS1
  2. bugs
  3. de-construction
  4. debugging
  5. errors
  6. intentional bugs
  7. novice programmer

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

UKICER
UKICER: UK & Ireland Computing Education Research Conference
September 5 - 6, 2019
Canterbury, United Kingdom

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)7
Reflects downloads up to 10 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2022)A Review of Worked Examples in Programming ActivitiesACM Transactions on Computing Education10.1145/356026623:1(1-35)Online publication date: 29-Dec-2022
  • (2021)Can Students Learn from Grading Erroneous Computer Programs?2021 International Conference on Advanced Learning Technologies (ICALT)10.1109/ICALT52272.2021.00070(211-215)Online publication date: Jul-2021

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media