research-article

Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs

Authors:

David Weintrop,

Uri WilenskyAuthors Info & Claims

ICER '15: Proceedings of the eleventh annual International Conference on International Computing Education Research

Pages 101 - 110

https://doi.org/10.1145/2787622.2787721

Published: 09 August 2015 Publication History

Abstract

Blocks-based programming environments are becoming increasingly common in introductory programming courses, but to date, little comparative work has been done to understand if and how this approach affects students' emerging understanding of fundamental programming concepts. In an effort to understand how tools like Scratch and Blockly differ from more conventional text-based introductory programming languages with respect to conceptual understanding, we developed a set of "commutative" assessments. Each multiple-choice question on the assessment includes a short program that can be displayed in either a blocks- based or text-based form. The set of potential answers for each question includes the correct answer along with choices informed by prior research on novice programming misconceptions. In this paper we introduce the Commutative Assessment, discuss the theoretical and practical motivations for the assessment, and present findings from a study that used the assessment. The study had 90 high school students take the assessment at three points over the course of the first ten weeks of an introduction to programming course, alternating the modality (blocks vs. text) for each question over the course of the three administrations of the assessment. Our analysis reveals differences on performance between blocks-based and text-based questions as well as differences in the frequency of misconceptions based on the modality. Future work, potential implications, and limitations of these findings are also discussed.

References

[1]

Abelson, H. and DiSessa, A. A. 1986. Turtle geometry: The computer as a medium for exploring mathematics. MIT Press.

[2]

ACM/IEEE-CS Joint Task Force on Computing Curricula 2013. Computer Science Curricula 2013. ACM Press and IEEE Computer Society Press.

Digital Library

[3]

Armoni, M. and Ben-Ari, M. 2010. Computer Science Concepts in Scratch.

[4]

Armoni, M., Meerbaum-Salant, O. and Ben-Ari, M. 2015. From Scratch to "Real" Programming. ACM Transactions on Computing Education. 14, 4 (2015), 25.

Digital Library

[5]

Bayman, P. and Mayer, R. E. 1983. A diagnosis of beginning programmers' misconceptions of BASIC programming statements. Comm. of the ACM. 26, 9 (1983), 677--679.

Digital Library

[6]

Begel, A. 1996. LogoBlocks: A graphical programming language for interacting with the world. Electrical Engineering and Computer Science Department. MIT.

[7]

Begel, A. and Klopfer, E. 2007. Starlogo TNG: An introduction to game development. Journal of E-Learning. (2007).

[8]

Blikstein, P. and Wilensky, U. 2009. An Atom is Known by the Company it Keeps: A Constructionist Learning Environment for Materials Science Using Agent-Based Modeling. Int. Journal of Computers for Mathematical Learning. 14, 2 (2009), 81--119.

[9]

Bonar, J. and Liffick, B. W. 1987. A visual programming language for novices. Principles of Visual Programming Systems. S. K. Chang, ed. Prentice-Hall, Inc.

Digital Library

[10]

Du Boulay, B. 1986. Some difficulties of learning to program. Journal of Educational Computing Research. 2, 1 (1986), 57--73.

[11]

Brusilovsky, P., Calabrese, E., Hvorecky, J., Kouchnirenko, A. and Miller, P. 1997. Mini-languages: a way to learn programming principles. Education and Information Technologies. 2, 1 (1997), 65--83.

Digital Library

[12]

Buffum, P. S., Lobene, E. V., Frankosky, M. H., Boyer, K. E., Wiebe, E. N. and Lester, J. C. 2015. A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School. (2015).

[13]

Cooper, S., Dann, W. and Pausch, R. 2000. Alice: a 3-D tool for introductory programming concepts. Journal of Computing Sciences in Colleges. 15, 5 (2000), 107--116.

Digital Library

[14]

Denny, P., Luxton-Reilly, A., Tempero, E. and Hendrickx, J. 2011. Understanding the syntax barrier for novices. Proc. of the 16th Annual ITiCSE (2011), 208--212.

Digital Library

[15]

Van Deursen, A., Klint, P. and Visser, J. 2000. Domain-specific languages: An annotated bibliography. ACM Sigplan Notices. 35, 6 (2000), 26--36.

Digital Library

[16]

Dijkstra, E. W. 1982. How do we tell truths that might hurt? Selected Writings on Computing: A Personal Perspective. Springer. 129--131.

[17]

diSessa, A. A. 2000. Changing minds: Computers, learning, and literacy. MIT Press.

Digital Library

[18]

diSessa, A. A., Hammer, D., Sherin, B. L. and Kolpakowski, T. 1991. Inventing graphing: Meta-representational expertise. Journal of Mathematical Behavior. 10, (1991), 117--160.

[19]

Donzeau-Gouge, V., Huet, G., Lang, B. and Kahn, G. 1984. Programming environments based on structured editors: The MENTOR experience. Interactive Programming Environments. McGraw Hill.

[20]

Doukakis, D., Grigoriadou, M. and Tsaganou, G. 2007. Understanding the programming variable concept with animated interactive analogies. Proc. of the 8th HERCMA Conference (2007).

[21]

Esper, S., Foster, S. R. and Griswold, W. G. 2013. CodeSpells: embodying the metaphor of wizardry for programming. Proc. of the 18th annual ITiCSE conference (2013), 249--254.

Digital Library

[22]

Feurzeig, W., Papert, S., Bloom, M., Grant, R. and Solomon, C. 1970. Programming-languages as a conceptual framework for teaching mathematics. SIGCUE Outlook. 4(1970), 13--17.

Digital Library

[23]

Fraser, N. 2013. Blockly. Google.

[24]

Garlick, R. and Cankaya, E. C. 2010. Using Alice in CS1: A quantitative experiment. Proc. of the 15th Annual ITiCSE Conference (2010), 165--168.

Digital Library

[25]

Gilmore, D. J. and Green, T. R. G. 1984. Comprehension and recall of miniature programs. Int. Journal of Man-Machine Studies. 21, 1 (1984), 31--48.

Digital Library

[26]

Gross, P. and Powers, K. 2005. Evaluating Assessments of Novice Programming Environments. Proc. of the 1st Annual ICER Conference (NY, USA, 2005), 99--110.

Digital Library

[27]

Grover, S., Cooper, S. and Pea, R. 2014. Assessing computational learning in K-12. (2014), 57--62.

Digital Library

[28]

Harvey, B. and Mönig, J. 2010. Bringing "no ceiling" to Scratch: Can one language serve kids and computer scientists? Proc. of Constructionism 2010 (Paris, Fr.), 1--10.

[29]

Koh, K. H., Basawapatna, A., Nickerson, H. and Repenning, A. 2014. Real Time Assessment of Computational Thinking. Visual Languages and Human-Centric Computing, 49--52.

[30]

Kölling, M. and Rosenberg, J. 1996. Blue - a language for teaching object-oriented programming. ACM SIGCSE Bulletin (1996), 190--194.

Digital Library

[31]

Lave, J. 1988. Cognition in practice: Mind, mathematics, and culture in everyday life. Cambridge Univ Press.

[32]

Lewis, C. M. 2010. How programming environment shapes perception, learning and goals: Logo vs. Scratch. Proc. of the 41st Annual ACM SIGCSE Conference (NY, 2010), 346--350.

Digital Library

[33]

Luria, A. R. 1982. Language and cognition. Winston; Wiley, Washington, D.C. : New York; Chichester:

[34]

Meerbaum-Salant, O., Armoni, M. and Ben-Ari, M. 2011. Habits of programming in Scratch. Proc. of the 16th Annual ITiCSE Conference (Darmstadt, Germany, 2011), 168--172.

Digital Library

[35]

Meerbaum-Salant, O., Armoni, M. and Ben-Ari, M. M. 2010. Learning computer science concepts with scratch. Proc. of the 6th Annual ICER Conference (2010), 69--76.

Digital Library

[36]

Mendelsohn, P., Green, T. R. G. and Brna, P. 1990. Programming languages in education: The search for an easy start. Academic Press London.

[37]

Motil, J. and Epstein, D. 1998. JJ: a Language Designed for Beginners.

[38]

Nemirovsky, R. 1994. On ways of symbolizing: The case of Laura and the velocity sign. The Journal of Mathematical Behavior. 13, 4 (1994), 389--422.

[39]

Noss, R. and Hoyles, C. 1996. Windows on mathematical meanings: Learning cultures and computers. Kluwer.

[40]

Olson, I. C. and Horn, M. S. 2011. Modeling on the table: agent-based modeling in elementary school with NetTango. Proc. of the 10th Annual IDC Conference. (2011), 189--192.

Digital Library

[41]

Ong, W. 1982. Orality and Literacy: The technologizing of the world. Routledge.

[42]

Palmer, S. E. 1978. Fundamental aspects of cognitive representation. Cognition and categorization. E. Rosch and B. B. Lloyd, eds. Lawrence Erlbaum Associates. 259--303.

[43]

Parsons, D. and Haden, P. 2007. Programming osmosis: Knowledge transfer from imperative to visual programming environments. Proc.of The 12th Annual NACCQ Conference (Hamilton, New Zealand, 2007), 209--215.

[44]

Pattis, R. E. 1981. Karel the robot: a gentle introduction to the art of programming. John Wiley & Sons, Inc.

Digital Library

[45]

Pea, R. D. 1986. Language-independent conceptual" bugs" in novice programming. Journal of Educational Computing Research. 2, 1 (1986), 25--36.

[46]

Piaget, J. 1952. The origins of intelligence in children. International Universities Press, Inc.

[47]

Piech, C., Sahami, M., Koller, D., Cooper, S. and Blikstein, P. 2012. Modeling how students learn to program. Proc.of the 43rd ACM SIGCSE Conference (2012), 153--160.

Digital Library

[48]

Powers, K., Ecott, S. and Hirshfield, L. M. 2007. Through the looking glass: teaching CS0 with Alice. ACM SIGCSE Bulletin. 39, 1 (2007), 213--217.

Digital Library

[49]

Resnick, M. et al. 2009. Scratch: Programming for all. Comm. of the ACM. 52, 11 (2009), 60.

Digital Library

[50]

Sengupta, P. and Wilensky, U. 2009. Learning Electricity with NIELS: Thinking with Electrons and Thinking in Levels. Int. Journal of Computers for Mathematical Learning. 14, 1 (2009), 21--50.

[51]

Sherin, B. L. 2001. A comparison of programming languages and algebraic notation as expressive languages for physics. Int. Journal of Computers for Mathematical Learning. 6, 1 (2001), 1--61.

[52]

Sherin, B. L. 2000. How students invent representations of motion: A genetic account. The Journal of Mathematical Behavior. 19, 4 (2000), 399--441.

[53]

Slany, W. 2014. Tinkering with Pocket Code, a Scratch-like programming app for your smartphone. Proc. of Constructionism 2014 (Vienna, Austria, 2014).

[54]

Sleeman, D., Putnam, R. T., Baxter, J. and Kuspa, L. 1986. Pascal and high school students: A study of errors. Journal of Educational Computing Research. 2, 1 (1986), 5--23.

[55]

Sorva, J. 2008. The same but different students' understandings of primitive and object variables. Proc. of the 8th Annual ICER Conference (2008), 5--15.

Digital Library

[56]

Sorva, J. 2012. Visual Program Simulation in Introductory Programming Education. Aalto University.

[57]

Stefik, A. and Gellenbeck, E. 2011. Empirical studies on programming language stimuli. Software Quality Journal. 19, 1 (2011), 65--99.

Digital Library

[58]

Stefik, A. and Hanenberg, S. 2014. The Programming Language Wars: Questions and Responsibilities for the Programming Language Community. Proc. of the 2014 Int. Symposium on New Ideas, New Paradigms, and Reflections on Programming (NY, USA, 2014), 283--299.

Digital Library

[59]

Stefik, A. and Siebert, S. 2013. An Empirical Investigation into Programming Language Syntax. ACM Transactions on Computing Education. 13, 4 (2013), 1--40.

Digital Library

[60]

Sudol, L. A. and Studer, C. 2010. Analyzing Test Items: Using Item Response Theory to Validate Assessments. Proc. of the 41st ACM SIGCSE Conference (NY, 2010), 436--440.

Digital Library

[61]

Swetz, F. 1989. Capitalism and arithmetic: The new math of the 15th century. Open Court.

[62]

Tew, A. E. and Dorn, B. 2013. The Case for Validated Tools in Computer Science Education Research. Computer. 46, 9 (2013), 60--66.

Digital Library

[63]

Tew, A. E. and Guzdial, M. 2010. Developing a validated assessment of fundamental CS1 concepts. Proc. of the 41st Annual ACM SIGCSE Conference (2010), 97--101.

Digital Library

[64]

Tew, A. E. and Guzdial, M. 2011. The FCS1: a language independent assessment of CS1 knowledge. Proc. of the 42nd Annual ACM SIGCSE Conference (2011), 111--116.

Digital Library

[65]

Vygotsky, L. 1978. Mind in society: The development of higher psychological processes. Harvard University Press.

[66]

Vygotsky, L. 1986. Thought and language. MIT Press.

[67]

Wagh, A. and Wilensky, U. 2012. Evolution in blocks: Building models of evolution using blocks. Proc. of Constructionism 2012 (Athens, Gr, 2012).

[68]

Weintrop, D. and Wilensky, U. 2012. RoboBuilder: A program-to-play constructionist video game. Proc. of Constructionism 2012 (Athens, Gr, 2012).

[69]

Weintrop, D. and Wilensky, U. 2014. Situating programming abstractions in a constructionist video game. Proc. of Constructionism 2014 (Vienna, Au, 2014).

[70]

Weintrop, D. and Wilensky, U. 2015. To Block or not to Block, That is the Question: Students' Perceptions of Blocks-based Programming. Proc. of the 14th Annual IDC Conference (Boston, MA, 2015).

Digital Library

[71]

Werner, L., Denner, J., Campe, S. and Kawamoto, D. C. 2012. The fairy performance assessment: measuring computational thinking in middle school. Proc. of the 43rd Annual ACM SIGC Conference (2012), 215--220.

Digital Library

[72]

Wertsch, J. V. 1991. Voices of the mind: A sociocultural approach to mediated action. Harvard University Press.

[73]

Whorf, B. L., Carroll, J. B. and Chase, S. 1956. Language, thought, and reality: Selected writings of Benjamin Lee Whorf. MIT press Cambridge, MA.

[74]

Wilensky, U. and Papert, S. 2010. Restructurations: Reformulating knowledge disciplines through new representational forms. Proc. of Constructionism 2010 (Paris, Fr., 2010).

[75]

Wilensky, U. and Reisman, K. 2006. Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories - an embodied modeling approach. Cognition and Instruction. 24, 2 (2006), 171--209.

[76]

Wilkerson-Jerde, M. H. and Wilensky, U. 2010. Restructuring Change, Interpreting Changes: The DeltaTick Modeling and Analysis Toolkit. Proc. of Constructionism 2010 (Paris, Fr, 2010).

Cited By

Aristizábal Zapata JGutiérrez Posada JDiago P(2024)Design and Validation of a Computational Thinking Test for Children in the First Grades of Elementary EducationMultimodal Technologies and Interaction10.3390/mti80500398:5(39)Online publication date: 9-May-2024
https://doi.org/10.3390/mti8050039
Hartmann MHielscher MMarinus E(2024)Prevalence of Programming Misconceptions in Primary School StudentsProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699568(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699568
Houchins JLuttgen KOwen RMartinez Rivera LSilberglitt MKao YStephenson BStone JBattestilli LRebelsky SShoop L(2024)Building a Mixed-format Computer Science Assessment for Middle SchoolProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635578(1688-1689)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635578
Show More Cited By

Index Terms

Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
        Information science education
2. Software and its engineering
  1. Software notations and tools
    1. Context specific languages
      1. Visual languages

Recommendations

To block or not to block, that is the question: students' perceptions of blocks-based programming
IDC '15: Proceedings of the 14th International Conference on Interaction Design and Children

Blocks-based programming tools are becoming increasingly common in high-school introductory computer science classes. Such contexts are quite different than the younger audience and informal settings where these tools are more often used. This paper ...
Minding the Gap Between Blocks-Based and Text-Based Programming (Abstract Only)
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

Graphical blocks-based programming environments, such as Scratch and Snap!, are becoming increasingly popular tools for introducing learners to programming in formal educational settings. However, a growing body of research is finding that students ...
Teaching Text-based Programming in a Blocks-based World (Abstract Only)
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

This poster presents an environment and set of pedagogical strategies designed to explore how best to use blocks-based programming tools to prepare learners for future, text-based programming languages. Starting with the snap! programming environment, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '15: Proceedings of the eleventh annual International Conference on International Computing Education Research

July 2015

300 pages

ISBN:9781450336307

DOI:10.1145/2787622

General Chair:
Brian Dorn
University of Nebraska at Omaha, USA
,
Program Chairs:
Judy Sheard
Monash University, Australia
,
Quintin Cutts
University of Glasgow, UK

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICER '15

Sponsor:

SIGCSE

ICER '15: International Computing Education Research Conference

July 9 - 13, 2015

Nebraska, Omaha, USA

Acceptance Rates

ICER '15 Paper Acceptance Rate 25 of 96 submissions, 26%;

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025

Sponsor:
sigcse

ACM Conference on International Computing Education Research

August 3 - 6, 2025

Charlottesville , VA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

106
Total Citations
View Citations
1,871
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aristizábal Zapata JGutiérrez Posada JDiago P(2024)Design and Validation of a Computational Thinking Test for Children in the First Grades of Elementary EducationMultimodal Technologies and Interaction10.3390/mti80500398:5(39)Online publication date: 9-May-2024
https://doi.org/10.3390/mti8050039
Hartmann MHielscher MMarinus E(2024)Prevalence of Programming Misconceptions in Primary School StudentsProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699568(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699568
Houchins JLuttgen KOwen RMartinez Rivera LSilberglitt MKao YStephenson BStone JBattestilli LRebelsky SShoop L(2024)Building a Mixed-format Computer Science Assessment for Middle SchoolProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635578(1688-1689)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635578
Negrini LMarras AGiang CBabazadeh M(2024)Exploring Programming Language Choices to Boost Student Interest in Coding and Educational RoboticsHigher Education Learning Methodologies and Technologies Online10.1007/978-3-031-67351-1_1(3-17)Online publication date: 19-Sep-2024
https://doi.org/10.1007/978-3-031-67351-1_1
Sun DZhu CXu FLi YOuyang FCheng M(2023)Transitioning From Introductory to Professional Programming in Secondary Education: Comparing Learners’ Computational Thinking Skills, Behaviors, and AttitudesJournal of Educational Computing Research10.1177/0735633123120465362:3(647-674)Online publication date: 3-Oct-2023
https://doi.org/10.1177/07356331231204653
Branthôme M(2023)Pyrates: Design and Evaluation of a Serious Game Aimed at Introducing Python Programming and Easing the Transition from BlocksACM Transactions on Computing Education10.1145/363906124:1(1-24)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3639061
Lu KKrishnamurthi SFisler KTshukudu E(2023)What Happens When Students Switch (Functional) Languages (Experience Report)Proceedings of the ACM on Programming Languages10.1145/36078577:ICFP(796-812)Online publication date: 31-Aug-2023
https://dl.acm.org/doi/10.1145/3607857
Kunz KMoeller KNinaus MTrautwein UTsarava K(2023)Making the Transition to Text-Based Programming: The Pilot Evaluation of a Computational Thinking Intervention for Primary School StudentsProceedings of the 18th WiPSCE Conference on Primary and Secondary Computing Education Research10.1145/3605468.3609770(1-6)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3605468.3609770
Simmonds JDiaz CEstay CLaakso MMonga MSimon Sheard J(2023)Introducing Computational Thinking at Vocational High SchoolsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588818(68-74)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588818
Israel MLi JYan WElagha NHuggins-Manley CLuo FFranklin D(2023)How are Elementary Students Demonstrating Understanding of Decomposition within Elementary Mathematics?Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600143(222-235)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600143
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents