research-article

Public Access

Development of a Lean Computational Thinking Abilities Assessment for Middle Grades Students

Authors:

Jennifer London,

Bradford W. Mott,

Kristy Elizabeth Boyer,

James C. LesterAuthors Info & Claims

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Pages 456 - 461

https://doi.org/10.1145/3287324.3287390

Published: 22 February 2019 Publication History

Abstract

The recognition of middle grades as a critical juncture in CS education has led to the widespread development of CS curricula and integration efforts. The goal of many of these interventions is to develop a set of underlying abilities that has been termed computational thinking (CT). This goal presents a key challenge for assessing student learning: we must identify assessment items associated with an emergent understanding of key cognitive abilities underlying CT that avoid specialized knowledge of specific programming languages. In this work we explore the psychometric properties of assessment items appropriate for use with middle grades (US grades 6-8; ages 11-13) students. We also investigate whether these items measure a single ability dimension. Finally, we strive to recommend a "lean" set of items that can be completed in a single 50-minute class period and have high face validity. The paper makes the following contributions: 1) adds to the literature related to the emerging construct of CT, and its relationship to the existing CTt and Bebras instruments, and 2) offers a research-based CT assessment instrument for use by both researchers and educators in the field.

References

[1]

Goode, J. and Chapman, G. Exploring Computer Science. University of Oregon, Eugene, OR, 2016.

[2]

Lee, I., Martin, F. and Apone, K. Integrating computational thinking across the K-8 curriculum. ACM Inroads, 5, 4 (2014), 64--71.

Digital Library

[3]

Mannila, L., Dagiene, V., Demo, B., Grgurina, N., Mirolo, C., Rolandsson, L. and Settle, A. 2014. Computational Thinking in K-9 Education. In Proceedings of the Proceedings of the Working Group Reports of the 2014 on Innovation and Technology in Computer Science Education Conference. ACM, 1--29.

Digital Library

[4]

Zur-Bargury, I., Parv, B. and Lanzberg, D. 2013. A nationwide exam as a tool for improving a new curriculum. In Proceedings of the Proceedings of the 18th ACM conference on Innovation and technology in computer science education. ACM, 267--272.

Digital Library

[5]

Witherspoon, E. B., Higashi, R. M., Schunn, C. D., Baehr, E. C. and Shoop, R. Developing Computational Thinking through a Virtual Robotics Programming Curriculum. ACM Trans. Comput. Educ., 18, 1 (2017), 1--20.

Digital Library

[6]

Martin, L. The promise of the Maker Movement for education. Journal of Pre-College Engineering Education Research (J-PEER), 5, 1 (2015), 4.

[7]

Rodger, S. H., Hayes, J., Lezin, G., Qin, H., Nelson, D., Tucker, R., Lopez, M., Cooper, S., Dann, W. and Slater, D. 2009. Engaging middle school teachers and students with alice in a diverse set of subjects. In Proceedings of the Proceedings of the 40th ACM technical symposium on Computer science education (SIGCSE). ACM, 271--275.

Digital Library

[8]

Bienkowski, M., Snow, E., Rutstein, D. W. and Grover, S. Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI International, Menlo Park, CA, 2015.

[9]

Grover, S. and Pea, R. Computational Thinking in K--12: A Review of the State of the Field. Educational Researcher, 42, 1 (2013), 38--43.

[10]

{10Wing, J. M. Computational Thinking. Communications of the ACM, 49, 3 (2006), 33--35.

Digital Library

[11]

K12CS K--12 Computer Science Framework. 2016.

[12]

NRC, N. R. C. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. The National Academies, Washington, DC, 2011.

[13]

Buffum, P. S., Martinez-Arocho, A. G., Frankosky, M. H., Rodriguez, F. J., Wiebe, E. N. and Boyer, K. E. 2014. CS principles goes to middle school: learning how to teach Big Data. Proceedings of the 45th ACM technical symposium on computer science education (SIGCSE '14). ACM, 151--156.

Digital Library

[14]

Jona, K., Wilensky, U., Trouille, L., Horn, M., Orton, K., Weintrop, D. and Beheshti, E. 2014. Embedding computational thinking in science, technology, engineering, and math (CT-STEM). Future Directions in Computer Science Education Summit Meeting.

[15]

Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L. and Wilensky, U. Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25, 1 (2016), 127--147.

[16]

Grover, S. 2017. Assessing Algorithmic and Computational Thinking in K-12: Lessons from a Middle School Classroom. In Emerging Research, Practice, and Policy on Computational Thinking. Educational Communications and Technology: Issues and Innovations. Springer, 269--288.

[17]

Grover, S., Cooper, S. and Pea, R. 2014. Assessing computational learning in K-12. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITTICSE '14). ACM, 57--62.

Digital Library

[18]

Shute, V. J., Sun, C. and Asbell-Clarke, J. Demystifying computational thinking. Educational Research Review, 22 (2017), 142--158.

[19]

Brennan, K. and Resnick, M. 2012. New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American Educational Research Association. AERA.

[20]

Denner, J., Werner, L. and Ortiz, E. Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58, 1 (1// 2012), 240--249.

Digital Library

[21]

Tew, A. E. and Guzdial, M. 2011. The FCS1: a language independent assessment of CS1 knowledge. In Proceedings of the Proceedings of the 42nd ACM technical symposium on Computer science education. ACM, 111--116.

Digital Library

[22]

Weintrop, D. and Wilensky, U. 2015. Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs. In International Computing Education Research Conference (ICER '15). ACM, 101--110.

Digital Library

[23]

Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B. and Clancy, M. Computer science concept inventories: past and future. Computer Science Education, 24, 4 (2014), 253--276.

[24]

Curzon, P., McOwan, P. W., Plant, N. and Meagher, L. R. 2014. Introducing teachers to computational thinking using unplugged storytelling. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 89--92.

Digital Library

[25]

Román-González, M., Moreno-León, J. and Robles, G. 2017. Complementary tools for computational thinking assessment. In Proceedings of international conference on computational thinking education (CTE 2017). The Education University of Hong Kong, 154--159.

[26]

Román-González, M., Pérez-González, J.-C. and Jiménez-Fernández, C. Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior 72 (2016), 678--691.

Digital Library

[27]

Román-González, M., Pérez-González, J.-C., Moreno-León, J. and Robles, G. Extending the nomological network of computational thinking with non-cognitive factors. Computers in Human Behavior, 80 (2018), 441--459.

Digital Library

[28]

Alfonso, V. C., Flanagan, D. P. and Radwan, S. The impact of the Cattell-Horn-Carroll theory on test development and interpretation of cognitive and academic abilities. Guilford Publications, City, 2005.

[29]

McGrew, K. S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1 (2009), 1--10.

[30]

Ambrósio, A. P., Xavier, C. and Georges, F. 2014. Digital ink for cognitive assessment of computational thinking. In Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE). IEEE, 1--7.

[31]

Basawapatna, A. R., Koh, K. H. and Repenning, A. 2010. Using scalable game design to teach computer science from middle school to graduate school. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education. ACM, 224--228.

Digital Library

[32]

Werner, L., Denner, J. and Campe, S. 2012. The Fairy Performance Assessment: Measuring Computational Thinking in Middle School. In Proceeding of the 44th ACM technical symposium on computer science education (SIGCSE '12). ACM, 421--426.

Digital Library

[33]

Dagiene, V. and Futschek, G. 2008. Bebras international contest on informatics and computer literacy: Criteria for good tasks. In International Conference on Informatics in Secondary Schools-Evolution and Perspectives. Springer, 19--30.

Digital Library

[34]

Dagiene, V. and Sentance, S. 2016. It's Computational Thinking! Bebras Tasks in the Curriculum. In International Conference on Informatics in Schools: Situation, Evolution, and Perspectives. Springer, 28--39.

[35]

Aksit, O. Enhancing Science Learning through Computational Thinking and Modeling in Middle School Classrooms: A Mixed Methods Study. Dissertation, North Carolina State University, Raleigh, NC, 2018.

[36]

Moreno-León, J. and Robles, G. 2015. Dr. Scratch: A web tool to automatically evaluate Scratch projects. In Proceedings of the workshop in primary and secondary computing education (WiPSCE '15 ). ACM, 132--133.

Digital Library

[37]

Blokhuis, D., Millican, P., Roffey, C., Schrijvers, E. and Sentance, S. UK Bebras Computational Thinking Challenge 2016. University of Oxford, Oxford, UK, 2015.

[38]

Barendsen, E., Mannila, L., Demo, B., Nata, Grgurina, A., Izu, C., Mirolo, C., Sentance, S., Settle, A., Gabriel, S. 2015. Concepts in K-9 Computer Science Education. In Proceedings of the 2015 ITiCSE on Working Group Reports. ACM, 85--116.

Digital Library

[39]

Dagiene, V., Stupurien, G. and Vinikien, L. 2016. Promoting Inclusive Informatics Education Through the Bebras Challenge to All K-12 Students. In Proceedings of the Proceedings of the 17th International Conference on Computer Systems and Technologies 2016. ACM, 407--414.

Digital Library

[40]

Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A. and Torelli, M. 2015. How Challenging are Bebras Tasks?: An IRT Analysis Based on the Performance of Italian Students. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 27--32.

Digital Library

[41]

Gujberova, M. and Kalas, I. 2013. Designing productive gradations of tasks in primary programming education. In Proceedings of the 8th Workshop in Primary and Secondary Computing Education. ACM, 108--117.

Digital Library

[42]

Hubwieser, P. and Muhling, A. 2014. Playing PISA with Bebras. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 128--129.

Digital Library

[43]

Izu, C., Mirolo, C., Settle, A., Mannila, L. and Stupuriene, G. Exploring Bebras Tasks Content and Performance: A Multinational Study. Informatics in Education, 16, 1 (2017), 39--59.

[44]

Dagiene, V., Mannila, L., Poranen, T., Rolandsson, L. and Derhjelm, S. 2014. Students' performance on programming-related tasks in an informatics contest in Finland, Sweden and Lithuania. In Proceedings of the 2014 conference on Innovation; technology in computer science education. ACM, 153--158.

Digital Library

[45]

Fischer, G. H. and Molenaar, I. W. Rasch models: Foundations, recent developments, and applications. Springer Science, 2012.

[46]

Linacre, J. M. Winsteps®. Winsteps.com, 2018.

[47]

de Ayala, R. J. The Theory and Practice of Item Response Theory. Guilford Press, New York, 2009.

[48]

Chalmers, R. P. mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 6 (2012), 1--29.

Cited By

Blanc SConchado ABenlloch-Dualde JMonteiro AGrindei L(2025)Digital competence development in schools: a study on the association of problem-solving with autonomy and digital attitudesInternational Journal of STEM Education10.1186/s40594-025-00534-612:1Online publication date: 23-Feb-2025
https://doi.org/10.1186/s40594-025-00534-6
Le Du JAlvarez JSchmitt D(2024)Développer la pensée informatique à travers la conception de jeux vidéo éducatifs. Educational Game & Play Design Project: élaboration de la méthodeDistances et médiations des savoirs10.4000/12xon48Online publication date: 2024
https://doi.org/10.4000/12xon
Asbell-Clarke JDahlstrom-Hakki IVoiklis JAttaway BBarchas-Lichtenstein JEdwards TBardar ERobillard TPaulson KGrover SIsrael MKe FWeintrop D(2024)Including neurodiversity in computational thinkingFrontiers in Education10.3389/feduc.2024.13584929Online publication date: 9-Apr-2024
https://doi.org/10.3389/feduc.2024.1358492
Show More Cited By

Recommendations

Computational thinking
Self managed systems

It represents a universally applicable attitude and skill set everyone, not just computer scientists, would be eager to learn and use.
Development of Computational Thinking Skills through Unplugged Activities in Primary School
WiPSCE '17: Proceedings of the 12th Workshop on Primary and Secondary Computing Education

Computational thinking is nowadays being widely adopted and investigated. Educators and researchers are using two main approaches to teach these skills in schools: with computer programming exercises, and with unplugged activities that do not require ...
Beyond Programming: A Computer-Based Assessment of Computational Thinking Competency
Computational Thinking (CT), entailing both domain-general and domain-specific skills, is a competency fundamental to computing education and beyond. However, as a cross-domain competency, appropriate assessment design and method remain equivocal. Indeed, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

February 2019

1364 pages

ISBN:9781450358903

DOI:10.1145/3287324

General Chairs:
Elizabeth K. Hawthorne
Union County College, USA
,
Manuel A. Pérez-Quiñones
University of North Carolina at Charlotte, USA
,
Program Chairs:
Sarah Heckman
North Carolina State University, USA
,
Jian Zhang
Texas Woman's University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGCSE '19

Sponsor:

SIGCSE

SIGCSE '19: The 50th ACM Technical Symposium on Computer Science Education

February 27 - March 2, 2019

MN, Minneapolis, USA

Acceptance Rates

SIGCSE '19 Paper Acceptance Rate 169 of 526 submissions, 32%;

Overall Acceptance Rate 1,787 of 5,146 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
1,714
Total Downloads

Downloads (Last 12 months)256
Downloads (Last 6 weeks)33

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Blanc SConchado ABenlloch-Dualde JMonteiro AGrindei L(2025)Digital competence development in schools: a study on the association of problem-solving with autonomy and digital attitudesInternational Journal of STEM Education10.1186/s40594-025-00534-612:1Online publication date: 23-Feb-2025
https://doi.org/10.1186/s40594-025-00534-6
Le Du JAlvarez JSchmitt D(2024)Développer la pensée informatique à travers la conception de jeux vidéo éducatifs. Educational Game & Play Design Project: élaboration de la méthodeDistances et médiations des savoirs10.4000/12xon48Online publication date: 2024
https://doi.org/10.4000/12xon
Asbell-Clarke JDahlstrom-Hakki IVoiklis JAttaway BBarchas-Lichtenstein JEdwards TBardar ERobillard TPaulson KGrover SIsrael MKe FWeintrop D(2024)Including neurodiversity in computational thinkingFrontiers in Education10.3389/feduc.2024.13584929Online publication date: 9-Apr-2024
https://doi.org/10.3389/feduc.2024.1358492
de Jong ISichterman BJeuring J(2024)Use of Computational Thinking Skills when solving Bebras Tasks: a Think-aloud StudyProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699543(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699543
Bumbacher EBrender JDavis RStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Comparative Analysis of Tools & Task Types for Measuring Computational Problem-SolvingProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635547(1580-1581)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635547
Liu T(2024)Assessing implicit computational thinking in game‐based learning: A logical puzzle game studyBritish Journal of Educational Technology10.1111/bjet.1344355:5(2357-2382)Online publication date: 23-Feb-2024
https://doi.org/10.1111/bjet.13443
Shen LMirakhur ZLaCour S(2024)Investigating the psychometric features of a locally designed computational thinking assessment for elementary studentsComputer Science Education10.1080/08993408.2024.2344400(1-20)Online publication date: 27-Apr-2024
https://doi.org/10.1080/08993408.2024.2344400
Kastner-Hauler OTengler KSabitzer BLavicza Z(2024)A Learning Environment to Promote the Computational Thinker: A Bebras Perspective EvaluationInformatics in Schools. Innovative Approaches to Computer Science Teaching and Learning10.1007/978-3-031-73474-8_7(85-98)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1007/978-3-031-73474-8_7
Jeuring JGroot RKeuning H(2023)What Skills Do You Need When Developing Software Using ChatGPT? (Discussion Paper)Proceedings of the 23rd Koli Calling International Conference on Computing Education Research10.1145/3631802.3631807(1-6)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3631802.3631807
Bahr T(2023)Developing a Computer Science Content Knowledge Test for 10th Grade StudentsProceedings of the 18th WiPSCE Conference on Primary and Secondary Computing Education Research10.1145/3605468.3609778(1-2)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3605468.3609778
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten