research-article

Public Access

Development of a Lean Computational Thinking Abilities Assessment for Middle Grades Students

Authors:

Jennifer London,

Bradford W. Mott,

Kristy Elizabeth Boyer,

James C. LesterAuthors Info & Claims

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

Pages 456 - 461

https://doi.org/10.1145/3287324.3287390

Published: 22 February 2019 Publication History

Abstract

The recognition of middle grades as a critical juncture in CS education has led to the widespread development of CS curricula and integration efforts. The goal of many of these interventions is to develop a set of underlying abilities that has been termed computational thinking (CT). This goal presents a key challenge for assessing student learning: we must identify assessment items associated with an emergent understanding of key cognitive abilities underlying CT that avoid specialized knowledge of specific programming languages. In this work we explore the psychometric properties of assessment items appropriate for use with middle grades (US grades 6-8; ages 11-13) students. We also investigate whether these items measure a single ability dimension. Finally, we strive to recommend a "lean" set of items that can be completed in a single 50-minute class period and have high face validity. The paper makes the following contributions: 1) adds to the literature related to the emerging construct of CT, and its relationship to the existing CTt and Bebras instruments, and 2) offers a research-based CT assessment instrument for use by both researchers and educators in the field.

References

[1]

Goode, J. and Chapman, G. Exploring Computer Science. University of Oregon, Eugene, OR, 2016.

[2]

Lee, I., Martin, F. and Apone, K. Integrating computational thinking across the K-8 curriculum. ACM Inroads, 5, 4 (2014), 64--71.

Digital Library

[3]

Mannila, L., Dagiene, V., Demo, B., Grgurina, N., Mirolo, C., Rolandsson, L. and Settle, A. 2014. Computational Thinking in K-9 Education. In Proceedings of the Proceedings of the Working Group Reports of the 2014 on Innovation and Technology in Computer Science Education Conference. ACM, 1--29.

Digital Library

[4]

Zur-Bargury, I., Parv, B. and Lanzberg, D. 2013. A nationwide exam as a tool for improving a new curriculum. In Proceedings of the Proceedings of the 18th ACM conference on Innovation and technology in computer science education. ACM, 267--272.

Digital Library

[5]

Witherspoon, E. B., Higashi, R. M., Schunn, C. D., Baehr, E. C. and Shoop, R. Developing Computational Thinking through a Virtual Robotics Programming Curriculum. ACM Trans. Comput. Educ., 18, 1 (2017), 1--20.

Digital Library

[6]

Martin, L. The promise of the Maker Movement for education. Journal of Pre-College Engineering Education Research (J-PEER), 5, 1 (2015), 4.

[7]

Rodger, S. H., Hayes, J., Lezin, G., Qin, H., Nelson, D., Tucker, R., Lopez, M., Cooper, S., Dann, W. and Slater, D. 2009. Engaging middle school teachers and students with alice in a diverse set of subjects. In Proceedings of the Proceedings of the 40th ACM technical symposium on Computer science education (SIGCSE). ACM, 271--275.

Digital Library

[8]

Bienkowski, M., Snow, E., Rutstein, D. W. and Grover, S. Assessment design patterns for computational thinking practices in secondary computer science: A first look. SRI International, Menlo Park, CA, 2015.

[9]

Grover, S. and Pea, R. Computational Thinking in K--12: A Review of the State of the Field. Educational Researcher, 42, 1 (2013), 38--43.

[10]

{10Wing, J. M. Computational Thinking. Communications of the ACM, 49, 3 (2006), 33--35.

Digital Library

[11]

K12CS K--12 Computer Science Framework. 2016.

[12]

NRC, N. R. C. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. The National Academies, Washington, DC, 2011.

[13]

Buffum, P. S., Martinez-Arocho, A. G., Frankosky, M. H., Rodriguez, F. J., Wiebe, E. N. and Boyer, K. E. 2014. CS principles goes to middle school: learning how to teach Big Data. Proceedings of the 45th ACM technical symposium on computer science education (SIGCSE '14). ACM, 151--156.

Digital Library

[14]

Jona, K., Wilensky, U., Trouille, L., Horn, M., Orton, K., Weintrop, D. and Beheshti, E. 2014. Embedding computational thinking in science, technology, engineering, and math (CT-STEM). Future Directions in Computer Science Education Summit Meeting.

[15]

Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L. and Wilensky, U. Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25, 1 (2016), 127--147.

[16]

Grover, S. 2017. Assessing Algorithmic and Computational Thinking in K-12: Lessons from a Middle School Classroom. In Emerging Research, Practice, and Policy on Computational Thinking. Educational Communications and Technology: Issues and Innovations. Springer, 269--288.

[17]

Grover, S., Cooper, S. and Pea, R. 2014. Assessing computational learning in K-12. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITTICSE '14). ACM, 57--62.

Digital Library

[18]

Shute, V. J., Sun, C. and Asbell-Clarke, J. Demystifying computational thinking. Educational Research Review, 22 (2017), 142--158.

[19]

Brennan, K. and Resnick, M. 2012. New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American Educational Research Association. AERA.

[20]

Denner, J., Werner, L. and Ortiz, E. Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58, 1 (1// 2012), 240--249.

Digital Library

[21]

Tew, A. E. and Guzdial, M. 2011. The FCS1: a language independent assessment of CS1 knowledge. In Proceedings of the Proceedings of the 42nd ACM technical symposium on Computer science education. ACM, 111--116.

Digital Library

[22]

Weintrop, D. and Wilensky, U. 2015. Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs. In International Computing Education Research Conference (ICER '15). ACM, 101--110.

Digital Library

[23]

Taylor, C., Zingaro, D., Porter, L., Webb, K. C., Lee, C. B. and Clancy, M. Computer science concept inventories: past and future. Computer Science Education, 24, 4 (2014), 253--276.

[24]

Curzon, P., McOwan, P. W., Plant, N. and Meagher, L. R. 2014. Introducing teachers to computational thinking using unplugged storytelling. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 89--92.

Digital Library

[25]

Román-González, M., Moreno-León, J. and Robles, G. 2017. Complementary tools for computational thinking assessment. In Proceedings of international conference on computational thinking education (CTE 2017). The Education University of Hong Kong, 154--159.

[26]

Román-González, M., Pérez-González, J.-C. and Jiménez-Fernández, C. Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior 72 (2016), 678--691.

Digital Library

[27]

Román-González, M., Pérez-González, J.-C., Moreno-León, J. and Robles, G. Extending the nomological network of computational thinking with non-cognitive factors. Computers in Human Behavior, 80 (2018), 441--459.

Digital Library

[28]

Alfonso, V. C., Flanagan, D. P. and Radwan, S. The impact of the Cattell-Horn-Carroll theory on test development and interpretation of cognitive and academic abilities. Guilford Publications, City, 2005.

[29]

McGrew, K. S. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence, 37, 1 (2009), 1--10.

[30]

Ambrósio, A. P., Xavier, C. and Georges, F. 2014. Digital ink for cognitive assessment of computational thinking. In Proceedings of the 2014 IEEE Frontiers in Education Conference (FIE). IEEE, 1--7.

[31]

Basawapatna, A. R., Koh, K. H. and Repenning, A. 2010. Using scalable game design to teach computer science from middle school to graduate school. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education. ACM, 224--228.

Digital Library

[32]

Werner, L., Denner, J. and Campe, S. 2012. The Fairy Performance Assessment: Measuring Computational Thinking in Middle School. In Proceeding of the 44th ACM technical symposium on computer science education (SIGCSE '12). ACM, 421--426.

Digital Library

[33]

Dagiene, V. and Futschek, G. 2008. Bebras international contest on informatics and computer literacy: Criteria for good tasks. In International Conference on Informatics in Secondary Schools-Evolution and Perspectives. Springer, 19--30.

Digital Library

[34]

Dagiene, V. and Sentance, S. 2016. It's Computational Thinking! Bebras Tasks in the Curriculum. In International Conference on Informatics in Schools: Situation, Evolution, and Perspectives. Springer, 28--39.

[35]

Aksit, O. Enhancing Science Learning through Computational Thinking and Modeling in Middle School Classrooms: A Mixed Methods Study. Dissertation, North Carolina State University, Raleigh, NC, 2018.

[36]

Moreno-León, J. and Robles, G. 2015. Dr. Scratch: A web tool to automatically evaluate Scratch projects. In Proceedings of the workshop in primary and secondary computing education (WiPSCE '15 ). ACM, 132--133.

Digital Library

[37]

Blokhuis, D., Millican, P., Roffey, C., Schrijvers, E. and Sentance, S. UK Bebras Computational Thinking Challenge 2016. University of Oxford, Oxford, UK, 2015.

[38]

Barendsen, E., Mannila, L., Demo, B., Nata, Grgurina, A., Izu, C., Mirolo, C., Sentance, S., Settle, A., Gabriel, S. 2015. Concepts in K-9 Computer Science Education. In Proceedings of the 2015 ITiCSE on Working Group Reports. ACM, 85--116.

Digital Library

[39]

Dagiene, V., Stupurien, G. and Vinikien, L. 2016. Promoting Inclusive Informatics Education Through the Bebras Challenge to All K-12 Students. In Proceedings of the Proceedings of the 17th International Conference on Computer Systems and Technologies 2016. ACM, 407--414.

Digital Library

[40]

Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A. and Torelli, M. 2015. How Challenging are Bebras Tasks?: An IRT Analysis Based on the Performance of Italian Students. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 27--32.

Digital Library

[41]

Gujberova, M. and Kalas, I. 2013. Designing productive gradations of tasks in primary programming education. In Proceedings of the 8th Workshop in Primary and Secondary Computing Education. ACM, 108--117.

Digital Library

[42]

Hubwieser, P. and Muhling, A. 2014. Playing PISA with Bebras. In Proceedings of the 9th Workshop in Primary and Secondary Computing Education. ACM, 128--129.

Digital Library

[43]

Izu, C., Mirolo, C., Settle, A., Mannila, L. and Stupuriene, G. Exploring Bebras Tasks Content and Performance: A Multinational Study. Informatics in Education, 16, 1 (2017), 39--59.

[44]

Dagiene, V., Mannila, L., Poranen, T., Rolandsson, L. and Derhjelm, S. 2014. Students' performance on programming-related tasks in an informatics contest in Finland, Sweden and Lithuania. In Proceedings of the 2014 conference on Innovation; technology in computer science education. ACM, 153--158.

Digital Library

[45]

Fischer, G. H. and Molenaar, I. W. Rasch models: Foundations, recent developments, and applications. Springer Science, 2012.

[46]

Linacre, J. M. Winsteps®. Winsteps.com, 2018.

[47]

de Ayala, R. J. The Theory and Practice of Item Response Theory. Guilford Press, New York, 2009.

[48]

Chalmers, R. P. mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 6 (2012), 1--29.

Cited By

Asbell-Clarke JDahlstrom-Hakki IVoiklis JAttaway BBarchas-Lichtenstein JEdwards TBardar ERobillard TPaulson KGrover SIsrael MKe FWeintrop D(2024)Including neurodiversity in computational thinkingFrontiers in Education10.3389/feduc.2024.13584929Online publication date: 9-Apr-2024
https://doi.org/10.3389/feduc.2024.1358492
de Jong ISichterman BJeuring J(2024)Use of Computational Thinking Skills when solving Bebras Tasks: a Think-aloud StudyProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699543(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699543
Bumbacher EBrender JDavis RStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Comparative Analysis of Tools & Task Types for Measuring Computational Problem-SolvingProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635547(1580-1581)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635547
Show More Cited By

Index Terms

Development of a Lean Computational Thinking Abilities Assessment for Middle Grades Students
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Infusing computational thinking into middle grade science classrooms: lessons learned
WiPSCE '18: Proceedings of the 13th Workshop in Primary and Secondary Computing Education

There is a growing need to present all students with an opportunity to learn computer science and computational thinking (CT) skills during their primary and secondary education. Traditionally, these opportunities are available outside of the core ...
Teaching how to teach computational thinking
ITiCSE 2018: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education

Computational Thinking is argued to be an essential skill for the workforce of the 21st century. As a skill, Computational Thinking should be taught in all schools, employing computational ideas integrated into other disciplines. Up until now, questions ...
Introducing Middle School Students to Computational Thinking with the CS First Curriculum
ACMSE '20: Proceedings of the 2020 ACM Southeast Conference

Computational thinking (CT) is a set of concepts and problem-solving skills that are not only imperative for computer scientists, but important and applicable to nearly every discipline. CT skills are essential in a world where technology is ubiquitous. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '19: Proceedings of the 50th ACM Technical Symposium on Computer Science Education

February 2019

1364 pages

ISBN:9781450358903

DOI:10.1145/3287324

General Chairs:
Elizabeth K. Hawthorne
Union County College, USA
,
Manuel A. Pérez-Quiñones
University of North Carolina at Charlotte, USA
,
Program Chairs:
Sarah Heckman
North Carolina State University, USA
,
Jian Zhang
Texas Woman's University, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 February 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGCSE '19

Sponsor:

SIGCSE

SIGCSE '19: The 50th ACM Technical Symposium on Computer Science Education

February 27 - March 2, 2019

MN, Minneapolis, USA

Acceptance Rates

SIGCSE '19 Paper Acceptance Rate 169 of 526 submissions, 32%;

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE Virtual 2024

Sponsor:
sigcse

1st ACM Virtual Global Computing Education Conference

December 5 - 8, 2024

Virtual Event , NC , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
1,625
Total Downloads

Downloads (Last 12 months)257
Downloads (Last 6 weeks)41

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Asbell-Clarke JDahlstrom-Hakki IVoiklis JAttaway BBarchas-Lichtenstein JEdwards TBardar ERobillard TPaulson KGrover SIsrael MKe FWeintrop D(2024)Including neurodiversity in computational thinkingFrontiers in Education10.3389/feduc.2024.13584929Online publication date: 9-Apr-2024
https://doi.org/10.3389/feduc.2024.1358492
de Jong ISichterman BJeuring J(2024)Use of Computational Thinking Skills when solving Bebras Tasks: a Think-aloud StudyProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699543(1-11)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699543
Bumbacher EBrender JDavis RStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Comparative Analysis of Tools & Task Types for Measuring Computational Problem-SolvingProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 210.1145/3626253.3635547(1580-1581)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3626253.3635547
Liu T(2024)Assessing implicit computational thinking in game‐based learning: A logical puzzle game studyBritish Journal of Educational Technology10.1111/bjet.1344355:5(2357-2382)Online publication date: 23-Feb-2024
https://doi.org/10.1111/bjet.13443
Shen LMirakhur ZLaCour S(2024)Investigating the psychometric features of a locally designed computational thinking assessment for elementary studentsComputer Science Education10.1080/08993408.2024.2344400(1-20)Online publication date: 27-Apr-2024
https://doi.org/10.1080/08993408.2024.2344400
Kastner-Hauler OTengler KSabitzer BLavicza Z(2024)A Learning Environment to Promote the Computational Thinker: A Bebras Perspective EvaluationInformatics in Schools. Innovative Approaches to Computer Science Teaching and Learning10.1007/978-3-031-73474-8_7(85-98)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1007/978-3-031-73474-8_7
Jeuring JGroot RKeuning H(2023)What Skills Do You Need When Developing Software Using ChatGPT? (Discussion Paper)Proceedings of the 23rd Koli Calling International Conference on Computing Education Research10.1145/3631802.3631807(1-6)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3631802.3631807
Bahr T(2023)Developing a Computer Science Content Knowledge Test for 10th Grade StudentsProceedings of the 18th WiPSCE Conference on Primary and Secondary Computing Education Research10.1145/3605468.3609778(1-2)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3605468.3609778
Hu CWu CLaakso MMonga MSimon Sheard J(2023)A Computational Thinking Test for Senior High School StudentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 210.1145/3587103.3594151(587-588)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587103.3594151
Muñoz RAlegría JRobles G(2023)Assessment of Computational Thinking Skills: A Systematic Review of the LiteratureIEEE Revista Iberoamericana de Tecnologias del Aprendizaje10.1109/RITA.2023.332376218:4(319-330)Online publication date: Nov-2023
https://doi.org/10.1109/RITA.2023.3323762
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents