short-paper

A Dashboard to Provide Instructors with Automated Feedback on Students’ Peer Review Comments

Authors:

Kapotaksha Das,

Solaire Finkenstaedt-Quinn,

Ginger ShultzAuthors Info & Claims

LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

Pages 619 - 625

https://doi.org/10.1145/3576050.3576087

Published: 13 March 2023 Publication History

Abstract

Writing-to-Learn (WTL) is an evidence-based instructional practice which can help students construct knowledge across many disciplines. Though it is known to be an effective practice, many instructors do not implement WTL in their courses due to time constraints and inability to provide students with personalized feedback. One way to address this is to include peer review, which allows students to receive feedback on their writing and benefits them as they act as reviewers. To further ease the implementation of peer review and provide instructors with feedback on their students’ work, we labeled students’ peer review comments across courses for type of feedback provided and trained a machine learning model to automatically classify those comments, improving upon models reported in prior work. We then created a dashboard which takes students’ comments, labels the comments using the model, and allows instructors to filter through their students’ comments based on how the model labels the comments. This dashboard can be used by instructors to monitor the peer review collaborations occurring in their courses. The dashboard will allow them to efficiently use information provided by peers to identify common issues in their students’ writing and better evaluate the quality of their students’ peer review.

References

[1]

Paul Anderson, Chris M. Anson, Robert M. Gonyea, and Charles Paine. 2015. The Contributions of Writing to Learning and Development: Results from a Large-Scale Multi-institutional Study. Research in the Teaching of English 50, 2 (2015), 199–235. https://www.jstor.org/stable/24890033 Publisher: National Council of Teachers of English.

[2]

Robert L. Bangert-Drowns, Marlene M. Hurley, and Barbara Wilkinson. 2004. The Effects of School-Based Writing-to-Learn Interventions on Academic Achievement: A Meta-Analysis. Review of Educational Research 74, 1 (2004), 29–58. https://www.jstor.org/stable/3516060

[3]

Sabri Boughorbel, Fethi Jarray, and Mohammed El-Anbari. 2017. Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric. PLOS ONE 12, 6 (June 2017), e0177678. https://doi.org/10.1371/journal.pone.0177678 Publisher: Public Library of Science.

[4]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1960), 37–46. https://doi.org/10.1177/001316446002000104

[5]

Rosario Delgado and Xavier-Andoni Tibau. 2019. Why Cohen’s Kappa should be avoided as performance measure in classification. PLOS ONE 14, 9 (Sept. 2019), e0222916. https://doi.org/10.1371/journal.pone.0222916 Publisher: Public Library of Science.

[6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805(2018). http://arxiv.org/abs/1810.04805 _eprint: 1810.04805.

[7]

Amber Dood, Blair Winograd, Solaire Finkenstaedt-Quinn, Anne Gere, and Ginger Shultz. 2022. PeerBERT: Automated Characterization of Peer Review Comments across Courses. In LAK22: 12th International Learning Analytics and Knowledge Conference(LAK22). Association for Computing Machinery, New York, NY, USA, 492–499. https://doi.org/10.1145/3506860.3506892

Digital Library

[8]

Amber J. Dood, John C. Dood, Daniel Cruz-Ramírez de Arellano, Kimberly B. Fields, and Jeffrey R. Raker. 2020. Analyzing explanations of substitution reactions using lexical analysis and logistic regression techniques. Chemistry Education Research and Practice 21, 1 (Jan. 2020), 267–286. https://doi.org/10.1039/C9RP00148D Publisher: The Royal Society of Chemistry.

[9]

Amber J. Dood, John C. Dood, Daniel Cruz-Ramírez de Arellano, Kimberly B. Fields, and Jeffrey R. Raker. 2020. Using the Research Literature to Develop an Adaptive Intervention to Improve Student Explanations of an SN1 Reaction Mechanism. Journal of Chemical Education 97, 10 (Oct. 2020), 3551–3562. https://doi.org/10.1021/acs.jchemed.0c00569 Publisher: American Chemical Society.

[10]

Amber J. Dood, Kimberly B. Fields, Daniel Cruz-Ramírez de Arellano, and Jeffrey R. Raker. 2019. Development and evaluation of a Lewis acid–base tutorial for use in postsecondary organic chemistry courses. Canadian Journal of Chemistry 97, 10 (July 2019), 711–721. https://doi.org/10.1139/cjc-2018-0479 Publisher: NRC Research Press.

[11]

Amber J. Dood, Kimberly B. Fields, and Jeffrey R. Raker. 2018. Using Lexical Analysis To Predict Lewis Acid–Base Model Use in Responses to an Acid–Base Proton-Transfer Reaction. Journal of Chemical Education 95, 8 (Aug. 2018), 1267–1275. https://doi.org/10.1021/acs.jchemed.8b00177

[12]

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why Does Unsupervised Pre-training Help Deep Learning?Journal of Machine Learning Research 11, 19 (2010), 625–660. http://jmlr.org/papers/v11/erhan10a.html

[13]

Solaire A. Finkenstaedt-Quinn, Anne Ruggles Gere, Jason E. Dowd, Robert J. Thompson, Audrey S. Halim, Julie A. Reynolds, Leslie A. Schiff, Pamela Flash, and Ginger V. Shultz. 2022. Postsecondary Faculty Attitudes and Beliefs about Writing-Based Pedagogies in the STEM Classroom. CBE—Life Sciences Education 21, 3 (Sept. 2022), ar54. https://doi.org/10.1187/cbe.21-09-0285 Publisher: American Society for Cell Biology (lse).

[14]

Solaire A. Finkenstaedt-Quinn, Michael Petterson, Anne Gere, and Ginger Shultz. 2021. Praxis of Writing-to-Learn: A Model for the Design and Propagation of Writing-to-Learn in STEM. Journal of Chemical Education 98, 5 (May 2021), 1548–1555. https://doi.org/10.1021/acs.jchemed.0c01482 Publisher: American Chemical Society.

[15]

Solaire A. Finkenstaedt-Quinn, Noelle Polakowski, Brenda Gunderson, Ginger V. Shultz, and Anne Ruggles Gere. 2021. Utilizing Peer Review and Revision in STEM to Support the Development of Conceptual Knowledge Through Writing. Written Communication 38, 3 (July 2021), 351–379. https://doi.org/10.1177/07410883211006038 Publisher: SAGE Publications Inc.

[16]

S. A. Finkenstaedt-Quinn, E. P. Snyder-White, M. C. Connor, A. Ruggles Gere, and G. V. Shultz. 2019. Characterizing Peer Review Comments and Revision from a Writing-to-Learn Assignment Focused on Lewis Structures. Journal of Chemical Education 96, 2 (Feb. 2019), 227–237. https://doi.org/10.1021/acs.jchemed.8b00711 Publisher: American Chemical Society.

[17]

Anne Ruggles Gere, Naitnaphit Limlamai, Emily Wilson, Kate MacDougall Saylor, and Raymond Pugh. 2019. Writing and Conceptual Learning in Science: An Analysis of Assignments. Written Communication 36, 1 (Jan. 2019), 99–135. https://doi.org/10.1177/0741088318804820 Publisher: SAGE Publications Inc.

[18]

Murat Gunel, Brian Hand, and Vaughan Prain. 2007. Writing for Learning in Science: A Secondary Analysis of Six Studies. International Journal of Science and Mathematics Education 5, 4 (Dec. 2007), 615–637. https://doi.org/10.1007/s10763-007-9082-y

[19]

Kevin C. Haudek, Jennifer J. Kaplan, Jennifer Knight, Tammy Long, John Merrill, Alan Munn, Ross Nehm, Michelle Smith, and Mark Urban-Lurain. 2011. Harnessing technology to improve formative assessment of student conceptions in STEM: forging a national network. CBE-Life Sciences Education 10, 2 (June 2011), 149–155. https://doi.org/10.1187/cbe.11-03-0019

[20]

Bart Huisman, Nadira Saab, Paul van den Broek, and Jan van Driel. 2019. The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis. Assessment & Evaluation in Higher Education 44, 6 (Aug. 2019), 863–880. https://doi.org/10.1080/02602938.2018.1545896 Publisher: Routledge _eprint: https://doi.org/10.1080/02602938.2018.1545896.

[21]

Timothy P. Jurka, Loren Collingwood, Amber E. Boydstun, Emiliano Grossman, and Wouter van Atteveldt. 2012. RTextsTools: Automatic Text Classification via Supervised Learning. http://CRAN.R-project.org/package=RTextTools

[22]

Cannannore Nidhi Kamath, Syed Saqib Bukhari, and Andreas Dengel. 2018. Comparative Study between Traditional Machine Learning and Deep Learning Approaches for Text Classification. In Proceedings of the ACM Symposium on Document Engineering 2018(DocEng ’18). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3209280.3209526

Digital Library

[23]

Jennifer J. Kaplan, Kevin C. Haudek, Minsu Ha, Neal Rogness, and Diane G. Fisher. 2014. Using Lexical Analysis Software to Assess Student Writing in Statistics. Technology Innovations in Statistics Education 8, 1 (2014). https://escholarship.org/uc/item/57r90703

[24]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs] (July 2019). http://arxiv.org/abs/1907.11692 arXiv:1907.11692.

[25]

Kristi Lundstrom and Wendy Baker. 2009. To give is better than to receive: The benefits of peer review to the reviewer’s own writing. Journal of Second Language Writing 18, 1 (March 2009), 30–43. https://doi.org/10.1016/j.jslw.2008.06.002

[26]

Alena Moon, Anne Ruggles Gere, and Ginger V. Shultz. 2018. Writing in the STEM classroom: Faculty conceptions of writing and its role in the undergraduate classroom. Science Education 102, 5 (2018), 1007–1028. https://doi.org/10.1002/sce.21454 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sce.21454.

[27]

Melissa M. Patchan, Christian D. Schunn, and Richard J. Correnti. 2016. The nature of feedback: How peer feedback features affect students’ implementation rate and quality of revisions. Journal of Educational Psychology 108, 8 (2016), 1098–1120. https://doi.org/10.1037/edu0000103 Place: US Publisher: American Psychological Association.

[28]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (Oct. 2011), 2825−2830. http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html

Digital Library

[29]

Nancy J. Pelaez. 2002. Problem-based writing with peer review improves academic performance in physiology. Advances in Physiology Education 26, 3 (Sept. 2002), 174–184. https://doi.org/10.1152/advan.00041.2001 Publisher: American Physiological Society.

[30]

Luanna B Prevost, Kevin C Haudek, Emily Norton Henry, Mark Urban-Lurain, and Matthew C Berry. 2013. Automated text analysis facilitates using written formative assessments for just-in-time teaching in large enrollment courses.

[31]

Afsar Rouhi and Elnaz Azizian. 2013. Peer review: Is Giving Corrective Feedback Better than Receiving it in L2 Writing?Procedia - Social and Behavioral Sciences 93 (Oct. 2013), 1349–1354. https://doi.org/10.1016/j.sbspro.2013.10.042

[32]

Selma Tekir and Yalin Bastanlar. 2020. Deep Learning: Exemplar Studies in Natural Language Processing and Computer Vision. IntechOpen. https://doi.org/10.5772/intechopen.91813 Publication Title: Data Mining - Methods, Applications and Systems.

[33]

Field M. Watts and Solaire A. Finkenstaedt-Quinn. 2021. The current state of methods for establishing reliability in qualitative chemistry education research articles. Chemistry Education Research and Practice 22, 3 (July 2021), 565–578. https://doi.org/10.1039/D1RP00007A Publisher: The Royal Society of Chemistry.

[34]

David M. Williamson, Xiaoming Xi, and F. Jay Breyer. 2012. A Framework for Evaluation and Use of Automated Scoring. Educational Measurement: Issues and Practice 31, 1 (2012), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1745-3992.2011.00223.x.

[35]

Blair A. Winograd, Amber J. Dood, Solaire A. Finkenstaedt-Quinn, Anne R. Gere, and Ginger V. Shultz. 2021. Automating Characterization of Peer Review Comments in Chemistry Courses. In 14th Computer-Supported Collaborative Learning (CSCL) - Proceedings. Bochum, Germany, 11–18. https://2021.isls.org/proceedings/

[36]

Brandon J. Yik, Amber J. Dood, Daniel Cruz-Ramírez de Arellano, Kimberly B. Fields, and Jeffrey R. Raker. 2021. Development of a machine learning-based tool to evaluate correct Lewis acid–base model use in written responses to open-ended formative assessment items. Chemistry Education Research and Practice 22 (July 2021), 866–885. https://doi.org/10.1039/D1RP00111F Publisher: The Royal Society of Chemistry.

[37]

Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels. https://doi.org/10.48550/arXiv.1805.07836 arXiv:1805.07836 [cs, stat].

Cited By

Alfredo REcheverria VJin YSwiecki ZGašević DMartinez-Maldonado R(2024)SLADE: A Method for Designing Human-Centred Learning Analytics SystemsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636847(24-34)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636847
Dood AWatts FConnor MShultz G(2024)Automated Text Analysis of Organic Chemistry Students’ Written HypothesesJournal of Chemical Education10.1021/acs.jchemed.3c00757101:3(807-818)Online publication date: 15-Feb-2024
https://doi.org/10.1021/acs.jchemed.3c00757
Yik BSchreurs DRaker J(2023)Implementation of an R Shiny App for Instructors: An Automated Text Analysis Formative Assessment Tool for Evaluating Lewis Acid–Base Model UseJournal of Chemical Education10.1021/acs.jchemed.3c00400100:8(3107-3113)Online publication date: 27-Jul-2023
https://doi.org/10.1021/acs.jchemed.3c00400

Index Terms

A Dashboard to Provide Instructors with Automated Feedback on Students’ Peer Review Comments
1. Applied computing
  1. Education

Recommendations

Automated, content-focused feedback for a writing-to-learn assignment in an undergraduate organic chemistry course
LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

Writing-to-learn (WTL) pedagogy supports the implementation of writing assignments in STEM courses to engage students in conceptual learning. Recent studies in the undergraduate STEM context demonstrate the value of implementing WTL, with findings that ...
PeerBERT: Automated Characterization of Peer Review Comments across Courses
LAK22: LAK22: 12th International Learning Analytics and Knowledge Conference

Writing-to-learn pedagogies are an evidence-based practice known to aid students in constructing knowledge. Barriers exist for the implementation of such assignments; namely, instructors feel they do not have time to provide each student with feedback. ...
Lightweight preliminary peer review: does in-class peer review make sense?
SIGCSE '07: Proceedings of the 38th SIGCSE technical symposium on Computer science education

Peer review is widely recognized for advancing student learning, in particular for developing reflective processes like critical thinking. The classroom is ripe for peer review because the subject matter is fresh and in-depth interactivity is possible. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference

March 2023

692 pages

ISBN:9781450398657

DOI:10.1145/3576050

Program Chairs:
Isabel Hilliger
Pontificia Universidad Católica de Chile, Chile
,
Hassan Khosravi
University of Queensland, Australia
,
Bart Rienties
Open University, United Kingdom
,
Shane Dawson
University of South Australia, Australia

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

LAK 2023

LAK 2023: 13th International Learning Analytics and Knowledge Conference

March 13 - 17, 2023

TX, Arlington, USA

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
218
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alfredo REcheverria VJin YSwiecki ZGašević DMartinez-Maldonado R(2024)SLADE: A Method for Designing Human-Centred Learning Analytics SystemsProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636847(24-34)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636847
Dood AWatts FConnor MShultz G(2024)Automated Text Analysis of Organic Chemistry Students’ Written HypothesesJournal of Chemical Education10.1021/acs.jchemed.3c00757101:3(807-818)Online publication date: 15-Feb-2024
https://doi.org/10.1021/acs.jchemed.3c00757
Yik BSchreurs DRaker J(2023)Implementation of an R Shiny App for Instructors: An Automated Text Analysis Formative Assessment Tool for Evaluating Lewis Acid–Base Model UseJournal of Chemical Education10.1021/acs.jchemed.3c00400100:8(3107-3113)Online publication date: 27-Jul-2023
https://doi.org/10.1021/acs.jchemed.3c00400

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten