research-article

Predicting Academic Success Based on Learning Material Usage

Authors:

Petri Ihantola,

Arto HellasAuthors Info & Claims

SIGITE '17: Proceedings of the 18th Annual Conference on Information Technology Education

Pages 13 - 18

https://doi.org/10.1145/3125659.3125695

Published: 27 September 2017 Publication History

Abstract

In this work, we explore students' usage of online learning material as a predictor of academic success. In the context of an introductory programming course, we recorded the amount of time that each element such as a text paragraph or an image was visible on the students' screen. Then, we applied machine learning methods to study to what extent material usage predicts course outcomes. Our results show that the time spent with each paragraph of the online learning material is a moderate predictor of student success even when corrected for student time-on-task, and that the information can be used to identify at-risk students. The predictive performance of the models is dependent on the quantity of data, and the predictions become more accurate as the course progresses. In a broader context, our results indicate that course material usage can be used to predict academic success, and that such data can be collected in-situ with minimal interference to the students' learning process.

References

[1]

Philip C Abrami, Bette Chambers, Sylvia d'Apollonia, Mona Farrell, and Christina De Simone. 1992. Group outcome: The relationship between group learning outcome, attributional style, academic achievement, and self-concept. Contemporary Educational Psychology 17, 3 (1992), 201--210.

[2]

Alireza Ahadi, Raymond Lister, Heikki Haapala, and Arto Vihavainen. 2015. Exploring machine learning methods to automatically identify students in need of assistance. In Proc. of the 11th Annual International Conference on International Computing Education Research. ACM, 121--130.

Digital Library

[3]

Saleema Amershi and Cristina Conati. 2009. Combining unsupervised and supervised classification to build user models for exploratory. Journal of Educational Data Mining 1, 1 (2009), 18--71.

[4]

Vladimir Bacvanski and Jürgen Börstler. 1997. Doing Your First OO Project: OO Education Issues in Industry and Academia. In Addendum to the 1997 ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications (Addendum) (OOPSLA '97). ACM, 93--96.

Digital Library

[5]

Albert Bandura. 1977. Self-efficacy: toward a unifying theory of behavioral change. Psychological review 84, 2 (1977), 191.

[6]

Debasish Basak, Srimanta Pal, and Dipak Chandra Patranabis. 2007. Support vector regression. Neural Information Processing-Letters and Reviews 11, 10 (2007), 203--224.

[7]

Brett A Becker. 2016. A new metric to quantify repeated compiler errors for novice programmers. In Proc. of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 296--301.

Digital Library

[8]

Christopher JC Burges. 1998. A tutorial on support vector machines for pattern recognition. Data mining and knowledge discovery 2, 2 (1998), 121--167.

Digital Library

[9]

Adam S Carter, Christopher D Hundhausen, and Olusola Adesope. 2015. The normalized programming state model: Predicting student performance in computing courses based on programming behavior. In Proc. of the 11th annual International Conference on International Computing Education Research. ACM, 141--150.

Digital Library

[10]

Rich Caruana and Alexandru Niculescu-Mizil. 2006. An empirical comparison of supervised learning algorithms. In Proc. of the 23rd international conference on Machine learning. ACM, 161--168.

Digital Library

[11]

Karo Castro-Wunsch, Alireza Ahadi, and Andrew Petersen. 2017. Evaluating Neural Networks As a Method for Identifying Students in Need of Assistance. In Proc. of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE '17). ACM, 111--116.

Digital Library

[12]

Cynthia L Corritore and Susan Wiedenbeck. 1991. What do novices learn during program comprehension? International Journal of Human-Computer Interaction 3, 2 (1991), 199--222.

[13]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273--297.

Digital Library

[14]

K Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological review 100, 3 (1993), 363.

[15]

Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim. 2014. Do we need hundreds of classifiers to solve real world classification problems. J. Mach. Learn. Res 15, 1 (2014), 3133--3181.

Digital Library

[16]

Sally Fincher, Anthony Robins, Bob Baker, Ilona Box, Quintin Cutts, Michael de Raadt, Patricia Haden, John Hamer, Margaret Hamilton, Raymond Lister, and others. 2006. Predictors of success in a first programming course. In Proc. of the 8th Australasian Conference on Computing Education-Volume 52. Australian Computer Society, Inc., 189--196.

Digital Library

[17]

Sandra L Fisher and J Kevin Ford. 1998. Differential effects of learner effort and goal orientation on two learning outcomes. Personnel Psychology 51, 2 (1998), 397--420.

[18]

John H Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American psychologist 34, 10 (1979), 906.

[19]

J Kevin Ford, Eleanor M Smith, Daniel A Weissbein, Stanley M Gully, and Eduardo Salas. 1998. Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of applied psychology 83, 2 (1998), 218.

[20]

Mark Guzdial. 1995. Centralized Mindset: A Student Problem with Object-oriented Programming. In Proc. of the 26th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '95). ACM, 182--185.

Digital Library

[21]

Petri Ihantola, Arto Vihavainen, Alireza Ahadi, Matthew Butler, Jürgen Börstler, Stephen H Edwards, Essi Isohanni, Ari Korhonen, Andrew Petersen, Kelly Rivers, and others. 2015. Educational data mining and learning analytics in programming: Literature review and case studies. In Proc. of the 2015 ITiCSE on Working Group Reports. ACM, 41--63.

Digital Library

[22]

Matthew C Jadud. 2006. Methods and tools for exploring novice compilation behaviour. In Proc. of the 2nd international workshop on Computing education research. ACM, 73--84.

Digital Library

[23]

Young-Ju Joo, Mimi Bong, and Ha-Jeen Choi. 2000. Self-efficacy for self-regulated learning, academic self-efficacy, and Internet self-efficacy in Web-based instruction. Educational Technology Research and Development 48, 2 (2000), 5--17.

[24]

Gregor Kennedy and Quintin Cutts. 2005. The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning 21, 4 (2005), 260--268.

[25]

Juho Leinonen, Krista Longi, Arto Klami, and Arto Vihavainen. 2016. Automatic inference of programming performance and experience from typing patterns. In Proc. of the 47th ACM Technical Symposium on Computing Science Education. ACM, 132--137.

Digital Library

[26]

Leo Leppänen, Juho Leinonen, and Arto Hellas. 2016. Pauses and spacing in learning to program. In Proc. of the 16th Koli Calling International Conference on Computing Education Research. ACM, 41--50.

Digital Library

[27]

Leo Leppänen, Juho Leinonen, Petri Ihantola, and Arto Hellas. 2017. Using and Collecting Fine-grained Usage Data to Improve Online Learning Materials. In Proceedings of the 39th International Conference on Software Engineering: Software Engineering and Education Track (ICSE-SEET '17). IEEE Press, 4--12.

Digital Library

[28]

Mark R Lepper and Diana I Cordova. 1992. A desire to be taught: Instructional consequences of intrinsic motivation. Motivation and emotion 16, 3 (1992), 187-- 208.

[29]

Leo Leppänen. 2017. Using element-level website usage data to improve online learning materials and predict learning outcomes. Master's thesis. University of Helsinki, Helsinki, Finland.

[30]

Soohyun Nam Liao, Daniel Zingaro, Michael A Laurenzano, William G Griswold, and Leo Porter. 2016. Lightweight, Early Identification of At-Risk CS1 Students. In Proc. of the 2016 ACM Conference on International Computing Education Research. ACM, 123--131.

Digital Library

[31]

Leah P Macfadyen and Shane Dawson. 2010. Mining LMS data to develop an "early warning system" for educators: A proof of concept. Computers & education 54, 2 (2010), 588--599.

Digital Library

[32]

Christoph Mengelkamp and Maria Bannert. 2010. Accuracy of confidence judgments: Stability and generality in the learning process and predictive validity for learning outcome. Memory & cognition 38, 4 (2010), 441--451.

[33]

Craig S. Miller and Amber Settle. 2016. Some Trouble with Transparency: An Analysis of Student Errors with Object-oriented Python. In Proc. of the 2016 ACM Conf. on International Computing Education Research (ICER '16). ACM, 133--141.

Digital Library

[34]

Behrouz Minaei-Bidgoli, Deborah A Kashy, Gerd Kortemeyer, and William F Punch. 2003. Predicting student performance: an application of data mining methods with an educational web-based system. In Frontiers in education, 2003. FIE 2003 33rd annual, Vol. 1. IEEE, T2A--13.

[35]

Andrew Petersen, Jaime Spacco, and Arto Vihavainen. 2015. An exploration of error quotient in multiple contexts. In Proc. of the 15th Koli Calling Conference on Computing Education Research. ACM, 77--86.

Digital Library

[36]

Chris Piech, Mehran Sahami, Daphne Koller, Steve Cooper, and Paulo Blikstein. 2012. Modeling how students learn to program. In Proc. of the 43rd ACM technical symposium on Computer Science Education. ACM, 153--160.

Digital Library

[37]

Leo Porter, Daniel Zingaro, and Raymond Lister. 2014. Predicting student success using fine grain clicker data. In Proc. of the 10th annual conference on International computing education research. ACM, 51--58.

Digital Library

[38]

Steven B Robbins, Kristy Lauver, Huy Le, Daniel Davis, Ronelle Langley, and Aaron Carlstrom. 2004. Do psychosocial and study skill factors predict college outcomes? A meta-analysis. (2004).

[39]

Cristobal Romero, Pedro G. Espejo, Amelia Zafra, Jose Raul Romero, and Sebastian Ventura. 2013. Web usage mining for predicting final marks of students that use Moodle courses. Computer Applications in Engineering Education 21, 1 (2013), 135--146.

[40]

Dale H Schunk. 1981. Modeling and attributional effects on children's achievement: A self-efficacy analysis. J. of educational psychology 73, 1 (1981), 93.

[41]

Alex J Smola and Bernhard Schölkopf. 2004. A tutorial on support vector regression. Statistics and computing 14, 3 (2004), 199--222.

Digital Library

[42]

Elliot Soloway and Kate Ehrlich. 1984. Empirical studies of programming knowledge. IEEE Transactions on software engineering 5 (1984), 595--609.

Digital Library

[43]

Arto Vihavainen. 2013. Predicting students' performance in an introductory programming course using data from students' own programming process. In 13th Int. Conf. on Advanced Learning Technologies (ICALT). IEEE, 498--499.

Digital Library

[44]

Christopher Watson, Frederick WB Li, and Jamie L Godwin. 2013. Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In Advanced Learning Technologies (ICALT), 2013 IEEE 13th International Conference on. IEEE, 319--323.

Digital Library

[45]

Susan Wiedenbeck, Deborah Labelle, and Vennila NR Kain. 2004. Factors affecting course outcomes in introductory programming. In 16th Annual Workshop of the Psychology of Programming Interest Group. 97--109.

[46]

David H Wolpert, William G Macready, and others. 1995. No free lunch theorems for search. Technical Report. Technical Report SFI-TR-95-02-010, Santa Fe Institute.

Cited By

Rys-Recker NHao Q(2024)Early Identification of Struggling Students in Large Computer Science Courses: A Replication Study2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00022(88-93)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00022
Dol SJawandhiya P(2024)Systematic Review and Analysis of EDM for Predicting the Academic Performance of StudentsJournal of The Institution of Engineers (India): Series B10.1007/s40031-024-00998-0105:4(1021-1071)Online publication date: 4-Feb-2024
https://doi.org/10.1007/s40031-024-00998-0
Ahmed MRony RHadi MHossain EAhmed N(2023)A Minimalistic Approach to Predict and Understand the Relation of App Usage with Students' Academic PerformanceProceedings of the ACM on Human-Computer Interaction10.1145/36042407:MHCI(1-28)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604240
Show More Cited By

Index Terms

Predicting Academic Success Based on Learning Material Usage

Recommendations

Predicting Student Academic Performance Using Machine Learning
Computational Science and Its Applications – ICCSA 2021
Abstract
The introduction of the Internet of Things (IoT), Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL), and Big Data have paved the way for research focused on improving the student learning experience and help to address ...
Do Open Educational Resources and Cloud Classroom Really Improve Students' Learning?

More and more educational institutions are using educational technologies and online learning materials to help students achieve satisfactory learning effects. However, not all teachers are able to prepare and design digital learning materials for ...
Predicting Academic Success for Business and Computing Students

Various means to predict the success rate of students have been introduced by a number of educational institutions worldwide. The aim of this research was to identify predictors of success for tertiary education students. Participants were 353 students ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGITE '17: Proceedings of the 18th Annual Conference on Information Technology Education

September 2017

202 pages

ISBN:9781450351003

DOI:10.1145/3125659

Conference Chair:
Stephen Zilora
Rochester Institute of Technology, USA
,
Program Chairs:
Thomas Ayers
Broward College, USA
,
Daniel Bogaard
Rochester Institute of Technology, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGITE: ACM Special Interest Group on Information Technology Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Suomen Akatemia

Conference

SIGITE/RIIT 2017

Sponsor:

SIGITE

SIGITE/RIIT 2017: The 18th Annual Conference on Information Technology Education and the 6th Annual Conference on Research in Information Technology

October 4 - 7, 2017

New York, Rochester, USA

Acceptance Rates

SIGITE '17 Paper Acceptance Rate 23 of 58 submissions, 40%;

Overall Acceptance Rate 176 of 429 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
336
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)2

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rys-Recker NHao Q(2024)Early Identification of Struggling Students in Large Computer Science Courses: A Replication Study2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00022(88-93)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00022
Dol SJawandhiya P(2024)Systematic Review and Analysis of EDM for Predicting the Academic Performance of StudentsJournal of The Institution of Engineers (India): Series B10.1007/s40031-024-00998-0105:4(1021-1071)Online publication date: 4-Feb-2024
https://doi.org/10.1007/s40031-024-00998-0
Ahmed MRony RHadi MHossain EAhmed N(2023)A Minimalistic Approach to Predict and Understand the Relation of App Usage with Students' Academic PerformanceProceedings of the ACM on Human-Computer Interaction10.1145/36042407:MHCI(1-28)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604240
Dol SJawandhiya P(2023)Classification Technique and its Combination with Clustering and Association Rule Mining in Educational Data Mining — A surveyEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106071122(106071)Online publication date: Jun-2023
https://doi.org/10.1016/j.engappai.2023.106071
Koutcheme CSarsa SHellas AHaaranen LLeinonen J(2022)Methodological Considerations for Predicting At-risk StudentsProceedings of the 24th Australasian Computing Education Conference10.1145/3511861.3511873(105-113)Online publication date: 14-Feb-2022
https://dl.acm.org/doi/10.1145/3511861.3511873
Skalka JDrlik M(2020)Automated Assessment and Microlearning Units as Predictors of At-Risk Students and Students’ Outcomes in the Introductory Programming CoursesApplied Sciences10.3390/app1013456610:13(4566)Online publication date: 30-Jun-2020
https://doi.org/10.3390/app10134566
O'Malley CAggarwal ALuxton-Reilly ASzabo C(2020)Evaluating the Use and Effectiveness of Ungraded Practice Problems in an Introductory Programming CourseProceedings of the Twenty-Second Australasian Computing Education Conference10.1145/3373165.3373185(177-184)Online publication date: 3-Feb-2020
https://dl.acm.org/doi/10.1145/3373165.3373185
Buschetto Macarini LCechinel CBatista Machado MFaria Culmant Ramos VMunoz R(2019)Predicting Students Success in Blended Learning—Evaluating Different Interactions Inside Learning Management SystemsApplied Sciences10.3390/app92455239:24(5523)Online publication date: 15-Dec-2019
https://doi.org/10.3390/app9245523
Liao SZingaro DAlvarado CGriswold WPorter LHawthorne EPérez-Quiñones MHeckman SZhang J(2019)Exploring the Value of Different Data Sources for Predicting Student Performance in Multiple CS CoursesProceedings of the 50th ACM Technical Symposium on Computer Science Education10.1145/3287324.3287407(112-118)Online publication date: 22-Feb-2019
https://dl.acm.org/doi/10.1145/3287324.3287407
Hellas AIhantola PPetersen AAjanovski VGutica MHynninen TKnutas ALeinonen JMessom CLiao SRößling GScharlau B(2018)Predicting academic performance: a systematic literature reviewProceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education10.1145/3293881.3295783(175-199)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3293881.3295783

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents