short-paper

From prediction to impact: evaluation of a learning analytics retention program

Authors:

Jelena Jovanovic,

Dragan Gašević,

Abelardo PardoAuthors Info & Claims

LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference

Pages 474 - 478

https://doi.org/10.1145/3027385.3027405

Published: 13 March 2017 Publication History

Abstract

Learning analytics research has often been touted as a means to address concerns regarding student retention outcomes. However, few research studies to date, have examined the impact of the implemented intervention strategies designed to address such retention challenges. Moreover, the methodological rigor of some of the existing studies has been challenged. This study evaluates the impact of a pilot retention program. The study contrasts the findings obtained by the use of different methods for analysis of the effect of the intervention. The pilot study was undertaken between 2012 and 2014 resulting in a combined enrolment of 11,160 students. A model to predict attrition was developed, drawing on data from student information system, learning management system interactions, and assessment. The predictive model identified some 1868 students as academically at-risk. Early interventions were implemented involving learning and remediation support. Common statistical methods demonstrated a positive association between the intervention and student retention. However, the effect size was low. The use of more advanced statistical methods, specifically mixed-effect methods explained higher variability in the data (over 99%), yet found the intervention had no effect on the retention outcomes. The study demonstrates that more data about individual differences is required to not only explain retention but to also develop more effective intervention approaches.

References

[1]

Arnold, K.E., 2010. Signals: Applying Academic Analytics. In EDUCAUSE Quarterly Magazine EDUCAUSE.

[2]

Arnold, K.E. and Pistilli, M., 2012. Course Signals at Purdue: Using Learning Analytics to Increase Student Success In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (Vancouver, British Columbia, Canada2012), ACM, 267--270.

Digital Library

[3]

Baker, R.S.J.D. and Siemens, G., 2014. Educational data mining and learning analytics. In Cambridge Handbook of the Learning Sciences (2nd Edition), K. Sawyer Ed. Cambridge University, New York.

[4]

Barber, R. and Sharkey, M., 2012. Course correction: Using analytics to predict course success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK'12) (Vancouver, Canada2012), ACM, New York, USA, 259--262.

Digital Library

[5]

Bates, D., Maechler, M., Bolker, B., and Walker, S., 2014. lme4: Linear mixed-effects models using Eigen and S4. R package version 1, 7.

[6]

Brooman, S. and Darwent, S., 2014. Measuring the beginning: a quantitative study of the transition to higher education. Studies in Higher Education 39, 9, 1523--1541.

[7]

Campbell, J., 2007. Utilizing student data within the course management system to determine undergraduate student academic success: An exploratory study Purdue University.

[8]

Caulfield, M., 2013. What the Course Signals "Kerfuffle" is About, and What it Means to You. Educause, http://www.educause.edu/blogs/mcaulfield/what-course-signals-kerfuffle-about-and-what-it-means-you.

[9]

Clow, D., Ferguson, R., Macfadyen, L., Prinsloo, P., and Slade, S., 2016. LAK Failathon. In Sixth International Conference on Learning Analytics & Knowledge, S. Dawson, H. Drachsler and C. Rose Eds. ACM, Edinburgh, Scotland, 509--511.

Digital Library

[10]

Gašević, D., Dawson, S., Rogers, T., and Gasevic, D., 2016. Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education 28, 68--84.

[11]

Gilardi, S. and Guglielmetti, C., 2011. University life of non-traditional students: Engagement styles and impact on attrition. The Journal of Higher Education 82, 1, 33--53.

[12]

Hattie, J. and Timperley, H., 2007. The power of feedback. Review of Educational Research 77, 1, 81--112.

[13]

Hayes, A.F., 2006. A Primer on Multilevel Modeling. Human Communication Research 32, 4, 385--410.

[14]

Jayaprakash, S.M., Moody, E.W., Lauría, E., Regan, J.R., and Baron, J.D., 2014. Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative Journal of Learning Analytics 1, 1, 6--47.

[15]

Kahu, E.R., 2013. Framing student engagement in higher education. Studies in Higher Education 38, 5, 758--773.

[16]

Kift, S., Nelson, K., and Clarke, J., 2010. Transition Pedagogy: A third generation approach to FYE - A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education 1, 1, 1--20.

[17]

Kovanovic, V., Gasevic, D., Dawson, S., Joksimovic, S., Baker, R.S., and Hatala, M., 2015. Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics 2, 3, 81--110.

[18]

Kovanovic, V., Gasevic, D., Dawson, S., Joksimovic, S., Baker, R.S., and Hatala, M., 2015. Penetrating the black box of time-on-task estimation. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge ACM, Poughkeepsie, New York, 184--193.

Digital Library

[19]

Krumm, A.E., Waddington, R.J., Teasley, S.D., and Lonn, S., 2014. A Learning Management System-Based Early Warning System for Academic Advising in Undergraduate Engineering. In Learning Analytics: From Research to Practice, J.A. Larusson and B. White Eds. Springer Science+Business Media, New York, USA, 103--119.

[20]

Laidlaw, A., Mclellan, J., and Ozakinci, G., 2015. Understanding undergraduate student perceptions of mental health, mental well-being and help-seeking behaviour. Studies in Higher Education Published online, 1--13.

[21]

Macfadyen, L. and Dawson, S., 2010. Mining LMS data to develop an "early warning system" for educators: A proof of concept. Computers & Education 54, 2, 588--599.

Digital Library

[22]

Morris, L.V., Finnegan, C., and Wu, S., 2005. Tracking student behavior, persistence, and achievement in online courses. Internet and Higher Education 8, 3, 221--231.

[23]

Nakagawa, S. and Schielzeth, H., 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. . Methods in Ecology and Evolution 4, 2, 133--142.

[24]

Nichols, M., 2010. Student perceptions of support services and the influence of targeted interventions on retention in distance education. Distance Education 31, 1, 93--113.

[25]

Pardo, A. and Dawson, S., 2016. Learning Analytics. How can Data be used to Improve Learning Practice?. In Measuring and visualising learning in the information-rich classroom, P. Reimann, S. Bull, M. Kickmeier-Rust, R. Vatrapu and B. Wasson Eds. Routledge, USA, 41--55.

[26]

Pardo, A., Mirriahi, N., Martinez-Maldonado, R., Jovanovic, J., Dawson, S., and Gašević, D., 2016. Generating actionable predictive models of academic performance. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (Edinburgh, UK2016), ACM, New York, USA, 474--478.

Digital Library

[27]

Romero, C., Espejo, P.G., Zafra, A., Romero, J.R., and Ventura, S., 2013. Web usage mining for predicting final marks of students that use Moodle courses. Comput. Appl. Eng. Educ 21, 135--146.

[28]

Romero, C., López, M.-I., Luna, J.-M., and Ventura, S., 2013. Predicting students' final performance from participation in on-line discussion forums. Computers & Education 68(10//), 458--472.

[29]

Romero, C. and Ventura, S., 2010. Educational Data Mining: A Review of the State of the Art. IEEE Transactions on Systems, Man and Cubernetics- PartC: Applications and Reviews 40, 6, 601--618.

Digital Library

[30]

Siemens, G., Dawson, S., and Lynch, G., 2013. Improving the Productivity of the Higher Education Sector: Policy and Strategy for Systems-Level Deployment of Learning Analytics. Society for Learning Analytics Research; and the Australian Government Office for Learning and Teaching.

[31]

Wolff, A., Zdrahal, Z., Nikolov, A., and Pantucek, M., 2013. Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment. In Proceedings of the Proceedings of the Third International Conference on Learning Analytics and Knowledge (Leuven, Belgium2013), ACM, New York, USA, 145--149.

Digital Library

[32]

Wright, M.C., Mckay, T., Hershock, C., Miller, K., and Tritz, J., 2014. Better Than Expected: Using Learning Analytics to Promote Student Success in Gateway Science. Change: The Magazine of Higher Learning 46, 1, 28--34.

[33]

Zaitseva, E., Milsom, C., and Stewart, M., 2013. Connecting the dots: Using concept maps for interpreting student satisfaction. Quality in Higher Education 19, 2, 225--247.

Cited By

Knox J(2024)Learning with MachinesTranslations and Participation10.14361/9783839471005-006(133-156)Online publication date: 5-Feb-2024
https://doi.org/10.14361/9783839471005-006
Alalawi KAthauda RChiong R(2024)An Extended Learning Analytics Framework Integrating Machine Learning and Pedagogical Approaches for Student Performance Prediction and InterventionInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00429-7Online publication date: 17-Sep-2024
https://doi.org/10.1007/s40593-024-00429-7
da Silveira Fleck AZhu T(2024)Impact of Embedded Learning Strategy Activities: Student Engagement and PerformanceTechnology, Knowledge and Learning10.1007/s10758-023-09715-029:3(1475-1498)Online publication date: 9-Jan-2024
https://doi.org/10.1007/s10758-023-09715-0
Show More Cited By

Index Terms

From prediction to impact: evaluation of a learning analytics retention program
1. Applied computing
  1. Education
  2. Enterprise computing
2. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

The combined effect of self-efficacy and academic integration on higher education students studying IT majors in Taiwan

The purpose of this study is to examine the combined effect of self-efficacy and academic integration on higher education students studying IT (Information Technology) majors in Taiwan. We introduced self-efficacy, which is a psychological factor that ...
An Improved Grade Point Average, With Applications to CS Undergraduate Education Analytics
Special Issue on Learning Analytics and Regular Papers

We present a methodological improvement for calculating Grade Point Averages (GPAs). Heterogeneity in grading between courses systematically biases observed GPAs for individual students: the GPA observed depends on course selection. We show how a ...
Exploring student predictive model that relies on institutional databases and open data instead of traditional questionnaires
WWW '13 Companion: Proceedings of the 22nd International Conference on World Wide Web

Research in student retention and progression to completion is traditionally survey-based, where researchers collect data through questionnaires and interviewing students. The major issues with survey-based study are the potentially low response rates ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge Conference

March 2017

631 pages

ISBN:9781450348706

DOI:10.1145/3027385

General Chairs:
Alyssa Wise
New York University, USA
,
Phillip H. Winne
Simon Fraser University, Canada
,
Grace Lynch
Society for Learning Analytics Research
,
Program Chairs:
Xavier Ochoa
Escuela Superior Politecnica del Litoral, Ecuador
,
Inge Molenaar
Radboud University, The Netherlands
,
Shane Dawson
University of South Australia, Australia
,
Publications Chair:
Marek Hatala
Simon Fraser University, Canada

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

LAK '17

LAK '17: 7th International Learning Analytics and Knowledge Conference

March 13 - 17, 2017

British Columbia, Vancouver, Canada

Acceptance Rates

LAK '17 Paper Acceptance Rate 36 of 114 submissions, 32%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
1,050
Total Downloads

Downloads (Last 12 months)91
Downloads (Last 6 weeks)5

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Knox J(2024)Learning with MachinesTranslations and Participation10.14361/9783839471005-006(133-156)Online publication date: 5-Feb-2024
https://doi.org/10.14361/9783839471005-006
Alalawi KAthauda RChiong R(2024)An Extended Learning Analytics Framework Integrating Machine Learning and Pedagogical Approaches for Student Performance Prediction and InterventionInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00429-7Online publication date: 17-Sep-2024
https://doi.org/10.1007/s40593-024-00429-7
da Silveira Fleck AZhu T(2024)Impact of Embedded Learning Strategy Activities: Student Engagement and PerformanceTechnology, Knowledge and Learning10.1007/s10758-023-09715-029:3(1475-1498)Online publication date: 9-Jan-2024
https://doi.org/10.1007/s10758-023-09715-0
Heilala VJääskelä PSaarela MKärkkäinen T(2023)Adapting Teaching and Learning in Higher Education Using Explainable Student Agency AnalyticsPrinciples and Applications of Adaptive Artificial Intelligence10.4018/979-8-3693-0230-9.ch002(20-51)Online publication date: 29-Dec-2023
https://doi.org/10.4018/979-8-3693-0230-9.ch002
Bernacki M(2023)Development, Sustainment, and Scaling of Self-Regulated Learning AnalyticsSupporting Self-Regulated Learning and Student Success in Online Courses10.4018/978-1-6684-6500-4.ch012(255-281)Online publication date: 24-Feb-2023
https://doi.org/10.4018/978-1-6684-6500-4.ch012
Han F(2023)Relations between Students’ Study Approaches, Perceptions of the Learning Environment, and Academic Achievement in Flipped Classroom Learning: Evidence from Self-Reported and Process DataJournal of Educational Computing Research10.1177/0735633123116282361:6(1252-1274)Online publication date: 8-May-2023
https://doi.org/10.1177/07356331231162823
Bond MViberg OBergdahl NHilliger IKhosravi HRienties BDawson S(2023)The current state of using learning analytics to measure and support K-12 student engagement: A scoping reviewLAK23: 13th International Learning Analytics and Knowledge Conference10.1145/3576050.3576085(240-249)Online publication date: 13-Mar-2023
https://dl.acm.org/doi/10.1145/3576050.3576085
McConvey KGuha SKuzminykh A(2023)A Human-Centered Review of Algorithms in Decision-Making in Higher EducationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580658(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580658
Liu LWang SBritton TAbebe R(2023)Reimagining the machine learning life cycle to improve educational outcomes of studentsProceedings of the National Academy of Sciences10.1073/pnas.2204781120120:9Online publication date: 24-Feb-2023
https://doi.org/10.1073/pnas.2204781120
Matz SBukow CPeters HDeacons CDinu AStachl C(2023)Using machine learning to predict student retention from socio-demographic characteristics and app-based engagement metricsScientific Reports10.1038/s41598-023-32484-w13:1Online publication date: 7-Apr-2023
https://doi.org/10.1038/s41598-023-32484-w
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents