research-article

Heterogeneity of Treatment Effects of a Video Recommendation System for Algebra

Authors:

Walter L. Leite,

Nilanjana Chakraborty,

George Michailidis,

Sidney D'Mello,

Wanli XingAuthors Info & Claims

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

Pages 12 - 23

https://doi.org/10.1145/3491140.3528275

Published: 01 June 2022 Publication History

Abstract

Previous research has shown that providing video recommendations to students in virtual learning environments implemented at scale positively affects student achievement. However, it is also critical to evaluate whether the treatment effects are heterogeneous, and whether they depend on contextual variables such as disadvantaged student status and characteristics of the school settings. The current study extends the evaluation of a novel video recommendation system by performing an exploratory search for sources of heterogeneity of treatment effects. This study's design is a multi-site randomized controlled trial with an assignment at the student level across three large and diverse school districts in the southeast United States. The study occurred in Spring 2021, when some students were in regular classrooms and others in online classrooms. The results of the current study replicate positive effects found in a previous field experiment that occurred in Spring 2020, at the onset of the COVID-19 pandemic. Then, causal forests were used to investigate the heterogeneity of treatment effects. This study contributes to the literature on content sequencing systems and recommendation systems by showing how these systems can disproportionally benefit the groups of students who had higher levels of previous algebra ability, followed more recommendations, learned remotely, were Hispanic, and received free or reduced-price lunch, which has implications for the fairness of implementation of educational technology solutions.

References

[1]

Bernacki, M.L., M.J. Greene, and N.G. Lobczowski, A Systematic Review of Research on Personalized Learning: Personalized by Whom, to What, How, and for What Purpose(s)? Educational Psychology Review, 2021. 33(4): p. 1675--1715.

[2]

Plass, J.L. and S. Pawar, Toward a taxonomy of adaptivity for learning. Journal of Research on Technology in Education, 2020. 52(3): p. 275--300.

[3]

Zhang, L., J.D. Basham, and S. Yang, Understanding the implementation of personalized learning: A research synthesis. Educational Research Review, 2020: p. 100339.

[4]

Doroudi, S., V. Aleven, and E. Brunskill, Where's the reward? International Journal of Artificial Intelligence in Education, 2019. 29(4): p. 568--620.

[5]

Kizilcec, R.F. and H. Lee Algorithmic fairness in education. 2021.

[6]

Sies, A., K. Demyttenaere, and I. Van Mechelen, Studying treatment-effect heterogeneity in precision medicine through induced subgroups. J Biopharm Stat, 2019. 29(3): p. 491--507.

[7]

Lipkovich, I., A. Dmitrienko, and S. B. R. D' Agostino, Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials. Stat Med, 2017. 36(1): p. 136--196.

[8]

Kulik, J.A. and J.D. Fletcher, Effectiveness of Intelligent Tutoring Systems. Review of Educational Research, 2016. 86(1): p. 42--78.

[9]

Ma, W., et al., Intelligent tutoring systems and learning outcomes: A meta- analysis. Journal of Educational Psychology, 2014. 106(4): p. 901--918.

[10]

Steenbergen-Hu, S. and H. Cooper, A meta-analysis of the effectiveness of intelligent tutoring systems on college students' academic learning. Journal of Educational Psychology, 2014. 106(2): p. 331--347.

[11]

Lastinger Center for Learning and University of Florida. Algebra Nation. 2019 [cited 9/20/2019]; Available from: http://lastingercenter.com/portfolio/algebra-nation-2/.

[12]

Athey, S. and S. Wager, Estimating Treatment Effects with Causal Forests: An Application. Observational Studies, 2019. 5(2): p. 37--51.

[13]

Wager, S. and S. Athey, Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association, 2018. 113(523): p. 1228--1242.

[14]

Rubin, D.B., Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 1974. 66: p. 688--701.

[15]

Leite, W.L., et al. A novel video recommendation system for algebra: An effectiveness evaluation study. in LAK22: 12th International Learning Analytics and Knowledge Conference (LAK22). 2022. Online, USA: ACM, New York, NY, USA.

Digital Library

[16]

VanLehn, K., The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist, 2011. 46(4): p. 197--221.

[17]

Xu, Z., et al., The effectiveness of intelligent tutoring systems on K-12 students' reading comprehension: A meta-analysis. British Journal of Educational Technology, 2019. 50(6): p. 3119--3137.

[18]

Steenbergen-Hu, S. and H. Cooper, A meta-analysis of the effectiveness of intelligent tutoring systems on K--12 students' mathematical learning. Journal of Educational Psychology, 2013. 105(4): p. 970--987.

[19]

Dong, N., et al., Power analyses for moderator effects with (non)random slopes in cluster randomized trials. Methodology, 2021. 17(2): p. 92--110.

[20]

Raudenbush, S.W. and H.S. Bloom, Learning About and From a Distribution of Program Impacts Using Multisite Trials. American Journal of Evaluation, 2015. 36(4): p. 475--499.

[21]

Ding, P., A. Feller, and L. Miratrix, Randomization inference for treatment effect variation. Journal of the Royal Statistical Society: Series B: Statistical Methodology, 2016. 78(3): p. 655--671.

[22]

Ding, P., A. Feller, and L. Miratrix, Decomposing Treatment Effect Variation. Journal of the American Statistical Association, 2018. 114(525): p. 304--317.

[23]

Athey, S. and G. Imbens, Recursive partitioning for heterogeneous causal effects. Proc Natl Acad Sci U S A, 2016. 113(27): p. 7353--60.

[24]

Anoke, S.C., S.L. Normand, and C.M. Zigler, Approaches to treatment effect heterogeneity in the presence of confounding. Stat Med, 2019. 38(15): p. 2797--2815.

[25]

ACM computing surveys. 1971, New York, N.Y.: Association for Computing Machinery.

[26]

Carnegie, N., V. Dorie, and J.L. Hill, Examining treatment effect heterogeneity using BART. Observational Studies, 2019. 5(2): p. 52--70.

[27]

Hill, J.L., Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 2011. 20(1): p. 217--240.

[28]

Mitten, C., Z.K. Collier, and W.L. Leite, Online Resources for Mathematics: Exploring the Relationship between Teacher Use and Student Performance. Investigations in Mathematics Learning, 2021: p. 1--18.

[29]

Lord, F. and M. Novick, Statistical theories of mental test scores. 1968, Reading, MA: Addison-Wesley.

[30]

Embretson, S.E. and S.P. Reise, Item response theory for psychologists. 2000, Mahwah, NJ: Lawrence Erlbaum Associates.

[31]

Hutt, S., J. Grafsgaard, and S.K. D'Mello, Time to Scale: Generalizable Affect Detection for Tens of Thousands of Students across An Entire School Year, in Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI 2019). 2019, ACM: New York.

Digital Library

[32]

Jensen, E., S. Hutt, and S.K. D'Mello, Generalizability of Sensor-Free Affect Detection Models in a Longitudinal Dataset of Tens of Thousands of Students, in The 12th International Conference on Educational Data Mining, M. Desmarais, et al., Editors. 2019. p. 324--329.

[33]

D'Mello, S.K., E. Dieterle, and A.L. Duckworth, Advanced, Analytic, Automated (AAA) Measurement of Engagement during Learning. Educational Psychologist, 2017. 52(2): p. 104--123.

[34]

Vygotsky, L.S., Mind in society. 1978, Cambridge: Harvard University Press.

[35]

Raudenbush, S.W., What Are Value-Added Models Estimating and What Does This Imply for Statistical Practice? Journal of Educational and Behavioral Statistics, 2004. 29(1): p. 121--129.

[36]

Ing, C.K. and T.L. Lai, A stepwise regression method and consistent model selection for high-dimensional sparse linear models. Statistica Sinica, 2011: p. 1473--1513.

[37]

Chen, Y., et al., Recommendation System for Adaptive Learning. Appl Psychol Meas, 2018. 42(1): p. 24--41.

[38]

Mu, T., et al., Combining adaptivity with progression ordering for intelligent tutoring systems, in Proceedings of the Fifth Annual ACM Conference on Learning at Scale. 2018. p. 1--4.

Digital Library

[39]

Hsu, H.-L., C.-K. Ing, and T.L. Lai, Analysis of High-Dimensional Regression Models Using Orthogonal Greedy Algorithms, in Handbook of Big Data Analytics. 2018. p. 263--283.

[40]

van Buuren, S. and K. Groothuis-Oudshoorn, mice: multivariate imputation by chained equations in R. Journal of Statistical Software, 2011. 45(3): p. 1--67.

[41]

Kleinke, K., Multiple Imputation Under Violated Distributional Assumptions: A Systematic Evaluation of the Assumed Robustness of Predictive Mean Matching. Journal of Educational and Behavioral Statistics, 2017. 42(4): p. 371--404.

[42]

Raudenbush, S.W. and X. Liu, Statistical power and optimal design for multisite randomized trials. Psychological Methods, 2000. 5(2): p. 199--213.

[43]

Bates, D.M., et al., Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 2015. 67(1): p. 1--48.

[44]

R Development Core Team, R: A language and environment for statistical computing. 2022, R Foundation for Statistical Computing: Vienna, Austria.

[45]

Rubin, D.B., Multiple Imputation for Nonresponse in Surveys. 1987, New York: Wiley.

[46]

Hedges, L.V., Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 2007. 32(4): p. 341--370.

[47]

Tibshirani, J., et al., grf: Generalized Random Forests. R package version 2.0.2. 2021.

[48]

Yan, L., et al., Students' experience of online learning during the COVID-19 pandemic: A province-wide survey study. Br J Educ Technol, 2021.

[49]

Mahmood, S., Instructional strategies for online teaching in COVID-19 pandemic. Human Behavior and Emerging Technologies, 2021. 3(1): p. 199--203.

[50]

Holstein, K., Designing Real-time Teacher Augmentation to Combine Strengths of Human and AI Instruction, in School of Computer Science. 2019, Carnegie Mellon University: Pittsburgh, Pennsylvania.

[51]

Segedy, J., B. Sulcer, and G. Biswas, Are ILEs ready for the classroom? Bringing teachers into the feedback loop. Intelligent Tutoring Systems, 2010: p. 405--407.

[52]

Carrillo, C. and M.A. Flores, COVID-19 and teacher education: a literature review of online teaching and learning practices European Journal of Teacher Education, 2020. 43(4): p. 466--487.

[53]

Hensley, L.C., R. Iaconelli, and C.A. Wolters, "This weird time we're in": How a sudden change to remote education impacted college students' self-regulated learning. Journal of Research on Technology in Education, 2021: p. 1--16.

[54]

Fong, C.J., Academic motivation in a pandemic context: a conceptual review of prominent theories and an integrative model. Educational Psychology, 2022: p. 1--19.

[55]

Broadbent, J. and W.L. Poon, Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education, 2015. 27: p. 1--13.

[56]

Dillenbourg, P., Design for classroom orchestration. Computers & Education, 2013. 69: p. 485--492.

[57]

Drijvers, P., et al., The teacher and the tool: instrumental orchestrations in the technology-rich mathematics classroom. Educational Studies in Mathematics, 2010. 75(2): p. 213--234.

[58]

Prieto, L.P., et al., Orchestrating technology enhanced learning: a literature review and a conceptual framework. International Journal of Technology Enhanced Learning, 2011. 3(6): p. 583--598.

[59]

Wang, C.H., D.M. Shannon, and M.E. Ross, Students' characteristics, self-regulated learning, technology self-efficacy, and course outcomes in online learning. Distance Education, 2013. 34(3): p. 302--323.

[60]

Alemayehu, L., & Chen, H. L., The influence of motivation on learning engagement: the mediating role of learning self-efficacy and self-monitoring in online learning environments. Interactive Learning Environments, 2021: p. 1--14.

[61]

Olaussen, B.S. and I. Bråten, Students' Use of Strategies for Self-regulated Learning: cross-cultural perspectives. Scandinavian journal of educational research, 1999. 43(4): p. 409--432.

[62]

Wan, Z., D. Compeau, and N. Haggerty, The effects of self-regulated learning processes on e-learning outcomes in organizational settings. Journal of Management Information Systems, 2012. 29(1): p. 307--340.

[63]

Leite, W. L., et al. Teacher Strategies to Use Virtual Learning Environments to Facilitate Algebra Learning During School Closures. 2022.

[64]

Carvalho, C., et al., Assessing Treatment Effect Variation in Observational Studies: Results from a Data Challenge. Observational Studies, 2019. 5(2): p. 21--35.

[65]

Schochet, P.Z. and H.S. Chiang, Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs. Journal of Educational and Behavioral Statistics, 2011. 36(3): p. 307--345.

[66]

Major, L., G.A. Francis, and M. Tsapali, The effectiveness of technology'supported personalised learning in low- and middle-income countries: A meta- analysis. British Journal of Educational Technology, 2021. 52(5): p. 1935--1964.

[67]

Holmes, W., et al., Technology-enhanced personalised learning: Untangling the evidence. 2018, Robert Bosch Stiftung GmbH: Stuttgart, Germany.

[68]

FitzGerald, E., et al., A literature synthesis of personalised technology-enhanced learning: what works and why. Research in Learning Technology, 2018. 26(2095): p. 1--16.

Cited By

Li YXie TTang Y(2024)A Review of Recommender Systems Based on Video Learning Resources in Online Education2024 International Symposium on Educational Technology (ISET)10.1109/ISET61814.2024.00031(114-118)Online publication date: 29-Jul-2024
https://doi.org/10.1109/ISET61814.2024.00031
Leite WHatch AKuang HCavanaugh CXing WSpikol DViberg OMartínez-Monés AGuo P(2023)How Teachers Influence Student Adoption and Effectiveness of a Recommendation System for AlgebraProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593394(156-163)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593394
Nie AReuel ABrunskill E(2023)Understanding the Impact of Reinforcement Learning Personalization on Subgroups of Students in Math TutoringArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky10.1007/978-3-031-36336-8_106(688-694)Online publication date: 30-Jun-2023
https://doi.org/10.1007/978-3-031-36336-8_106

Index Terms

Heterogeneity of Treatment Effects of a Video Recommendation System for Algebra
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Online learning settings
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

June 2022

491 pages

ISBN:9781450391580

DOI:10.1145/3491140

General Chair:
René F. Kizilcec
Cornell University, USA
,
Program Chairs:
Katie Davis
University of Washington, USA
,
Xavier Ochoa
New York University, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Institute of Education Sciences

Conference

L@S '22

L@S '22: Ninth (2022) ACM Conference on Learning @ Scale

June 1 - 3, 2022

NY, New York City, USA

Acceptance Rates

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
310
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)8

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li YXie TTang Y(2024)A Review of Recommender Systems Based on Video Learning Resources in Online Education2024 International Symposium on Educational Technology (ISET)10.1109/ISET61814.2024.00031(114-118)Online publication date: 29-Jul-2024
https://doi.org/10.1109/ISET61814.2024.00031
Leite WHatch AKuang HCavanaugh CXing WSpikol DViberg OMartínez-Monés AGuo P(2023)How Teachers Influence Student Adoption and Effectiveness of a Recommendation System for AlgebraProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593394(156-163)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593394
Nie AReuel ABrunskill E(2023)Understanding the Impact of Reinforcement Learning Personalization on Subgroups of Students in Math TutoringArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky10.1007/978-3-031-36336-8_106(688-694)Online publication date: 30-Jun-2023
https://doi.org/10.1007/978-3-031-36336-8_106
Leite WKuang HJing ZXing WCavanaugh CHuggins-Manley A(2022)The relationship between self-regulated student use of a virtual learning environment for algebra and student achievementComputers & Education10.1016/j.compedu.2022.104615191:COnline publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.compedu.2022.104615

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents