research-article

Studying MOOC completion at scale using the MOOC replication framework

Authors:

Juan Miguel L. Andres,

Dragan Gašević,

George Siemens,

Scott A. Crossley,

Srećko JoksimovićAuthors Info & Claims

LAK '18: Proceedings of the 8th International Conference on Learning Analytics and Knowledge

Pages 71 - 78

https://doi.org/10.1145/3170358.3170369

Published: 07 March 2018 Publication History

Abstract

Research on learner behaviors and course completion within Massive Open Online Courses (MOOCs) has been mostly confined to single courses, making the findings difficult to generalize across different data sets and to assess which contexts and types of courses these findings apply to. This paper reports on the development of the MOOC Replication Framework (MORF), a framework that facilitates the replication of previously published findings across multiple data sets and the seamless integration of new findings as new research is conducted or new hypotheses are generated. In the proof of concept presented here, we use MORF to attempt to replicate 15 previously published findings across 29 iterations of 17 MOOCs. The findings indicate that 12 of the 15 findings replicated significantly across the data sets, and that two findings replicated significantly in the opposite direction. MORF enables larger-scale analysis of MOOC research questions than previously feasible, and enables researchers around the world to conduct analyses on huge multi-MOOC data sets without having to negotiate access to data.

References

[1]

Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The International Review of Research in Open and Distributed Learning, 15(1).

[2]

Adamopoulos, P. (2013). What makes a great MOOC? An interdisciplinary analysis of student retention in online courses.

[3]

Zhenghao, C., Alcorn, B., Christensen, G., Eriksson, N., Koller, D., & Emanuel, E. (2015). Who's Benefiting from MOOCs, and Why. Harvard Business Review

[4]

Clow, D. (2013). MOOCs and the funnel of participation. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 185--189). ACM

Digital Library

[5]

Yang, D., Sinha, T., Adamson, D., & Rose, C. P. (2013). Turn on, tune in, drop out: Anticipating student dropouts in massive open online courses. In Proceedings of the 2013 NIPS Data-driven education Workshop (Vol. 11, p. 14)

[6]

Andersson, U., Arvemo, T., & Gellerstedt, M. (2016). How well can completion of online courses be predicted using binary logistic regression?. In IRIS39-The 39th Information Systems Research Conference in Scandinavia, Ljungskile, Sweden, 7-10 August 2016.

[7]

Yang, D., Wen, M., Howley, I., Kraut, R., & Rose, C. (2015). Exploring the effect of confusion in discussion forums of massive open online courses. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale (pp. 121--130). ACM.

Digital Library

[8]

Łukasz, K., Sharma, K., Shirvani Boroujeni, M., & Dillenbourg, P. (2016). On generalizability of MOOC models. In Proceedings of the 9th International Conference on Educational Data Mining (No. EPFL-CONF-223613, pp. 406--411).

[9]

Andres, J.M.L., Baker, R.S., Siemens, G., Gašević, D., & Spann, C.A. (in press). Replicating 21 Findings on Student Success in Online Learning. Technology, Instruction, Cognition, & Learning.

[10]

Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty replication in the education sciences. Educational Researcher, 0013189X14545513.

[11]

Kim, J., Guo, P. J., Seaton, D. T., Mitros, P., Gajos, K. Z., & Miller, R. C. (2014, March). Understanding in-video dropouts and interaction peaks in online lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 31--40). ACM.

Digital Library

[12]

Whitehill, J., Williams, J. J., Lopez, G., Coleman, C. A., & Reich, J. (2015). Beyond prediction: First steps toward automatic intervention in MOOC student stopout.

[13]

edX. (2017). edX Research Guide. Retrieved from the edX website: http://edx.readthedocs.io/projects/devdata/en/latest/

[14]

Pardos, Z. A., & Kao, K. (2015, March). moocRP: An open-source analytics platform. In Proceedings of the Second (2015) ACM conference on learning@ scale (pp. 103--110). ACM.

Digital Library

[15]

Schmidt, F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings. Sage publications.

[16]

Koedinger, K. R., Baker, R. S., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining, 43.

[17]

Anderson, J. R., Matessa, M., & Lebiere, C. (1997). ACT-R: A theory of higher level cognition and its relation to visual attention. Human-Computer Interaction, 12(4), 439--462.

Digital Library

[18]

Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987). Soar: An architecture for general intelligence. Artificial intelligence, 33(1), 1--64.

Digital Library

[19]

Friedman-Hill, E. (2002). Jess, the expert system shell for the java platform.USA: Distributed Computing Systems.

[20]

Sinha, T., Jermann, P., Li, N., & Dillenbourg, P. (2014). Your click decides your fate: Inferring information processing and attrition behavior from MOOC video clickstream interactions. arXiv preprint arXiv:1407.7131.

[21]

Wang, Y. Baker, R. (2015) Content or Platform: Why do students complete MOOCs? MERLOT Journal of Online Learning and Teaching, 11 (1), 17--30.

[22]

DeBoer, J., Ho, A., Stump, G.S., Pritchard, D.E., Seaton, D. and Breslow, L. (2013). Bringing student backgrounds online: MOOC user demographics, site usage, and online learning. engineer, 2, pp.0--81.

[23]

Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining Associations between Sets of Items in Massive Databases. In Proceedings of the ACM-SIGMOD Int'l Conference on Management of Data (pp. 207--216).

Digital Library

[24]

Brin, S., Motwani, R., Ullman, J. D., & Tsur, S. (1997, June). Dynamic itemset counting and implication rules for market basket data. In ACM SIGMOD Record (Vol. 26, No. 2, pp. 255--264). ACM.

Digital Library

[25]

Stouffer, S.A., Suchman, E.A., DeVinney, L.C., Star, S.A. & Williams, R.M. Jr. (1949). The American Soldier, Vol. 1: Adjustment during Army Life. Princeton University Press, Princeton.

[26]

Crossley, S., McNamara, D. S., Baker, R., Wang, Y., Paquette, L., Barnes, T., & Bergner, Y. (2015). Language to Completion: Success in an Educational Data Mining Massive Open Online Class. International Educational Data Mining Society.

[27]

Ramesh, A., Goldwasser, D., Huang, B., Daumé III, H. & Getoor, L. (2013, December). Modeling learner engagement in MOOCs using probabilistic soft logic. In NIPS Workshop on Data Driven Education (Vol. 2, pp. 1--7).

[28]

Wen, M., Yang, D., & Rose, C. (2014, July). Sentiment Analysis in MOOC Discussion Forums: What does it tell us?. In Educational data mining 2014.

[29]

Wilkowski, J., Deutsch, A., & Russell, D. M. (2014, March). Student skill and goal achievement in the mapping with google MOOC. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 3--10). ACM.

Digital Library

[30]

Kyle, K., & Crossley, S. A. (2015). Automatically Assessing Lexical Sophistication: Indices, Tools, Findings, and Application. TESOL Quarterly, 49 (4), 757--786.

[31]

Crossley, S. A., Kyle, K., and McNamara, D. S. (2016). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods.

[32]

Thorndike, E. L., & Lorge, I. (1944). The teacher's word book of 30,000 words. New York, NY: Teachers College, Columbia University.

[33]

Brown, G. D. A. (1984). A frequency count of 190,000 words in the London- Lund Corpus of English Conversation. Behavior Research Methods, Instrumentation & Computers, 16, 502--532.

[34]

Svartvik, J., & Quirk, R. (1980). A corpus of English conversation. Lund, Sweden: Gleerup.

[35]

Kucera, H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.

[36]

BNC Consortium. (2007), British National Corpus, version 3 (BNC XML ed.). Retrieved from www.natcorp.ox.ac.uk

[37]

Brysbaert, M., & New, B. (2009). Moving beyond Kucera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Meth- ods, 41, 977--990.

[38]

Sharif, A., & Magrill, B. (2015). Discussion forums in MOOCs. International Journal of Learning, Teaching and Educational Research, 12(1).

[39]

Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching, 9(2), 149.

[40]

Comer, D., Baker, R., Wang, Y. (2015) Negativity in Massive Online Open Courses: Impacts on Learning and Teaching. InSight: A Journal of Scholarly Teaching, 10.

[41]

Wang, X., Yang, D., Wen, M., Koedinger, K., & Rosé, C. P. (2015). Investigating How Student's Cognitive Behavior in MOOC Discussion Forums Affect Learning Gains. International Educational Data Mining Society.

Cited By

Kwon CButler DUchidiuno JStamper JOgan A(2024)Investigating Demographics and Motivation in Engineering Education Using Radio and Phone-Based Educational TechnologiesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642221(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642221
Viktor P(2024)Analyzing the Relationship Between MOOC Family Systems and the Financial Status of Local College Students2024 IEEE 22nd World Symposium on Applied Machine Intelligence and Informatics (SAMI)10.1109/SAMI60510.2024.10432746(000121-000126)Online publication date: 25-Jan-2024
https://doi.org/10.1109/SAMI60510.2024.10432746
Gonçalves BGonçalves V(2023)Perspective Chapter: MOOC – A Decade Later! What Is the Current Situation in Teacher Education?Massive Open Online Courses - Current Practice and Future Trends10.5772/intechopen.1001117Online publication date: 17-Apr-2023
https://doi.org/10.5772/intechopen.1001117
Show More Cited By

Index Terms

Studying MOOC completion at scale using the MOOC replication framework
1. Applied computing
  1. Education

Recommendations

Replication in Massive Open Online Course Research Using the MOOC Replication Framework
Effects of In-Video Quizzes on MOOC Lecture Viewing
L@S '16: Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Online courses on sites such as Coursera use quizzes embedded inside lecture videos (in-video quizzes) to help learners test their understanding of the video. This paper analyzes how users interact with in-video quizzes, and how in-video quizzes ...
Temporal dynamics of MOOC learning trajectories
DATA '18: Proceedings of the First International Conference on Data Science, E-learning and Information Systems

Massive Open Online Courses (MOOCs) are a relatively new online learning phenomenon, whereby in 2017 more than 81 million learners have followed around 9,400 courses offered by more than 800 universities. Learners' retention has been one of the most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '18: Proceedings of the 8th International Conference on Learning Analytics and Knowledge

March 2018

489 pages

ISBN:9781450364003

DOI:10.1145/3170358

Conference Chairs:
Abelardo Pardo
The University of Sydney, AUS
,
Kathryn Bartimote-Aufflick
The University of Sydney, AUS
,
Grace Lynch
Society for Learning Analytics Research, AUS
,
Program Chairs:
Simon Buckingham Shum
University of Technology Sydney, AUS
,
Rebecca Ferguson
The Open University, UK
,
Agathe Merceron
Beuth University of Applied Sciences, GER
,
Xavier Ochoa
Escuela Superior Politécnica del Litoral, EC

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 March 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

LAK '18

LAK '18: International Conference on Learning Analytics and Knowledge

March 7 - 9, 2018

New South Wales, Sydney, Australia

Acceptance Rates

LAK '18 Paper Acceptance Rate 35 of 115 submissions, 30%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
471
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)8

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kwon CButler DUchidiuno JStamper JOgan A(2024)Investigating Demographics and Motivation in Engineering Education Using Radio and Phone-Based Educational TechnologiesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642221(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642221
Viktor P(2024)Analyzing the Relationship Between MOOC Family Systems and the Financial Status of Local College Students2024 IEEE 22nd World Symposium on Applied Machine Intelligence and Informatics (SAMI)10.1109/SAMI60510.2024.10432746(000121-000126)Online publication date: 25-Jan-2024
https://doi.org/10.1109/SAMI60510.2024.10432746
Gonçalves BGonçalves V(2023)Perspective Chapter: MOOC – A Decade Later! What Is the Current Situation in Teacher Education?Massive Open Online Courses - Current Practice and Future Trends10.5772/intechopen.1001117Online publication date: 17-Apr-2023
https://doi.org/10.5772/intechopen.1001117
Viktor PMolnár A(2023)Perceptions of MOOC Systems Among Domestic University Students at Different Levels2023 IEEE 17th International Symposium on Applied Computational Intelligence and Informatics (SACI)10.1109/SACI58269.2023.10158615(000333-000336)Online publication date: 23-May-2023
https://doi.org/10.1109/SACI58269.2023.10158615
Roy AYee MPerdue MStein JBell ACarter RMiyagawa SKizilcec RDavis KOchoa X(2022)How COVID-19 Affected Computer Science MOOC Learner Behavior and Achievements: A Demographic StudyProceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528328(345-349)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528328
Hutt SBaker RAshenafi MAndres‐Bray JBrooks C(2022) Controlled outputs, full data: A privacy‐protecting infrastructure for MOOC data British Journal of Educational Technology10.1111/bjet.1323153:4(756-775)Online publication date: 11-May-2022
https://doi.org/10.1111/bjet.13231
Saqr MJovanovic JViberg OGašević D(2022)Is there order in the mess? A single paper meta-analysis approach to identification of predictors of success in learning analyticsStudies in Higher Education10.1080/03075079.2022.206145047:12(2370-2391)Online publication date: 11-Apr-2022
https://doi.org/10.1080/03075079.2022.2061450
Fotso JBatchakui BNkambou ROkereke G(2022)Algorithms for the Development of Deep Learning Models for Classification and Prediction of Learner Behaviour in MOOCsArtificial Intelligence for Data Science in Theory and Practice10.1007/978-3-030-92245-0_3(41-73)Online publication date: 2022
https://doi.org/10.1007/978-3-030-92245-0_3
Chen XZou DXie HWang F(2021)Past, present, and future of smart learning: a topic-based bibliometric analysisInternational Journal of Educational Technology in Higher Education10.1186/s41239-020-00239-618:1Online publication date: 15-Jan-2021
https://doi.org/10.1186/s41239-020-00239-6
Brooks CQuintana RChoi HQuintana CNeCamp TGardner J(2021)Towards Culturally Relevant Personalization at Scale: Experiments with Data Science LearnersInternational Journal of Artificial Intelligence in Education10.1007/s40593-021-00262-231:3(516-537)Online publication date: 28-Jul-2021
https://doi.org/10.1007/s40593-021-00262-2
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents