Cited By
View all- Wang YLai YHuang X(2024)Innovations in Online Learning Analytics: A Review of Recent Research and Emerging TrendsIEEE Access10.1109/ACCESS.2024.349362112(166761-166775)Online publication date: 2024
SLADE: A Method for Designing Human-Centred Learning Analytics Systems
Authors: 
Riordan Alfredo, 
Vanessa Echeverria, Yueqiao Jin, Zachari Swiecki, 
Dragan Gašević, Roberto Martinez-MaldonadoAuthors Info & Claims
Riordan Alfredo, Vanessa Echeverria, Yueqiao Jin, Zachari Swiecki, Dragan Gašević, Roberto Martinez-MaldonadoAuthors Info & ClaimsPages 24 - 34
Abstract
There is a growing interest in creating Learning Analytics (LA) systems that incorporate student perspectives. Yet, many LA systems still lean towards a technology-centric approach, potentially overlooking human values and the necessity of human oversight in automation. Although some recent LA studies have adopted a human-centred design stance, there is still limited research on establishing safe, reliable, and trustworthy systems during the early stages of LA design. Drawing from a newly proposed framework for human-centred artificial intelligence, we introduce SLADE, a method for ideating and identifying features of human-centred LA systems that balance human control and computer automation. We illustrate SLADE’s application in designing LA systems to support collaborative learning in healthcare. Twenty-one third-year students participated in design sessions through SLADE’s four steps: i) identifying challenges and corresponding LA systems; ii) prioritising these LA systems; iii) ideating human control and automation features; and iv) refining features emphasising safety, reliability, and trustworthiness. Our results demonstrate SLADE’s potential to assist researchers and designers in: 1) aligning authentic student challenges with LA systems through both divergent ideation and convergent prioritisation; 2) understanding students’ perspectives on personal agency and delegation to teachers; and 3) fostering discussions about the safety, reliability, and trustworthiness of LA solutions.
References
[1]
R. D. Alfredo, L. Nie, P. Kennedy, T. Power, C. Hayes, H. Chen, C. McGregor, Z. Swiecki, D. Gašević, and R. Martinez-Maldonado. 2023. "That Student Should Be a Lion Tamer!" StressViz: Designing a Stress Analytics Dashboard for Teachers. In LAK23: 13th International Learning Analytics and Knowledge Conference(LAK2023). ACM, 57–67. https://doi.org/10.1145/3576050.3576058
[2]
A. S. Alzahrani, Y.-S. Tsai, N. Aljohani, E. Whitelock-Wainwright, and D. Gasevic. 2023. Do teaching staff trust stakeholders and tools in learning analytics? A mixed methods study. Educational technology research and development (2023), 1–31.
[3]
P. An, K. Holstein, B. d’Anjou, B. Eggen, and S. Bakker. 2020. The TA Framework: Designing Real-time Teaching Augmentation for K-12 Classrooms. In CHI Conference on Human Factors in Computing Systems. ACM, 1–17. https://doi.org/10.1145/3313831.3376277
[4]
R. S. Baker. 2016. Stupid Tutoring Systems, Intelligent Humans. Int. J. AI in Educ. 26, 2 (Jun 2016), 600–614. https://doi.org/10.1007/s40593-016-0105-0
[5]
C. Barreiros, P. Leitner, M. Ebner, E. Veas, and S. Lindstaedt. 2023. Students in Focus – Moving Towards Human-Centred Learning Analytics. In Practicable Learning Analytics, O. Viberg and Å. Grönlund (Eds.). Springer Inter. Publ., Cham, 77–94. https://doi.org/10.1007/978-3-031-27646-0_5
[6]
G. Biesta, M. Priestley, and S. Robinson. 2015. The Role of Beliefs in Teacher Agency. Teachers and Teaching 21, 6 (Aug. 2015), 624–640.
[7]
A. Bozkurt, A. Karadeniz, D. Baneres, A. E. Guerrero-Roldán, and M. E. Rodríguez. 2021. Artificial intelligence and reflections from educational landscape: a review of AI studies in half a century. Sustainability 13, 2 (2021), 800.
[8]
V. Braun and V. Clarke. 2012. Thematic Analysis.APA, Washington, DC, 57–71. https://doi.org/10.1037/13620-004
[9]
R. Brennan and D. Perouli. 2022. Generating and Evaluating Collective Concept Maps. In LAK22: LAK22: 12th International Learning Analytics and Knowledge Conference. Association for Computing Machinery, New York, NY, USA, 570–576. https://doi.org/10.1145/3506860.3506918
[10]
S. Buckingham Shum, R. Ferguson, and R. Martinez-Maldonado. 2019. Human-centred learning analytics. Journal of Learning Analytics 6, 2 (2019), 1–9.
[11]
B. Chen and H. Zhu. 2019. Towards Value-Sensitive Learning Analytics Design. In LAK’19: Proc. of the 9th Int. Conf. on Learning Analytics & Knowledge. 343–352. https://doi.org/10.1145/3303772.3303798 arxiv:1812.08335 [cs]
[12]
T. H. Davenport. 2018. From Analytics to Artificial Intelligence. 1, 2 (2018), 73–80. https://doi.org/10.1080/2573234X.2018.1543535
[13]
Design Council. 2023. Framework for Innovation. https://www.designcouncil.org.uk/our-resources/framework-for-innovation [Online; accessed 10. Aug. 2023].
[14]
D. Di Mitri, J. Schneider, and H. Drachsler. 2022. Keep Me in the Loop: Real-Time Feedback with Multimodal Data. Int. J. Artif. Intell. Educ. 32, 4 (Dec. 2022), 1093–1118. https://doi.org/10.1007/s40593-021-00281-z
[15]
M. E. Dogan, T. Goru Dogan, and A. Bozkurt. 2023. The Use of Artificial Intelligence (AI) in Online Learning and Distance Education Processes: A Systematic Review of Empirical Studies. Applied Sciences 13, 5 (Jan. 2023), 3056. https://doi.org/10.3390/app13053056
[16]
M. Dollinger, D. Liu, N. Arthars, and J. M. Lodge. 2019. Working together in learning analytics towards the co-creation of value.Journal of Learning Analytics 6, 2 (2019), 10–26.
[17]
A. Dood, K. Das, Z. Qian, S. Finkenstaedt-Quinn, A. Gere, and G. Shultz. 2023. A Dashboard to Provide Instructors with Automated Feedback on Students’ Peer Review Comments. In LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference. Association for Computing Machinery, New York, NY, USA, 619–625. https://doi.org/10.1145/3576050.3576087
[18]
H. Drachsler and W. Greller. 2016. Privacy and analytics: it’s a DELICATE issue a checklist for trusted learning analytics. In LAK ’16: Proc. of the 6th Intl. Conf. on Learning Analytics & Knowledge. ACM, New York, NY, USA, 89–98. https://doi.org/10.1145/2883851.2883893
[19]
V. Echeverria, M. Wong-Villacres, X. Ochoa, and K. Chiluiza. 2022. An Exploratory Evaluation of a Collaboration Feedback Report. In LAK22: LAK22: 12th International Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 478–484. https://doi.org/10.1145/3506860.3506890
[20]
J. A. Erickson, A. F. Botelho, S. McAteer, A. Varatharaj, and N. T. Heffernan. 2020. The automated grading of student open responses in mathematics. In LAK ’20: Proceedings of the 10th Intl. Conf. on Learning Analytics & Knowledge. ACM, New York, NY, USA, 615–624.
[21]
J. Fereday and E. Muir-Cochrane. 2006. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. Int. J. Qual. Methods 5, 1 (March 2006), 80–92. https://doi.org/10.1177/160940690600500107
[22]
G. M. Fernandez Nieto, K. Kitto, S. Buckingham Shum, and R. Martinez-Maldonado. 2022. Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling. In LAK22. ACM, New York, NY, USA, 219–229. https://doi.org/10.1145/3506860.3506895
[23]
R. Gubareva and R. Lopes. 2020. Virtual Assistants for Learning: A Systematic Literature Review:. In Proc. of the 12th Intl. Conf. on Computer Supported Education. SCITEPRESS - Science and Technology Publications, Prague, Czech Republic, 97–103. https://doi.org/10.5220/0009417600970103
[24]
B. Hanington and B. Martin. 2012. Universal Methods of Design: 100 Ways To Research Complex Problems, develop innovative ideas, and Design Effective Solutions. Rockport Publishers.
[25]
D. J. Haraway. 2016. Staying with the trouble: Making kin in the Chthulucene. Duke University Press.
[26]
M. Hartikainen, K. Väänänen, and T. Olsson. 2023. Towards a Human-Centred Artificial Intelligence Maturity Model. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). ACM, New York, NY, USA, Article 285, 7 pages.
[27]
P. Z. Harvard. 2023. Generate-Sort-Connect-Elaborate | Project Zero. https://pz.harvard.edu/resources/generate-sort-connect-elaborate
[28]
W. Holmes, K. Porayska-Pomsta, K. Holstein, E. Sutherland, T. Baker, S. B. Shum, O. C. Santos, M. T. Rodrigo, M. Cukurova, I. I. Bittencourt, 2022. Ethics of AI in education: Towards a community-wide framework. Int. J. AI in Educ. 32, 3 (2022), 504–526.
[29]
K. Holstein, V. Aleven, and N. Rummel. 2020. A Conceptual Framework for Human–AI Hybrid Adaptivity in Education. In AIED. Vol. 12163. 240–254.
[30]
K. Holstein, B. M. McLaren, and V. Aleven. 2019. Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity. Learning Analytics 6, 2 (July 2019), 27–52. https://doi.org/10.18608/jla.2019.62.3
[31]
D. Hooshyar, K. Tammets, T. Ley, K. Aus, and K. Kollom. 2023. Learning Analytics in Supporting Student Agency: A Systematic Review. Sustainability 15, 18 (Jan. 2023), 13662. https://doi.org/10.3390/su151813662
[32]
P. Jääskelä, V. Heilala, T. Kärkkäinen, and P. Häkkinen. 2021. Student agency analytics: learning analytics as a tool for analysing student agency in higher education. Behaviour & Information Technology 40, 8 (June 2021), 790–808. https://doi.org/10.1080/0144929X.2020.1725130
[33]
C. Kaendler, M. Wiedmann, N. Rummel, and H. Spada. 2015. Teacher Competencies for the Implementation of Collaborative Learning in the Classroom: a Framework and Research Review. Educ. Psychol. Rev. 27, 3 (Sept. 2015), 505–536. https://doi.org/10.1007/s10648-014-9288-9
[34]
O. Karademir, A. Ahmad, J. Schneider, D. Di Mitri, I. Jivet, and H. Drachsler. 2021. Designing the Learning Analytics Cockpit - A Dashboard that Enables Interventions. In Method. and Intel. Systems for Technology Enhanced Learning, 11th Inter. Conf.Springer, Cham, Switzerland, 95–104. https://doi.org/10.1007/978-3-030-86618-1_10
[35]
K. Kitto, S. Buckingham Shum, and A. Gibson. 2018. Embracing imperfection in learning analytics. In LAK ’18: Proc. of the 8th Int. Conf. on Learning Analytics and Knowledge. ACM, 451–460. https://doi.org/10.1145/3170358.3170413
[36]
R. Koster, J. Balaguer, A. Tacchetti, A. Weinstein, T. Zhu, O. Hauser, D. Williams, L. Campbell-Gillingham, P. Thacker, M. Botvinick, 2022. Human-centred mechanism design with Democratic AI. Nature Human Behaviour 6, 10 (2022), 1398–1407.
[37]
C. Lang and L. Davis. 2023. Learning Analytics and Stakeholder Inclusion: What Do We Mean When We Say "Human-Centered"?. In LAK23: 13th International Learning Analytics and Knowledge Conference. ACM, Arlington TX USA, 411–417. https://doi.org/10.1145/3576050.3576110
[38]
X. Li, L. Yan, L. Zhao, R. Martinez-Maldonado, and D. Gasevic. 2023. CVPE: A Computer Vision Approach for Scalable and Privacy-Preserving Socio-spatial, Multimodal Learning Analytics. In LAK23: 13th Inter. Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 175–185. https://doi.org/10.1145/3576050.3576145
[39]
R. Martinez-Maldonado. 2023. Human-Centred Learning Analytics: Four Challenges in Realising the Potential. Journal of Learning Letters (2023). https://doi.org/10.59453/FIZJ7007
[40]
R. Martinez-Maldonado, D. Elliott, C. Axisa, T. Power, V. Echeverria, and S. Buckingham Shum. 2022. Designing Translucent Learning Analytics with Teachers: An Elicitation Process. Interactive Learning Environments 30, 6 (July 2022), 1077–1091. https://doi.org/10.1080/10494820.2019.1710541
[41]
N. McDonald, S. Schoenebeck, and A. Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23. https://doi.org/10.1145/3359174
[42]
N. Mohammadhassan and A. Mitrovic. 2022. Investigating the Effectiveness of Visual Learning Analytics in Active Video Watching. In AIED. 127–139. https://doi.org/10.1007/978-3-031-11644-5_11
[43]
I. Molenaar. 2022. Towards hybrid human-AI learning technologies. Eur. J. Educ. 57, 4 (Dec. 2022), 632–645. https://doi.org/10.1111/ejed.12527
[44]
J. W. Moore. 2016. What Is the Sense of Agency and Why Does It Matter?Frontiers in Psychology 7 (Aug. 2016), 1272.
[45]
W. Morris, S. Crossley, L. Holmes, and A. Trumbore. 2023. Using Transformer Language Models to Validate Peer-Assigned Essay Scores in Massive Open Online Courses (MOOCs). In LAK23: 13th Inter. Learning Analytics and Knowledge Conf.ACM, 315–323. https://doi.org/10.1145/3576050.3576098
[46]
A. Muslim, M. A. Chatti, T. Mahapatra, and U. Schroeder. 2016. A Rule-Based Indicator Definition Tool for Personalized Learning Analytics. Association for Computing Machinery, 264–273. https://doi.org/10.1145/2883851.2883921
[47]
T. Nazaretsky, M. Cukurova, and G. Alexandron. 2022. An Instrument for Measuring Teachers’ Trust in AI-Based Educational Technology. In LAK22: 12th International Learning Analytics and Knowledge Conference. ACM, Online USA, 56–66. https://doi.org/10.1145/3506860.3506866
[48]
A. Pardo and G. Siemens. 2014. Ethical and privacy principles for learning analytics. Br. J. Educ. Technol. 45, 3 (May 2014), 438–450.
[49]
L. P. Prieto, M. J. Rodríguez-Triana, R. Martínez-Maldonado, Y. Dimitriadis, and D. Gašević. 2019. Orchestrating learning analytics (OrLA): Supporting inter-stakeholder communication about adoption of learning analytics at the classroom level. AJET 35 (2019). https://doi.org/10.14742/ajet.4314
[50]
C. G. Prieto-Alvarez, R. Martinez-Maldonado, and T. D. Anderson. 2018. Co-designing learning analytics tools with learners. In Learning analytics in the classroom, J. Lodge, K. Thompson, J. Horvath, P. De Barba, and M. Blumenstein (Eds.). Routledge, 93–110.
[51]
A. Renz and G. Vladova. 2021. Reinvigorating the discourse on human-centered artificial intelligence in educational technologies. Technology Innovation Management Review 11, 5 (2021).
[52]
J. P. Sarmiento and A. F. Wise. 2022. Participatory and Co-Design of Learning Analytics: An Initial Review of the Literature. In LAK22: 12th International Learning Analytics and Knowledge Conference, Vol. 1. ACM, Online USA, 535–541. https://doi.org/10.1145/3506860.3506910
[53]
M. Scheffel, Y.-S. Tsai, D. Gašević, and H. Drachsler. 2019. Policy Matters: Expert Recommendations for Learning Analytics Policy. In Transforming Learning with Meaningful Technologies. Springer, Cham, Switzerland, 510–524. https://doi.org/10.1007/978-3-030-29736-7_38
[54]
J. J. Shah, S. M. Smith, and N. Vargas-Hernandez. 2003. Metrics for Measuring Ideation Effectiveness. 24, 2 (2003), 111–134.
[55]
B. Shneiderman. 2022. Human-Centered AI. Oxford University Press, Oxford, New York.
[56]
S. Slade, P. Prinsloo, and M. Khalil. 2019. Learning Analytics at the Intersections of Student Trust, Disclosure and Benefit. In Proc. of the 9th Intl. Conf. on Learning Analytics & Knowledge (New York, NY, USA, 2019-03-04) (LAK19). ACM, 235–244. https://doi.org/10.1145/3303772.3303796
[57]
U. A. Usmani, A. Happonen, and J. Watada. 2023. Human-Centered Artificial Intelligence: Designing for User Empowerment and Ethical Considerations. In 2023 5th International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA). IEEE, Istanbul, Turkiye, 01–05. https://doi.org/10.1109/HORA58378.2023.10156761
[58]
O. Viberg, I. Jivet, and M. Scheffel. 2023. Designing Culturally Aware Learning Analytics: A Value Sensitive Perspective. Springer Inter. Publ., Cham, 177–192. https://doi.org/10.1007/978-3-031-27646-0_10
[59]
K. B. Yang, V. Echeverria, Z. Lu, H. Mao, K. Holstein, N. Rummel, and V. Aleven. 2023. Pair-Up: Prototyping Human-AI Co-orchestration of Dynamic Transitions between Individual and Collaborative Learning in the Classroom. In CHI ’23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3544548.3581398
[60]
S. J. H. Yang, H. Ogata, T. Matsui, and N.-S. Chen. 2021. Human-Centered Artificial Intelligence in Education: Seeing the Invisible through the Visible. Computers and Education: Artificial Intelligence 2 (2021). https://doi.org/10.1016/j.caeai.2021.100008
[61]
F. Zhao, G.-Z. Liu, J. Zhou, and C. Yin. 2023. A Learning Analytics Framework Based on Human-Centered Artificial Intelligence for Identifying the Optimal Learning Strategy to Intervene in Learning Behavior. 26, 1 (2023), 132–146. jstor:48707972https://www.jstor.org/stable/48707972
Index Terms
- SLADE: A Method for Designing Human-Centred Learning Analytics Systems
Recommendations
Desirability of Imagined Futures: Human-Centered Design and Analysis
Design, User Experience, and UsabilityAbstractThe future is known to be uncertain. Uncertainty is an inherent aspect of the future and impacts future predictions. Designers often require analyzing current situations to identify complexities, deficiencies, pain points etc., to gain empathy and ...
Human-Centered AI for Manufacturing – Design Principles for Industrial AI-Based Services
Artificial Intelligence in HCIAbstractAI-based services are becoming more and more common in manufacturing; however, the development, implementation, and operation of these services are associated with challenges. The design of Human-Centered AI (HCAI) is one approach to address these ...
Designing a Couples-Based Conversational Agent to Promote Safe Sex in New, Young Couples: A User-Centred Design Approach
CUI '24: Proceedings of the 6th ACM Conference on Conversational User InterfacesThe uptake of conversational agents (CAs) to deliver digital sexual health interventions is growing. While current CAs only address one user at a time, research suggests that couples-based interventions may be more effective at promoting safe sex in non-...
Comments
Information & Contributors
Information
Published In
March 2024
962 pages
ISBN:9798400716188
DOI:10.1145/3636555
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
LAK '24
Acceptance Rates
Overall Acceptance Rate 236 of 782 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 747Total Downloads
- Downloads (Last 12 months)747
- Downloads (Last 6 weeks)107
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
Cited By
View all- Wang YLai YHuang X(2024)Innovations in Online Learning Analytics: A Review of Recent Research and Emerging TrendsIEEE Access10.1109/ACCESS.2024.349362112(166761-166775)Online publication date: 2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in