Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3636555.3636886acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article
Open access

How do visualizations and automated personalized feedback engage professional learners in a Learning Analytics Dashboard?

Published: 18 March 2024 Publication History

Abstract

Learning Analytics Dashboards (LAD) are the subject of research in a multitude of schools and higher education institutions, but a lack of research into learner-facing dashboards in professional learning has been identified. This study took place in an authentic professional learning context and aims to contribute insights into LAD design by using an academic approach in a practice-based environment. An existing storytelling LAD created to support 81 accountants was evaluated using Technology Acceptance Model, finding a learner expectation for clarity, conciseness, understanding and guidance on next steps. High usage levels and a ‘take what you need’ approach was identified, with all visualizations and automated personalized feedback being considered useful although to varying degrees. Professional learners in this study focus on understanding and acting upon weaknesses rather than celebrating strengths. The lessons for LAD design are to offer choice and create elements which support learners to take action to improve performance at a multitude of time points and levels of success.

References

[1]
June Ahn, Fabio Campos, Maria Hays, and Daniela Digiacomo. 2019. Designing in context: Reaching beyond usability in learning analytics dashboard design, Journal of Learning Analytics 6, 2 (July 2019), 70–85. https://doi.org/10.18608/jla.2019.62.5
[2]
Markus Blut, Alain Chong, Zayayd Tsiga, Viswanath Venkatesh. 2021. Meta-analysis of the unified theory of acceptance and use of technology (UTAUT): Challenging its validity and charting a research agenda in the red ocean, Jais - Journal of the Association for Information Systems, forthcoming. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3834872
[3]
Simon Buckingham Shum, Allison Littlejohn, Kirsty Kitto and Ruth Crick. 2022. Framing Professional Learning Analytics as Reframing Oneself, IEEE Transactions on Learning Technologies (Early Access), 1–17. Https://doi.org/10.1109/TLT.2022.3190055.
[4]
David Carless and David Boud. 2018. The development of student feedback literacy: enabling uptake of feedback, Assessment and Evaluation in Higher Education, 43, 8 (May 2018), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354
[5]
Mohamed Amine Chatti, Arham Muslim, Manpriya Guliani and Mouadh Guesmi. 2020. The LAVA Model: Learning Analytics Meets Visual Analytics in Dirk Ifenthaler, David Gibson (eds). Adoption of Data Analytics in Higher Education Learning and Teaching. Advances in Analytics for Learning and Teaching. Springer, Cham. https://doi-org/10.1007/978-3-030-47392-1_5
[6]
Xieling Chen, Di Zou and Haoran Xie. 2022. A decade of learning analytics: Structural topic modeling based bibliometric analysis, Education and Information Technologies. https://doi-org/10.1007/s10639-022-11046-z
[7]
Doug Clow. 2012. The learning analytics cycle: Closing the loop effectively. In 2nd International Conference on Learning Analytics and Knowledge - LAK ’12, 134–138. https://doi.org/10.1145/2330601.2330636
[8]
Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly: Management Information Systems. 13, 3 (September 1989), 319–339. https://doi.org/10.2307/249008
[9]
Rachel Dickler, Janice Gobert and Michael Sao Pedro. 2021. Using innovative methods to explore the potential of an alerting dashboard for science inquiry, Journal of Learning Analytics, 8, 2 (September 2019) 105–122. https://doi.org/10.18608/jla.2021.7153
[10]
Blazenka Divjak, Barbi Svetec, and Damir Horvat. 2023. Learning analytics dashboards: What do students actually ask for? In LAK23: 13th International Learning Analytics and Knowledge Conference (LAK23), 44–56. https://doi.org/10.1145/3576050.3576141
[11]
Mollie Dollinger and Jason Lodge. 2018. Co-Creation strategies for learning analytics, In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK '18). 97–101. https://doi-org/10.1145/3170358.3170372
[12]
Vanessa Echeverria, Roberto Martinez-Maldonado, Simon Buckingham Shum, Katherine Chiluiza, Roger Granda and Cristina Conati. 2018. Exploratory versus Explanatory Visual Learning Analytics: Driving Teachers’ Attention through Educational Data Storytelling, Journal of Learning Analytics, 5, 3 (November 2018) 73-97. https://doi.org/0.18608/jla.2018.53.6
[13]
Rebecca Ferguson, Xavier Ochoa and Vitomir Kovanovi. 2022. Learning Analytics: Practitioners, Take Note: Journal of Learning Analytics Editorial 2022. Journal of Learning Analytics, 9, 3 (December 2022) 1–7. https://.doi.org/10.18608/jla.2022.7937
[14]
Gloria Milena Fernandez Nieto, Kirsty Kitto, Simon Buckingham Shum and Roberto Martinez-Maldonado. 2022. Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling, LAK22: 12th International Learning Analytics and Knowledge Conference, 219–229 https://doi.org/10.1145/3506860.3506895
[15]
Nigel G Fielding. 2012. Triangulation and Mixed Methods Designs: Data Integration With New Research Technologies, Journal of Mixed Methods Research, 6, 2 (April 2019) 124–136. https://doi.org/10.1177/1558689812437101
[16]
FRC (2021) Key Facts and Trends in the Accountancy Profession, London. Retrieved September 28, 2022 from https://www.frc.org.uk/getattachment/e976ff38-3597-4779-b192-1be7da79d175/FRC-Key-Facts-Trends-2021.pdf
[17]
Dragan Gasevic, Yi-Shan Tsai, Shane Dawson, and Abelardo Pardo. 2019. How do we start? An approach to learning analytics adoption in higher education, International Journal of Information and Learning Technology, 36, 4 (July 2019) 342–353. https://doi.org/10.1108/IJILT-02-2019-0024
[18]
Christothea Herodotou, Bart Rienties, Avinash Boroowa, Zdenek Zdrahal and Martin Hlosta. 2019. A large-scale implementation of predictive learning analytics in higher education: the teachers’ role and perspective, Educational Technology Research and Development, 67, 5 https://doi.org/10.1007/s11423-019-09685-0
[19]
John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research. 77, 1 (March 2007) 81–112. https://doi.org/10.3102/003465430298487
[20]
Rafael Hidalgo and Gerald Evans. 2020. Analytics for Action: Assessing effectiveness and impact of data informed interventions on online modules. Revista Iberoamericana de Educación a Distancia, 23, 2. 103–125. https://doi.org/10.5944/ried.23.2.26450
[21]
Ioana Jivet, Maren Scheffel, Marcus Specht and Hedrik Drachsler. 2018. License to evaluate: Preparing learning analytics dashboards for educational practice, LAK’18: International Conference on Learning Analytics and Knowledge, New York, ACM, 31–40. https://doi.org/10.1145/3170358.3170421
[22]
Jeonghyun Kim, Il-Hyun Jo and Yeonjeong Park. 2016. Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement, Asia Pacific Education Review, 17, 1, (March 2016) 13–24. https://doi.org/10.1007/s12564-015-9403-8
[23]
Joris Klerkx, Katrien Verbert and Erik Duval. 2017. Learning Analytics Dashboards. In Charles Lang, George Siemens, Alyssa Wise and Dragan Gasevic (eds) Handbook of Learning Analytics, Society of Learning Analytics Research (SoLAR), Alberta, Canada. https://doi.org/10.18608/hla17.012
[24]
Carolien Knoop-Van Campen and Inge Molenaar. 2020. How teachers integrate dashboards into their feedback practices. Frontline Learning Research 8, 4 (July 2020) 37–51. https://doi.org/10.14786/flr.v8i4.641
[25]
Phil Long and George Siemens. 2011. Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE Review. 46, 5 (September 2011). https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education
[26]
Benjamin A. Motz, Yoav Bergner, Christoper A. Brooks, Anna Gladden, Geraldine Gray, Charles Lang, Warren Li, Fernando Marmolejo-Ramos, Joshua D. Quick. 2023. LAK of Direction: Misalignment Between the Goals of Learning Analytics and its Research Scholarship. Journal of Learning Analytics 1–13. Https://doi.org/10.18608/jla.2023.7913
[27]
Kathe Pelletier, Mark McCormack, Jamie Reeves, Jenay Robert, Nichole Arbino, Maha Al-Freih, Camille Dickson-Deane, Carlos Guevara, Lisa Koster, Melchor Sánchez-Mendiola, Lee Skallerup Bessette, and Jake Stine. 2021. 2021 EDUCAUSE Horizon Report. Teaching and Learning Edition, Educause. Boulder. https://www.educause.edu/horizon-report-teaching-and-learning-2021
[28]
Irina Rets, Christothea Herodotou, Vaclav Bayer, Martin Hlosta and Bart Rienties. 2021. Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students, International Journal of Educational Technology in Higher Education, 18, 1 (September 2021) https://doi.org/10.1186/s41239-021-00284-9
[29]
Bart Rienties, Avinash Boroowa, Simon Cross, Chris Kubiak and Kevin Mayles. 2016. Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK, Journal of Interactive Media in Education, 2, (January 2016) 1–11. https://doi.org/10.5334/jime.394
[30]
Lynne D. Roberts, Joel A. Howell and Kristen Seaman. 2017. Give Me a Customizable Dashboard: Personalized Learning Analytics Dashboards in Higher Education. Technology, Knowledge and Learning. 22, 3 (June 2017) 317–333. https://doi.org/10.1007/s10758-017-9316-1
[31]
Madjid Sadallah and Jean-Marie Gilliot. 2023. Generating LADs that Make Sense. In Proceedings of the 15th International Conference on Computer Supported Education, 35–46. https://doi.org/10.5220/0011839800003470
[32]
Clara Schumacher and Dirk Ifenthaler. 2018. Features students really expect from learning analytics. Computers in Human Behaviour. 78 (June 2018) 397-407. https://doi-org/10.1016/j.chb.2017.06.030
[33]
Beat A Schwendimann, Maria Jesus Rodriguez-Triana, Andrii Vozniuk, Luis P Prieto, Mina Shirvani Boroujeni, Adrian Holzer, Denis Gillet and Pierre Dillenbourg. 2017. Perceiving learning at a glance: A systematic literature review of learning dashboard research, IEEE Transactions on Learning Technologies, 10, 1 (Jan-March 2017) 30–41. https://doi.org/10.1109/TLT.2016.2599522
[34]
Niall Sclater, Alices Peasgood, and Joel Mullan. 2016. Learning analytics in higher education: A review of UK and international practice Full report, Jisc. Retrieved 28 September, 2022 from https://www.jisc.ac.uk/sites/default/files/learning-analytics-in-he-v2_0.pdf.
[35]
Gayane Sedrakyan, Erik Mannens and Katrien Verbert. 2019. Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts. Journal of Visual Languages and Computing, 50 (June 2018) 19–38. https://doi.org/10.1016/j.jvlc.2018.11.002
[36]
Neil Selwyn. 2021. The human labour of school data: exploring the production of digital data in schools. Oxford Review of Education, Routledge, 47, 3. 353–368. https://doi.org/10.1080/03054985.2020.1835628
[37]
Jennifer Pei-Ling Tan, Elizabeth Koh, Christin Rekha Jonathan and Simon Yang. 2017. Learner Dashboards a Double-Edged Sword? Students’ Sense-Making of a Collaborative Critical Reading and Learning Analytics Environment for Fostering 21st Century Literacies. Journal of Learning Analytics, 4, 1 (March 2017) 117–140. https://doi.org/10.18608/jla.2017.41.7
[38]
Viswanath Venkatesh and Fred D Davis. 2000. Theoretical extension of the Technology Acceptance Model: Four longitudinal field studies. Management Science. 46, 2 (February 2000) 186–204. https://doi.org/10.1287/mnsc.46.2.186.11926
[39]
David Wiley. 2020. From Actionable Dashboards to Action Dashboards Improving Student Success (16 August 2020). Retrieved 25 January, 2021 from https://improvingstudentsuccess.org/2020/08/from-actionable-dashboards-to-action-dashboards
[40]
Alyssa F Wise, Simon Knight and Xavier Ochoa. 2021. What Makes Learning Analytics Research Matter. Journal of Learning Analytics 8, 3 (December 2021), 1–9. https://doi.org/0.18608/jla.2021.7647
[41]
Clara Schumacher and Dirk Ifenthaler. 2018. Features students really expect from learning analytics. Computers in Human Behaviour. 78 (June 2018) 397-407. https://doi-org/10.1016/j.chb.2017.06.030

Index Terms

  1. How do visualizations and automated personalized feedback engage professional learners in a Learning Analytics Dashboard?

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference
      March 2024
      962 pages
      ISBN:9798400716188
      DOI:10.1145/3636555
      This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 18 March 2024

      Check for updates

      Author Tags

      1. LAD
      2. Learning Analytics Dashboard
      3. Technology Acceptance Model
      4. accountancy
      5. assessment
      6. data storytelling
      7. feedback
      8. personalization
      9. professional learning

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      LAK '24

      Acceptance Rates

      Overall Acceptance Rate 236 of 782 submissions, 30%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 287
        Total Downloads
      • Downloads (Last 12 months)287
      • Downloads (Last 6 weeks)70
      Reflects downloads up to 12 Sep 2024

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Get Access

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media