research-article

Addressing learner issues with StepUp!: an evaluation

Authors:

Jose Luis Santos,

Katrien Verbert,

Erik DuvalAuthors Info & Claims

LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge

Pages 14 - 22

https://doi.org/10.1145/2460296.2460301

Published: 08 April 2013 Publication History

Abstract

This paper reports on our research on the use of learning analytics dashboards to support awareness, self-reflection, sensemaking and impact for learners. So far, little research has been done to evaluate such dashboards with students and to assess their impact on learning. In this paper, we present the results of an evaluation study of our dashboard, called StepUp!, and the extent to which it addresses issues and needs of our students. Through brainstorming sessions with our students, we identified and prioritized learning issues and needs. In a second step, we deployed StepUp! during one month and we evaluated to which extent our dashboard addresses the issues and needs identified earlier in different courses. The results show that our tool has potentially higher impact for students working in groups and sharing a topic than students working individually on different topics.

References

[1]

Ali, L., Hatala, M., Gašević, D., & Jovanovi&cacute;ć, J. A qualitative evaluation of evolution of a learning analytics tool. Computers and Education, 58(1), 470--489. 2012

Digital Library

[2]

Arnold, K. E. & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. In S. B. Shum, D. Gasevic, and R. Ferguson (Eds.) Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12),(pp. 267--270). NY: ACM

Digital Library

[3]

Baillie, C. Motivation and attrition in engineering students, Geraldine Fitzgerald European Journal of Engineering Education Vol. 25, Iss. 2, 2000

[4]

Bangor, A., Kortum, P. T., Miller, J. T.: An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interaction (2008) 574--594

[5]

Barber, R., Sharkey, M. Course correction: using analytics to predict course success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), 2012

Digital Library

[6]

Parra, G., Klerkx, J., Duval, E. What Should I Read Next? Awareness of Relevant Publications Through a Community of Practice. In SIGCHI '13 Case Studies. Accepted.

Digital Library

[7]

Blikstein, P. Using learning analytics to assess students' behavior in open-ended programming tasks. In Proceedings of the Learning Analytics and Knowledge conferencd (LAK11), 2011 Brooke, J. SUS: A quick and dirty usability scale. In: Usability Evaluation in Industry. Taylor & Francis, London, 1996.

Digital Library

[8]

Conole, G., Fill, K. (2005). A learning design toolkit to create pedagogically effective learning activities. Journal of Interactive Media in Education 2005(08).

[9]

Dawson, S., Bakharia, A., & Heathcote, E. SNAPP: Realising the affordances of real-time SNA within networked learning environments. In L. Dirckinck- Holmfeld, V. Hodgson, C. Jones, M. de Laat, D. McConnell, & T. Ryberg (Eds.), Proceedings of the 7th International Conference on Networked Learning (pp. 125--133) 2010

[10]

Dollár, A., Steif, P. S. Web-based Statics Course with Learning Dashboard for Instructors. In Uskov, V. (Ed.), Proceedings of Computers and Advanced Technology in Education (CATE 2012), June 25--27, 2012, Napoli, Italy. 2012

[11]

Duval, E.: Attention please! learning analytics for visualization and recommendation. In: Proceedings of LAK11: 1st International Conference on Learning Analytics and Knowledge, ACM (2011) 9--17

Digital Library

[12]

Duval, E., Klerkx, J., Verbert, K., Nagel, T., Govaerts, S., Parra Chico, G. A., Santos, J. L., Vandeputte, B.: Learning dashboards and learnscapes. In: Educational Interfaces, Software, and Technology,. (May 2012) 1--5

[13]

Essa, A., Ayad, H. Student success system: risk analytics and data visualization using ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), 2012

Digital Library

[14]

Ferguson, R., Buckingham, S. Shum. Social learning analytics: five approaches. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12) 2012.

Digital Library

[15]

Fischer, G.: Understanding, fostering, and supporting cultures of participation. interactions 18(3) (May 2011) 42--53

Digital Library

[16]

France, L., Heraud, J.-M., Marty, J.-C., Carron, T., & Heili, J. Monitoring Virtual Classroom: Visualization Techniques to Observe Student Activities in an e-Learning System. Proceedings of the Sixth International Conference on Advanced Learning Technologies, (pp. 716--720). 2006

Digital Library

[17]

García-Solórzano, D., Cobo, G., Santamaría, E., Morán, J. A., Monzo, C., Melenchón, J. Educational monitoring tool based on faceted browsing and data portraits. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). 2012

Digital Library

[18]

Govaerts, S., Verbert, K., Duval, E., & Pardo, A. The Student Activity Meter for Awareness and Self-reflection. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems Extended Abstracts (pp. 869--884). ACM. 2012

Digital Library

[19]

Kerly, A., Ellis, R. & Bull, S. CALMsystem: A Conversational Agent for Learner Modelling, in R. Ellis, T. Allen & M. Petridis (eds), Applications and Innovations in Intelligent Systems XV -- Proceedings of AI-2007, 27th SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence (pp. 89--102). Springer Verlag. 2007

[20]

Killen, R., Differences between students' and lecturers' perceptions of factors inuencing students' academic success at university. Higher Education Research and Development, 13, 199. 1994

[21]

Kobsa, E., Dimitrova, V., & Boyle, R. Using student and group models to support teachers in web-based distance education. Proc. of the 10th international conference on user modeling (pp. 124--133). Edinburgh, UK. 2005

Digital Library

[22]

Lafford, B. A. Review of Tell Me More Spanish, Journal on Language Learning & Technology, 8(3), 21--34. 2004

[23]

Leony, D., Pardo, A., Fuente, L, Sánchez, D., Delgado, C. GLASS: a learning analytics visualization tool. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), 2012

Digital Library

[24]

Lockyer, L., Dawson, S. Where learning analytics meets learning design. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). 2012

Digital Library

[25]

Mazza, R., & Milani, C. (2004). GISMO: a Graphical Interactive Student Monitoring Tool for Course Management Systems. TEL'04 Technology Enhanced Learning '04 International Conference (pp. 1--8). Citeseer

[26]

Niemann, K., Schmitz, H-C, Kirschenmann, U., Wolpers, M. Anna Schmidt, and Tim Krones. Clustering by usage: higher order co-occurrences of learning objects. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12) 2012

Digital Library

[27]

Pardo, A., Kloos, C. Stepping out of the box towards analytics outside the learning management system. In Proceedings of the Learning Analytics and Knowledge conference (LAK11), 2011

Digital Library

[28]

Podgorelec, V. & Kuhar, S. Taking Advantage of Education Data: Advanced Data Analysis and Reporting in Virtual Learning Environments. Electronics and Electrical Engineering, 114(8), 111--116.

[29]

Richards, G., DeVries, I. Revisiting formative evaluation: Dynamic monitoring for the improvement of learning activity design and delivery. In Proceedings of the Learning Analytics and Knowledge conference (LAK11), 2011

Digital Library

[30]

Roijers, D. M., Jeuring, J., Feelders, A. Probability estimation and a competence model for rule based e-tutoring systems. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), 2012

Digital Library

[31]

Santos, J. L., Verbert, K., Govaerts, S., Duval, E. Empowering students to reflect on their activity with StepUp!: Two case studies with engineering students, In proceedings of EFEPLE11 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning, CEUR WS (2012) (accepted)

[32]

Santos, J. L., Govaerts, S., Verbert, K., Duval, E.: Goal-oriented visualizations of activity tracking: a case study with engineering students. In: LAK12: International Conference on Learning Analytics and Knowledge, Vancouver, Canada, 29 April - 2 May 2012, ACM (May 2012)

Digital Library

[33]

Schreurs, B., De Laat, M. Network awareness tool - learning analytics in the workplace: detecting and analyzing informal workplace learning. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12). 2012

Digital Library

[34]

Silius, K., Miilumaki, T., Huhtamaki, J., Tebest, T., Merilainen, J., & Pohjolainen, S. Students' Motivations for Social Media Enhanced Studying and Learning. Knowledge Management & E-Learning: An International Journal (KM&EL), 2(1). 51--67. 2010

[35]

Tufte, E. R.: Beautiful Evidence. Graphics Press (2006)

Digital Library

[36]

Upton, K., Kay, J. Narcissus: Group and individual models to support small group work. In Proceedings of the 17th International Conference on User Modeling, Adaptation, and Personalization: formerly UM and AH, UMAP '09, pages 54--65, Berlin, Heidelberg, 2009. Springer-Verlag,. 2009

Digital Library

[37]

Verbert, K., Manouselis, N., Drachsler, H., Duval, E.: Dataset-driven research to support learning and knowledge analytics. Educational Technology and Society (2012) 1--21

[38]

Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Duval, E. Dataset-driven research for improving recommender systems for learning. In Proceedings of the Learning Analytics and Knowledge conference (LAK11), 2011

Digital Library

[39]

Wenger, E.: Communities of Practice: Learning, Meaning, and Identity (Learning in Doing: Social, Cognitive and Computational Perspectives). 1 edn. Cambridge University Press (September 1999)

[40]

Yacef, K., Zaïane, O., Hershkovitz, H., Yudelson, M., and Stamper, J. (eds.) Proceedings of the 5th International Conference on Educational Data Mining. 2012

[41]

Zinn, C., Scheuer, O. How did the e-learning session go? The Student Inspector. In In Luckin, R., Koedinger, K. R., & Greer, J. (Eds.) Proceeding of the 2007 conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work (pp. 487--494). IOS Press: Amsterdam, The Netherlands (2007)

Digital Library

[42]

Williams, K., Williams, C. Five key ingredients for improving student motivation (2011) Vol. 12. (pp. 1--23) Research in Higher Education Journal

Cited By

Kleinman EXu JPfau JSeif El-Nasr M(2024)"Trust the Process": An Exploratory Study of Process Visualizations for Self-Reflection in League of LegendsProceedings of the ACM on Human-Computer Interaction10.1145/36771118:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677111
Paulsen LLindsay E(2024)Learning analytics dashboards are increasingly becoming about learning and not just analytics - A systematic reviewEducation and Information Technologies10.1007/s10639-023-12401-429:11(14279-14308)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s10639-023-12401-4
Kleinman EVillareale JShergadwala MTeng ZBryant AZhu JEl-Nasr M(2023)"What else can I do?" Examining the Impact of Community Data on Adaptation and Quality of Reflection in an Educational GameProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580664(1-12)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580664
Show More Cited By

Index Terms

Addressing learner issues with StepUp!: an evaluation
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Goal-oriented visualizations of activity tracking: a case study with engineering students
LAK '12: Proceedings of the 2nd International Conference on Learning Analytics and Knowledge

Increasing motivation of students and helping them to reflect on their learning processes is an important driver for learning analytics research. This paper presents our research on the development of a dashboard that enables self-reflection on ...
Instructor Perspectives on Comparative Heatmap Visualizations of Student Engagement with Lecture Video
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

The use of online video in learning environments is prevalent -- serving in a variety of roles, such as multimedia textbook, video lecture, or medium for documenting student coursework. For instructors who employ video as a learning material, it is ...
Visualizing patterns of student engagement and performance in MOOCs
LAK '14: Proceedings of the Fourth International Conference on Learning Analytics And Knowledge

In the last five years, the world has seen a remarkable level of interest in Massive Open Online Courses, or MOOCs. A consistent message from universities participating in MOOC delivery is their eagerness to understand students' online learning ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LAK '13: Proceedings of the Third International Conference on Learning Analytics and Knowledge

April 2013

300 pages

ISBN:9781450317856

DOI:10.1145/2460296

Editors:
Dan Suthers
University of Hawaii, USA
,
Katrien Verbert
Eindhoven University of Technology, The Netherlands
,
Erik Duval
University of Leuven, Belgium
,
Xavier Ochoa
Escuela Superior Politécnica del Litoral, Ecuador

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web
K.U. Leuven: K.U. Leuven
SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Seventh Framework Programme

Conference

LAK '13

Sponsor:

SIGWEB
K.U. Leuven
SIGCHI

LAK '13: Third Conference on Learning Analytics and Knowledge

April 8 - 13, 2013

Leuven, Belgium

Acceptance Rates

LAK '13 Paper Acceptance Rate 16 of 58 submissions, 28%;

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
1,125
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kleinman EXu JPfau JSeif El-Nasr M(2024)"Trust the Process": An Exploratory Study of Process Visualizations for Self-Reflection in League of LegendsProceedings of the ACM on Human-Computer Interaction10.1145/36771118:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677111
Paulsen LLindsay E(2024)Learning analytics dashboards are increasingly becoming about learning and not just analytics - A systematic reviewEducation and Information Technologies10.1007/s10639-023-12401-429:11(14279-14308)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s10639-023-12401-4
Kleinman EVillareale JShergadwala MTeng ZBryant AZhu JEl-Nasr M(2023)"What else can I do?" Examining the Impact of Community Data on Adaptation and Quality of Reflection in an Educational GameProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580664(1-12)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580664
M. SS. K(2023)A novel approach to predict competency and the hidden risk factor by using various machine learning classifiersAutomatika10.1080/00051144.2023.220034764:3(550-564)Online publication date: 17-Apr-2023
https://doi.org/10.1080/00051144.2023.2200347
Ahmad AYordanov IYau JSchneider JDrachsler H(2023)A Trusted Learning Analytics Dashboard for Displaying OERDistributed Learning Ecosystems10.1007/978-3-658-38703-7_15(279-303)Online publication date: 21-Feb-2023
https://doi.org/10.1007/978-3-658-38703-7_15
Ahmad ASchneider JGriffiths DBiedermann DSchiffner DGreller WDrachsler H(2022)Connecting the dots – A literature review on learning analytics indicators from a learning design perspectiveJournal of Computer Assisted Learning10.1111/jcal.12716Online publication date: 26-Jul-2022
https://doi.org/10.1111/jcal.12716
Kleinman EGayle CSeif El-Nasr M(2021)“Because I'm Bad at the Game!” A Microanalytic Study of Self Regulated Learning in League of LegendsFrontiers in Psychology10.3389/fpsyg.2021.78023412Online publication date: 2-Dec-2021
https://doi.org/10.3389/fpsyg.2021.780234
Akçapınar GHasnine M(2021)Development and Evaluation of a Student-Facing Gamified Learning Analytics DashboardVisualizations and Dashboards for Learning Analytics10.1007/978-3-030-81222-5_13(269-287)Online publication date: 17-Dec-2021
https://doi.org/10.1007/978-3-030-81222-5_13
Verbert KOchoa XDe Croon RDourado RDe Laet TRensing CDrachsler HKovanović VPinkwart NScheffel MVerbert K(2020)Learning analytics dashboardsProceedings of the Tenth International Conference on Learning Analytics & Knowledge10.1145/3375462.3375504(35-40)Online publication date: 23-Mar-2020
https://dl.acm.org/doi/10.1145/3375462.3375504
Wang DHan H(2020)Applying learning analytics dashboards based on process‐oriented feedback to improve students' learning effectivenessJournal of Computer Assisted Learning10.1111/jcal.1250237:2(487-499)Online publication date: 26-Oct-2020
https://doi.org/10.1111/jcal.12502
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents