research-article

Auto-Generating Multimedia Language Learning Material for Children with Off-the-Shelf AI

Authors:

Albrecht Schmidt,

Lewis L. ChuangAuthors Info & Claims

MuC '22: Proceedings of Mensch und Computer 2022

Pages 96 - 105

https://doi.org/10.1145/3543758.3543777

Published: 15 September 2022 Publication History

Abstract

The unique affordances of mobile devices enable the design of novel language learning experiences with auto-generated learning materials. Thus, they can support independent learning without increasing the burden on teachers. In this paper, we investigate the potential and the design requirements of such learning experiences for children. We implement a novel mobile app that auto-generates context-based multimedia material for learning English. It automatically labels photos children take with the app and uses them as a trigger for generating content using machine translation, image retrieval, and text-to-speech. An exploratory study with 25 children showed that children were ready to engage to an equal extent with this app and a non-personal version using random instead of personal photos. Overall, the children appreciated the independence gained compared to learning at school but missed the teachers’ support. From a technological perspective, we found that auto-generation works in many cases. However, handling erroneous input, such as blurry images and spelling mistakes, is crucial for children as a target group. We conclude with design recommendations for future projects, including scaffolds for the photo-taking process and information redundancy for identifying inaccurate auto-generation results.

Supplementary Material

MP4 File (LearningbyExploring.mp4)

Supplemental video

Download
8.39 MB

References

[1]

Elham Beheshti, David Kim, Gabrielle Ecanow, and Michael S. Horn. 2017. Looking Inside the Wires: Understanding Museum Visitor Learning with an Augmented Circuit Exhibit. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM Press, Denver, Colorado, USA, 1583–1594. https://doi.org/10.1145/3025453.3025479

Digital Library

[2]

Walcir Cardoso. 2018. Learning L2 pronunciation with a text-to-speech synthesizer. Future-proof CALL: language learning as exploration and encounters–short papers from EUROCALL(2018), 16–21.

[3]

Childwise. 2020. Childhood 2020: new independent report. http://www.childwise.co.uk/uploads/3/1/6/5/31656353/childwise_press_release_-_monitor_2020_-_immediate_release.pdf

[4]

Dorothy M. Chun and Jan L. Plass. 1996. Effects of Multimedia Annotations on Vocabulary Acquisition. The Modern Language Journal 80, 2 (June 1996), 183–198. https://doi.org/10.1111/j.1540-4781.1996.tb01159.x

[5]

Gabriel Culbertson, Solace Shen, Erik Andersen, and Malte Jung. 2017. Have your Cake and Eat it Too: Foreign Language Learning with a Crowdsourced Video Captioning System. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, Portland Oregon USA, 286–296. https://doi.org/10.1145/2998181.2998268

Digital Library

[6]

Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. Comput. Surveys 51, 1 (Jan. 2018), 1–40. https://doi.org/10.1145/3148148

Digital Library

[7]

Fiona Draxler, Audrey Labrie, Albrecht Schmidt, and Lewis L. Chuang. 2020. Augmented Reality to Enable Users in Learning Case Grammar from Their Real-World Interactions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–12. https://doi.org/10.1145/3313831.3376537

Digital Library

[8]

Fiona Draxler, Elena Wallwitz, Albrecht Schmidt, and Lewis L. Chuang. 2020. An Environment-Triggered Augmented-Reality Application for Learning Case Grammar. In DELFI 2020 – Die 18. Fachtagung Bildungstechnologien der Gesellschaft für Informatik e.V., Raphael Zender, Dirk Ifenthaler, Thiemo Leonhardt, and Clara Schumacher (Eds.). Gesellschaft für Informatik e.V., Bonn, 389–390.

[9]

Matt Dunleavy and Chris Dede. 2014. Augmented Reality Teaching and Learning. In Handbook of Research on Educational Communications and Technology, J. Michael Spector, M. David Merrill, Jan Elen, and M. J. Bishop (Eds.). Springer New York, New York, NY, 735–745. https://doi.org/10.1007/978-1-4614-3185-5_59

[10]

Darren Edge, Elly Searle, Kevin Chiu, Jing Zhao, and James A. Landay. 2011. MicroMandarin: mobile language learning in context. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, Vancouver, BC, Canada, 3169. https://doi.org/10.1145/1978942.1979413

Digital Library

[11]

Raphaël Grasset, Andreas Dünser, and Mark Billinghurst. 2008. Edutainment with a mixed reality book: a visually augmented illustrative childrens’ book. In Proceedings of the 2008 International Conference in Advances on Computer Entertainment Technology - ACE ’08. ACM Press, Yokohama, Japan, 292. https://doi.org/10.1145/1501750.1501819

Digital Library

[12]

Wendy S Grolnick and Richard M Ryan. 1987. Autonomy in children’s learning: an experimental and individual difference investigation.Journal of personality and social psychology 52, 5(1987), 890. Publisher: American Psychological Association.

[13]

Ari Hautasaari, Takeo Hamada, Kuntaro Ishiyama, and Shogo Fukushima. 2019. VocaBura: A Method for Supporting Second Language Vocabulary Learning While Walking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (Dec. 2019), 1–23. https://doi.org/10.1145/3369824

Digital Library

[14]

Jan Herrington and Ron Oliver. 1995. Critical Characteristics of Situated Learning: Implications for the Instructional Design of Multimedia. In ASCILITE 1995 Conference. Melbourne, 253–262.

[15]

Suzanne Hidi and K. Ann Renninger. 2006. The Four-Phase Model of Interest Development. Educational Psychologist 41, 2 (June 2006), 111–127. https://doi.org/10.1207/s15326985ep4102_4

[16]

Katrin Hochberg, Jochen Kuhn, and Andreas Müller. 2018. Using Smartphones as Experimental Tools—Effects on Interest, Curiosity, and Learning in Physics Education. Journal of Science Education and Technology 27, 5 (Oct. 2018). https://doi.org/10.1007/s10956-018-9731-7

[17]

Brandon Huynh, Jason Orlosky, and Tobias Hollerer. 2019. In-Situ Labeling for Augmented Reality Language Learning. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan, 1606–1611. https://doi.org/10.1109/VR.2019.8798358

[18]

Daesang Kim and David A Gilman. 2008. Effects of text, audio, and graphic aids in multimedia instruction for vocabulary learning. Journal of educational technology & society 11, 3 (2008), 114–126. Publisher: JSTOR.

[19]

Nate Kornell. 2009. Optimising learning using flashcards: Spacing is more effective than cramming. Applied Cognitive Psychology 23, 9 (Dec. 2009), 1297–1317. https://doi.org/10.1002/acp.1537

[20]

Stephan Lewandowsky, Ullrich K. H. Ecker, Colleen M. Seifert, Norbert Schwarz, and John Cook. 2012. Misinformation and Its Correction: Continued Influence and Successful Debiasing. Psychological Science in the Public Interest 13, 3 (Dec. 2012), 106–131. https://doi.org/10.1177/1529100612451018

[21]

R.E. Mayer. 2017. Using multimedia for e‐learning. Journal of Computer Assisted Learning 33, 5 (Oct. 2017), 403–423. https://doi.org/10.1111/jcal.12197

[22]

Colleen McDonough, Lulu Song, Kathy Hirsh-Pasek, Roberta Michnick Golinkoff, and Robert Lannon. 2011. An image is worth a thousand words: why nouns tend to dominate verbs in early word learning: Imageability and early word learning. Developmental Science 14, 2 (March 2011), 181–189. https://doi.org/10.1111/j.1467-7687.2010.00968.x

[23]

Elisa D. Mekler, Florian Brühlmann, Alexandre N. Tuch, and Klaus Opwis. 2017. Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior 71 (June 2017), 525–534. https://doi.org/10.1016/j.chb.2015.08.048

Digital Library

[24]

Janet Metcalfe and Bridgid Finn. 2013. Metacognition and control of study choice in children. Metacognition and Learning 8, 1 (April 2013), 19–46. https://doi.org/10.1007/s11409-013-9094-7

[25]

Dan Motzenbecker. 2017. Thing Translator. https://experiments.withgoogle.com/thing-translator

[26]

Mauro Nicolao, Amy V. Beeston, and Thomas Hain. 2015. Automatic assessment of English learner pronunciation using discriminative classifiers. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, South Brisbane, Queensland, Australia, 5351–5355. https://doi.org/10.1109/ICASSP.2015.7178993

[27]

Jan L. Plass and Shashank Pawar. 2020. Toward a taxonomy of adaptivity for learning. Journal of Research on Technology in Education 52, 3 (July 2020), 275–300. https://doi.org/10.1080/15391523.2020.1719943

[28]

Y. Rogers, D. Stanton, M. Thompson, M. Weal, S. Price, G. Fitzpatrick, R. Fleck, E. Harris, H. Smith, C. Randell, H. Muller, and C. O’Malley. 2004. Ambient wood: designing new forms of digital augmentation for learning outdoors. In Proceeding of the 2004 conference on Interaction design and children building a community - IDC ’04. ACM Press, Maryland, 3–10. https://doi.org/10.1145/1017833.1017834

Digital Library

[29]

Richard M Ryan and Edward L Deci. 2009. Promoting self-determined school engagement: Motivation, learning, and well-being.(2009).

[30]

Rustam Shadiev, Ting-Ting Wu, and Yueh-Min Huang. 2020. Using image-to-text recognition technology to facilitate vocabulary acquisition in authentic contexts. ReCALL 32, 2 (May 2020), 195–212. https://doi.org/10.1017/S0958344020000038

[31]

Andrew Trusty and Khai N. Truong. 2011. Augmenting the web for second language vocabulary learning. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, Vancouver, BC, Canada, 3179. https://doi.org/10.1145/1978942.1979414

Digital Library

[32]

Wen-Ta Tseng, Hao-Jyuan Liou, and Hsi-Chin Chu. 2020. Vocabulary learning in virtual environments: Learner autonomy and collaboration. System 88 (Feb. 2020), 102190. https://doi.org/10.1016/j.system.2019.102190

[33]

Daniel Ullrich, Andreas Butz, and Sarah Diefenbach. 2021. The Development of Overtrust: An Empirical Simulation and Psychological Analysis in the Context of Human–Robot Interaction. Frontiers in Robotics and AI 8 (April 2021), 554578. https://doi.org/10.3389/frobt.2021.554578

[34]

Candace Walkington and Matthew L. Bernacki. 2014. Motivating Students by “Personalizing” Learning around Individual Interests: A Consideration of Theory, Design, and Implementation Issues. In Advances in Motivation and Achievement, Stuart A. Karabenickand Timothy C. Urdan (Eds.). Vol. 18. Emerald Group Publishing Limited, 139–176. https://doi.org/10.1108/S0749-742320140000018004

[35]

Kathryn R. Wentzel and Allan Wigfield (Eds.). 2009. Handbook of motivation at school. Routledge, New York ; London. OCLC: ocn268957399.

[36]

S. Wilder. 2014. Effects of parental involvement on academic achievement: a meta-synthesis. Educational Review 66, 3 (July 2014), 377–397. https://doi.org/10.1080/00131911.2013.780009

[37]

L-H. Wong and C-K. Looi. 2010. Vocabulary learning by mobile-assisted authentic content creation and social meaning-making: two case studies. Journal of Computer Assisted Learning 26, 5 (Oct. 2010), 421–433. https://doi.org/10.1111/j.1365-2729.2010.00357.x

[38]

Andrew Wright, David Betteridge, and Michael Buckby. 2006. Games for Language Learning(3rd ed ed.). Cambridge University Press, Cambridge ; New York.

[39]

Lea Wöbbekind, Thomas Mandl, and Christa Womser-Hacker. 2021. Construction and First Testing of the UX Kids Questionnaire (UXKQ): A Tool for Measuring Pupil’s User Experience in Interactive Learning Apps using Semantic Differentials. In Mensch und Computer 2021. ACM, Ingolstadt Germany, 444–455. https://doi.org/10.1145/3473856.3473875

Digital Library

[40]

Yeshuang Zhu, Yuntao Wang, Chun Yu, Shaoyun Shi, Yankai Zhang, Shuang He, Peijun Zhao, Xiaojuan Ma, and Yuanchun Shi. 2017. ViVo: Video-Augmented Dictionary for Vocabulary Learning. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver Colorado USA, 5568–5579. https://doi.org/10.1145/3025453.3025779

Digital Library

[41]

Heather Toomey Zimmerman, Susan M. Land, and Yong Ju Jung. 2016. Using Augmented Reality to Support Children’s Situational Interest and Science Learning During Context-Sensitive Informal Mobile Learning. In Mobile, Ubiquitous, and Pervasive Learning, Alejandro Peña-Ayala (Ed.). Vol. 406. Springer International Publishing, Cham, 101–119. https://doi.org/10.1007/978-3-319-26518-6_4

Cited By

Leong JPataranutaporn PDanry VPerteneder FMao YMaes P(2024)Putting Things into Context: Generative AI-Enabled Context Personalization for Vocabulary Learning Improves Learning MotivationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642393(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642393
Fan MWu YFan JCui XLin S(2024)Character Alive: Designing and Evaluating a Tangible System to Support Children’s Chinese Radical and Character LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2313922(1-16)Online publication date: 18-Feb-2024
https://doi.org/10.1080/10447318.2024.2313922
Draxler FSchmidt AChuang L(2023)Relevance, Effort, and Perceived Quality: Language Learners’ Experiences with AI-Generated Contextually Personalized Learning MaterialProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596112(2249-2262)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596112

Index Terms

Auto-Generating Multimedia Language Learning Material for Children with Off-the-Shelf AI

Recommendations

Relevance, Effort, and Perceived Quality: Language Learners’ Experiences with AI-Generated Contextually Personalized Learning Material
DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference

Artificial intelligence has enabled scalable auto-creation of context-aware personalized learning materials. However, it remains unclear how content personalization shapes the learners’ experience. We developed one personalized and two non-personalized, ...
Charting unknown territory: models of participation in mobile language learning

Current language learning provision seems ill-suited to meet the needs of 21st century learners. There is a growing expectation that mobile language learning will offer greater flexibility and that it will be better aligned with lifelong learning and ...
Mobile-assisted language learning and pronunciation instruction: A systematic literature review
Abstract
In recent years, learners worldwide have expressed increasing interest in mobile learning and the utilization of mobile devices for language learning purposes. However, few research synthesis studies have focused on pronunciation development in ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MuC '22: Proceedings of Mensch und Computer 2022

September 2022

624 pages

ISBN:9781450396905

DOI:10.1145/3543758

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft
European Research Council
Saxony State Ministry of Science and Art

Conference

MuC '22

MuC '22: Mensch und Computer 2022

September 4 - 7, 2022

Darmstadt, Germany

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
177
Total Downloads

Downloads (Last 12 months)74
Downloads (Last 6 weeks)8

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Leong JPataranutaporn PDanry VPerteneder FMao YMaes P(2024)Putting Things into Context: Generative AI-Enabled Context Personalization for Vocabulary Learning Improves Learning MotivationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642393(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642393
Fan MWu YFan JCui XLin S(2024)Character Alive: Designing and Evaluating a Tangible System to Support Children’s Chinese Radical and Character LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2313922(1-16)Online publication date: 18-Feb-2024
https://doi.org/10.1080/10447318.2024.2313922
Draxler FSchmidt AChuang L(2023)Relevance, Effort, and Perceived Quality: Language Learners’ Experiences with AI-Generated Contextually Personalized Learning MaterialProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596112(2249-2262)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596112

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten