Facilitating the Learning Engineering Process for Educational Conversational Modules Using Transformer-Based Language Models
Pages 1222 - 1235
Abstract
In this article, we investigate a systematic workflow that supports the learning engineering process of formulating the starting question for a conversational module based on existing learning materials, specifying the input that transformer-based language models need to function as classifiers, and specifying the adaptive dialogue structure, i.e., the turns the classifiers can choose between. Our primary purpose is to evaluate the effectiveness of conversational modules if a learning engineer follows our workflow. Notably, our workflow is technically lightweight, in the sense that no further training of the models is expected. To evaluate the workflow, we created three different conversational modules. For each, we assessed classifier quality and how coherent the follow-up question asked by the agent was based on the classification results of the user response. The classifiers reached F1-macro scores between 0.66 and 0.86, and the percentage of coherent follow-up questions asked by the agent was between 79% and 84%. These results highlight, first, the potential of transformer-based models to support learning engineers in developing dedicated conversational agents. Second, it highlights the necessity to consider the quality of the adaptation mechanism together with the adaptive dialogue. As such models continue to be improved, their benefits for learning engineering will rise. Future work would be valuable to investigate the usability of this workflow by learning engineers with different backgrounds and prior knowledge on the technical and pedagogical aspects of learning engineering.
References
[1]
C. Von Aufschnaiter, S. Erduran, J. Osborne, and S. Simon, “Arguing to learn and learning to argue: Case studies of how students' argumentation relates to their scientific knowledge,” J. Res. Sci. Teach.: Official J. Nat. Assoc. Res. Sci. Teach., vol. 45, no. 1, pp. 101–131, 2008.
[2]
S. Erduran, S. Simon, and J. Osborne, “Tapping into argumentation: Developments in the application of Toulmin's argument pattern for studying science discourse,” Sci. Educ., vol. 88, no. 6, pp. 915–933, 2004.
[3]
C. Chin, “Teacher questioning in science classrooms: Approaches that stimulate productive thinking,” J. Res. Sci. Teach., vol. 44, no. 6, pp. 815–843, 2007.
[4]
P. Black and C. Harrison, “Feedback in questioning and marking: The science teacher's role in formative assessment,” Sch. Sci. Rev., vol. 82, no. 301, pp. 55–61, 2001.
[5]
S. E. Toulmin, The Uses of Argument. Cambridge, U.K.: Cambridge Univ. Press, 2003.
[6]
B. Mirzababaei and V. Pammer-Schindler, “Developing a conversational agent's capability to identify structural wrongness in arguments based on Toulmin's model of arguments,” Front. AI, vol. 4, 2021, Art. no.
[7]
I. Habernal and I. Gurevych, “Argumentation mining in user-generated web discourse,” Comput. Linguistics, vol. 43, no. 1, pp. 125–179, 2017.
[8]
R. Whately, Elements of Logic. London, U.K.: Longman, Green, Longman, Roberts and Green, 1897.
[9]
B. Mirzababaei and V. Pammer-Schindler, “Learning to give a complete argument with a conversational agent: An experimental study in two domains of argumentation,” in Proc. Eur. Conf. Technol. Enhanced Learn., 2022, pp. 215–228.
[10]
T. Wambsganss, T. Kueng, M. Soellner, and J. M. Leimeister, “ArgueTutor: An adaptive dialog-based learning system for argumentation skills,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2021, pp. 1–13.
[11]
W. Wang, D. Arya, N. Novielli, J. Cheng, and J. L. Guo, “ArguLens: Anatomy of community opinions on usability issues using argumentation models,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2020, vol. 20, pp. 1–14.
[12]
Z. H. Amur, Y. Kwang Hooi, H. Bhanbhro, K. Dahri, and G. M. Soomro, “Short-text semantic similarity (STSS): Techniques, challenges and future perspectives,” Appl. Sci., vol. 13, no. 6, 2023, Art. no.
[13]
A. C. Graesser, K. VanLehn, C. P. Rosé, P. W. Jordan, and D. Harter, “Intelligent tutoring systems with conversational dialogue,” AI Mag., vol. 22, no. 4, pp. 39–39, 2001.
[14]
A. Vaswani et al., “Attention is all you need,” in Proc. Int. Conf. Adv. Neural Inf. Process. Syst., 2017, vol. 30, pp. 6000–6010.
[15]
S. Zhang, R. Fan, Y. Liu, S. Chen, Q. Liu, and W. Zeng, “Applications of transformer-based language models in bioinformatics: A survey,” Bioinf. Adv., vol. 3, no. 1, 2023, Art. no.
[16]
B. Mirzababaei and V. Pammer-Schindler, “An educational conversational agent for GDPR,” in Proc. 17th Educ. New Future: Mak. Sense Technol.-Enhanced Learn. Adoption, 2022, pp. 470–476.
[17]
S. Demetriadis et al., “Conversational agents as group-teacher interaction mediators in MOOCs,” in Proc. Conf. Learn. MOOCS, 2018, pp. 43–46.
[18]
F. Weber, T. Wambsganss, D. Rüttimann, and M. Söllner, “Pedagogical agents for interactive learning: A taxonomy of conversational agents in education,” in Proc. 42nd Int. Conf. Inf. Syst., 2021, pp. 1–17.
[19]
D. B. Clark, V. Sampson, A. Weinberger, and G. Erkens, “Analytic frameworks for assessing dialogic argumentation in online learning environments,” Educ. Psychol. Rev., vol. 19, pp. 343–374, 2007.
[20]
F. Loll and N. Pinkwart, “Guiding the process of argumentation: The effects of ontology and collaboration,” in Proc. Conf. Connecting Comput.-Supported Collaborative Learn. Policy Pract., vol. 1, 2011, pp. 296–303.
[21]
T. Afrin, O. Kashefi, C. Olshefski, D. Litman, R. Hwa, and A. Godley, “Effective interfaces for student-driven revision sessions for argumentative writing,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2021, pp. 1–13.
[22]
R. Flesch, “Marks of readable style: A study in adult education,” Teachers College Contributions Edu., vol. 897, 1943, Art. no.
[23]
O. Scheuer and B. M. McLaren, “CASE: A configurable argumentation support engine,” IEEE Trans. Learn. Technol., vol. 6, no. 2, pp. 144–157, Apr.–Jun. 2013.
[24]
J. Lawrence and C. Reed, “Argument mining: A survey,” Comput. Linguistics, vol. 45, no. 4, pp. 765–818, 2020.
[25]
P. Poudyal, T. Goncalves, and P. Quaresma, “Experiments on identification of argumentative sentences,” in Proc. Int. Conf. Softw., Knowl. Inf., Ind. Manage. Appl., 2016, pp. 398–403.
[26]
C. Stab and I. Gurevych, “Identifying argumentative discourse structures in persuasive essays,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2014, pp. 46–56.
[27]
R. M. Palau and M. F. Moens, “Argumentation mining: The detection, classification and structure of arguments in text,” in Proc. 12th Int. Conf. AI Law, 2009, pp. 98–107.
[28]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. Conf. North Amer. Chapter ACL: Hum. Lang. Technol., 2019, vol. 1, pp. 4171–4186.
[29]
T. Wambsganss, A. Janson, T. Käser, and J. M. Leimeister, “Improving students argumentation learning with adaptive self-evaluation nudging,” Proc. ACM Hum.- Comput. Interact., vol. 6, no. CSCW, 2022, Art. no.
[30]
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[31]
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proc. Int. Conf. Adv. Neural Inf. Process. Syst., 2013, vol. 26, pp. 3111–3119.
[32]
T. Wambsganss, C. Niklaus, M. Söllner, S. Handschuh, and J. M. Leimeister, “A corpus for argumentative writing support in German,” in Proc. 28th Int. Conf. Comput. Linguist., 2020, pp. 856–869.
[33]
M. Xia, Q. Zhu, X. Wang, F. Nie, H. Qu, and X. Ma, “Persua: A visual interactive system to enhance the persuasiveness of arguments in online discussion,” Proc. ACM Hum.- Comput. Interact., vol. 6, no. CSCW 2, 2022, Art. no.
[34]
W. A. Abro, A. Aicher, N. Rach, S. Ultes, W. Minker, and G. Qi, “Natural language understanding for argumentative dialogue systems in the opinion building domain,” Knowl. Based Syst., vol. 242, 2022, Art. no.
[35]
M. Lippi and P. Torroni, “Argumentation mining: State of the art and emerging trends,” ACM Trans. Internet Technol., vol. 16, no. 2, pp. 1–25, 2016.
[36]
N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2019, pp. 3982–3992.
[37]
N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distillation,” in Proc. Conf. Empirical Methods Natural Lang. Process., 2020, pp. 4512–4525.
[38]
F. Loll, N. Pinkwart, O. Scheuer, and B. M. McLaren, “How tough should it be? Simplifying the development of argumentation systems using a configurable platform,” in Educational Technologies for Teaching Argumentation Skills. Sharjah, United Arab Emirates: Bentham Sci. Pub., 2012, pp. 169–197.
[39]
C. Schulz, S. Eger, J. Daxenberger, T. Kahse, and I. Gurevych, “Multi-task learning for argumentation mining in low-resource settings,” in Proc. Conf. North Amer. Chapter ACL: Hum. Lang. Technol., 2018, vol. 2, pp. 35–41.
[40]
A. Lauscher, G. Glavaš, and S. P. Ponzetto, “An argument-annotated corpus of scientific publications,” in Proc. 5th Workshop Argument Mining, 2018, pp. 40–46.
[41]
M. Specht, M. Kravcik, L. Pesin, and R. Klemke, “Authoring adaptive educational hypermedia in WINDS,” in Proc. Workshop Adaptivität und Benutzermodellierung Interaktiven Softwaresystemen, 2001, pp. 1–8.
[42]
P. Brusilovsky, S. A. Sosnovsky, M. Yudelson, and G. Chavan, “Interactive authoring support for adaptive educational systems,” in Proc. Conf. AI Educ., 2005, pp. 96–103.
[43]
A. Mannekote, M. Celepkolu, J. B. Wiggins, and K. E. Boyer, “Exploring usability issues in instruction-based and schema-based authoring of task-oriented dialogue agents,” in Proc. 5th Int. Conf. Conversational User Interfaces, 2023, pp. 1–6.
[44]
T. A. Van Dijk, “Text and context: Explorations in the semantics and pragmatics of discourse,” London, U.K.: Longman, 1977.
[45]
I. Wolfbauer, V. Pammer-Schindler, K. Maitz, and C. P. Rosé, “A script for conversational reflection guidance: A field study on developing reflection competence with apprentices,” IEEE Trans. Learn. Technol., vol. 15, no. 5, pp. 554–566, Oct. 2022.
[46]
D. H. Jonassen and B. Kim, “Arguing to learn and learning to argue: Design justifications and guidelines,” Educ. Technol. Res. Develop., vol. 58, pp. 439–457, 2010.
[47]
D. Long and B. Magerko, “What is AI literacy? Competencies and design considerations,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2020, pp. 1–16.
[48]
S. J. Russell, Artificial Intelligence a Modern Approach. London, U.K.: Pearson, 2010.
[49]
D. Adamson, G. Dyke, H. Jang, and C. P. Rosé, “Towards an agile approach to adapting dynamic collaboration support to student needs,” Int. J. AI Educ., vol. 24, no. 1, pp. 92–124, 2014.
[50]
R. Kumar, J. Beuth, and C. Rosé, “Conversational strategies that support idea generation productivity in groups,” in Proc. Conf. Connecting Comput.-Supported Collaborative Learn. Policy Pract., 2011, vol. 1, pp. 398–405.
[51]
C. Stab and I. Gurevych, “Parsing argumentation structures in persuasive essays,” Comput. Linguistics, vol. 43, no. 3, pp. 619–659, 2017.
[52]
M.-F. Moens, E. Boiy, R. M. Palau, and C. Reed, “Automatic detection of arguments in legal texts,” in Proc. 11th Int. Conf. AI Law, 2007, pp. 225–230.
[53]
T. Wambsganss, C. Niklaus, M. Cetto, M. Söllner, S. Handschuh, and J. M. Leimeister, “Al: An adaptive learning support system for argumentation skills,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2020, pp. 1–14.
Index Terms
- Facilitating the Learning Engineering Process for Educational Conversational Modules Using Transformer-Based Language Models
Index terms have been assigned to the content through auto-classification.
Recommendations
Conversational Recommender System Using Deep Reinforcement Learning
RecSys '22: Proceedings of the 16th ACM Conference on Recommender SystemsDeep Reinforcement Learning (DRL) uses the best of both Reinforcement Learning and Deep Learning for solving problems which cannot be addressed by them individually. Deep Reinforcement Learning has been used widely for games, robotics etc. Limited work ...
Vicarious learning from educational dialogue
CSCL '99: Proceedings of the 1999 conference on Computer support for collaborative learningThe Vicarious Learner project is investigating the role of dialogue in learning and, more specifically, how learners benefit from opportunities to 'overhear' other learners. We describe evidence of such learning by students using "vicarious learning ...
Comments
Information & Contributors
Information
Published In
© 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
Publisher
IEEE Computer Society Press
Washington, DC, United States
Publication History
Published: 20 February 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 11 Aug 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in