Its 2002 Workshop

ITS 2002 WORKSHOP Individual and Group Modelling Methods that help Learners Understand Themselves Chairs: Paul Brna, Northumbria University, UK Vania Dimitrova, University of Leeds, UK Workshop Web Site: http://www.cbl.leeds.ac.uk/~paul/its2002workshop http://computing.unn.ac.uk/staff/cgpb4/its2002workshop Programme Committee: Fabio Akhras, University of Sao Paulo, Brazil (fabio.akhras@poli.usp.br) Paul Brna, Northumbria University, UK (Paul.Brna@unn.ac.uk) Susan Bull, University of Birmingham, UK (s.bull@bham.ac.uk) Vania Dimitrova, University of Leeds, UK (vania@comp.leeds.ac.uk) Jim Greer, University of Saskatchewan, Canada (greer@cs.usask.ca) Judy Kay, University of Sydney, Australia (judy@cs.su.oz.au) Gord McCalla, University of Saskatchewan, Canada (mccalla@cs.usask.ca) Tanja Mitrovic, University of Canterbury, New Zealand (tanja@cosc.canterbury.ac.nz) Rafael Morales Gamboa, Instituto di Investigaciones Electricas, Mexico (rmorales@iie.org.mx) Patricia Tedesco, Universidade Federal de Pernambuco, Brazil (ptedesco@elogica.com.br) Julita Vassileva, University of Saskatchewan, Canada (jiv@cs.usask.ca) Preface Adaptive systems have a great deal of potential to encourage learners to engage in a variety of reflective activities. Developments in open learner modelling allow for the possibility of the system reflecting back the contents of the learner model to the learner. Developments in dialogue management allows for the system to take the learner’s expressed intentions into account through the use of intentional interfaces. Developments in analysing interactions with the system or with other users allows for the possibility of feeding the system’s interpretation of events back to the learner (or learners) so that they can take note in order to decide what to do. Developments in the use of agents allows for the learner model play an important role in obtaining assistance. Developments in affective computing allow for the learner’s affective state to be modelled and that model may have an increased role in helping learners understand themselves. Various architectures for individual and group modelling that help learners understand themselves have been explored. The common theme is to provide learners with a feedback mechanism which facilitates a variety of metacognitive activities. There is an increased recognition of the need to engage the learner in an interactive process with the possibility of the learner model being revised by either the learner or the system. This interactive process depends on formal models of belief revision as well as models of interaction such as those advocated by Baker, Pilkington, Suthers amongst others. Additionally the importance of work which seeks to open the learner (or group learner) model is underlined by the increasing need to respect various stringent data protection laws which become highly relevant once adaptive Web-based learning environments are deployed. 1 The notion of models of learners that can be examined and possibly revised has received increased attention since the Workshop on “Open, interactive and other overt approaches to learner modelling” held at the 9 Conference on Artificial Intelligence in Education at Le Mans, France th World in July 1999. (see http://cbl.leeds.ac.uk/ijaied/abstracts/Vol_10/modelling.html) The analysis of interactions allows for the detection of patterns which can be presented to learners to encourage a change in the pattern of interaction to improve learning. This includes analysing an individual’s interactions with a computer system and reasoning about the process of learning followed by changing the environment in ways that create possibilities for processes of interaction that increase learning over time. More recently, group interactions have been monitored in order to detect aspects of group behaviour which can be used for provoking reflection and articulation in group problem solving. The purpose of this workshop is to explore various issues concerned with the development of novel computational architectures for individual and group modelling that help learners understand themselves. Empirical investigations and experience with existing architectures that suggest advantages, indicate potential problems, and propose further extensions will be encouraged. Different perspectives on developing learning environments that build models of learners and support meta-cognition will be discussed. These may include educational theories that justify the advantages of such approaches as well as formal methods that support the application of the architectures in a variety of learning situations. The papers that have been collected together in this set of proceedings provide a basis for discussion at the workshop. The topics of these papers span issues in giving learners control in diagnosis, interactive cognitive modelling, interactive modelling of affect, maintaining distributed student models e.g. to help the learner examine and use multiple models of themselves, modelling groups of learners to increase group performance and improve learning, and using student and group modelling methods to promote meta-cognitive skills. Bull and Nghiem are taking a pragmatic approach to explore the issues in system-learner and learner-learner interactions based on simple domain independent models of learners. This work discusses the role of inspectable student models to promote meta-cognition in various classroom situations: the student model is inspectable by the student and promotes reflection on his/her learning; students can view models of their peers and compare their own progress against that of others; and finally, instructors may view the learner models of their students and use this to support the students in their learning. The paper by Andrade and her colleagues seeks to provide a diagnostic agent that advises other agents about what to do at a task level or at a group recombination level conditioned by the generalised notion that a group of learners has a ZPD. The work combines cognition with some affective and motivational elements within an agent-based approach. They seek to use learner provided information and allow learners to modify this if required. Dufresne and Hudon seek to use humour in interactions with learners and make the learner's preferences about humour visible and changeable. In these two contexts, the learner's preferences are considered to be accurate and learners have ultimate control - unlike the systems developed by Dimitrova and Bull which are both based on the more generalised notion that the system can argue with the learner on some matters. Dufresne and Hudon follow a principle that people draw upon their experience in real interactions when participating in computer mediate interactions They argue that affective tutors (tutors that exploit humour are discussed here) may provide a more effective learning environment. The paper prompts an interesting research issue of how to employ learner control in the design of affective interactive tutors. There are many opportunities to encourage reflection within the context of working with simulations. Grigoriadou, Samarakou, Mitropoulos and Panagiotou provide a simulation environment that seeks to diagnose student misunderstandings and misconceptions and show this diagnosis in some form to the learners. Chesher, Kay and King provide a simulation environment encouraging professionals to manage multiple cases and consider how well they are doing. These two somewhat different environments (learning elementary physics and learning within a professional context) both offer excellent test beds for whether seeing the system's (possibly erroneous) judgement has any important positive effect on learning. 2 Finally, the paper by Dimitrova focuses on the student-computer interaction in inspectable student models. It seeks to provide intelligent support to develop an extended dialogue with learners where the learner and the system discuss the content of the learner model and it is through this interaction that reflective learning may take place. The paper discusses how far a combination of knowledge based systems and an improved model of dialogue can be used to help the learner argue about, and reflect on, the learner's own beliefs as reflected to them by the system presenting its model of the learner's beliefs. Paul Brna and Vania Dimitrova 3 Index Helping Learners to Understand Themselves with a Learner Model Open to 5 Students, Peers and Instructors Susan Bull, Theson Nghiem A Diagnostic Agent based on ZPD approach to improve Group Learning 14 Adja F. de Andrade, Paul Brna, Rosa Maria Vicari A Web-based Medical Case Simulation for Continuing Professional Education. 26 Douglas Chesher, Judy Kay, Nicholas King Knowledge-based Fuzzy Evaluation of Learners in Intelligent Educational Systems 32 Maria Grigoriadou, Maria Samarakou, Dionissis Mitropoulos, Michael Panagiotou Modeling the learner preferences for embodied agents : experimenting with the 43 control of Humor. Aude Dufresne, Martin Hudon Interactive Cognitive Modelling Agents - Potential and Challenges 52 Vania Dimitrova 4 Helping Learners to Understand Themselves with a Learner Model Open to Students, Peers and Instructors Susan Bull & Theson Nghiem Electronic, Electrical and Computer Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, U.K. s.bull@bham.ac.uk Abstract: This paper introduces work in progress on an open student model designed to help learners to better understand their learning. The aim of the system is to investigate issues relevant to open learner models in a large-scale, real use context. This will be achieved initially through the deployment of a simple, domainindependent system. The student model is inspectable by the student it represents, to help focus reflection on their learning. Students may also view the models of their peers, to enable them to compare their own progress against that of others. Furthermore, instructors may view the learner models of their students, to help them support students in their learning. The initial version of the system and the learner model are very simple, to enable early deployment in a variety of contexts. (Subsequent investigations may lead to more detailed recommendations for specific domains.) Planned extensions to enable student-system and peer-peer collaboration with reference to student models (based on existing work on other systems), are also discussed. Key words: open learner models, learner reflection. 1. Introduction There has recently been growing interest in opening the learner model to the individual it represents, with several systems demonstrating this approach (e.g. Bull & Pain, 1995; de Buen et al., 1999; Dimitrova et al., 2001; Kay, 1997; Morales et al., 2000; Silva et al., 2001; Specht et al., 1997; Weber et al., 2000). An important reason for rendering the learner model accessible is to help the student to better understand their learning opening the learner model to the modellee offers a source of information about their relationship with the target domain which is otherwise unavailable, encouraging them to reflect on their beliefs and on the learning process. Student reflection on their learning has also been encouraged in collaborative learning situations, where pairs of students view their own and each other's learner models, in order to provide a focus for collaborative interaction (Bull & Broady, 1997). Making peer models available more generally could also be beneficial as it would enable students to compare their progress against that of their peers, as proposed by Kay (1997). In addition, suggestions have been made to open learner models to instructors, allowing tutors to access information about those they teach. This can help instructors to adapt their teaching to the individual or to the group (Grigoriadou et al., 2001; Zapata-Rivera & Greer, 2001); or enable them to suggest suitable peer helpers, or organise learning groups (Mühlenbrock et al., 1998). Each of the above approaches to open learner modelling (self-access, peer access and instructor access to an individual's student model) can be used to help learners to better understand their learning of a target domain. However, one of the difficulties in effecting learner reflection in this manner on a wider scale, is that typically systems which have inspectable student models as a focus for encouraging reflection are quite complex, with the system requiring an understanding of the target domain in order to model in some detail, a student's knowledge, and infer their misconceptions, in that area. While this results in potentially very effective methods of promoting 5 learner reflection, it also renders the systems relatively expensive to implement, and often restricted to a single or limited set of domains. To complement such approaches, we suggest employing a very simple student model in a system that can be easily deployed in a variety of course types, with the aim of investigating the potential of open learner modelling in a range of realistic settings, with large numbers of users. While this more straightforward approach to student modelling will not allow the system to adapt its tutoring to specific misconceptions held by specific individuals, it will nevertheless allow investigations of broader questions such as whether students will pay attention to their learner models in a variety of disciplines. Some investigations have suggested that even when they know about the inspectability of their own student model, learners do not necessarily attempt to view it (Barnard & Sandberg, 1996; Kay, 1995). Nevertheless, small-scale studies have suggested that students might indeed use an open student model if it were available, in contexts where they may discuss the contents of their model with the system (Bull & Pain, 1995; Dimitrova et al., 2001). It is worthwhile, therefore, investigating whether the disinterested reactions from some students in the former situations are typical, or whether, as suggested in some other studies, students might find open learner models helpful to their learning at least in some contexts. Furthermore, it would be useful to discover whether instructors in a range of subjects find open learner models a useful way of helping them to recognise student difficulties, enabling them to respond to specific student populations or individuals in appropriate ways. With this as one of the system aims, the limitations in its ability to offer fine-grained adaptive interaction to an individual student, based on the student model, are less crucial. Hence the system is intended more as a practice environment, than a tutoring system. Another issue that can be investigated with simple open learner models is the means of externalising these to students, peers and instructors. As argued by Morales et al. (2000), presentation modality might be important for the comprehensibility of student models at least in some domains. Similarly, different individuals might work better with different model representations, thus even within a single domain it may be useful to offer alternative or complimentary learner model representations (e.g. Bull, 1998). Furthermore, Zapata-Rivera & Greer (2001) add that the goals of the system may influence the manner in which student model data should be presented. This paper describes work in progress on a learner model open to the student it represents, their peers and their instructor. A student inspecting their own student model can benefit through reflection on its contents; viewing peer models can enable learners to understand their progress against the context of the advancement of their peers; and allowing teachers to view individuals' learner models can help them to help those specific learners individually, and also adapt their teaching to the group's difficulties, where common problems arise. The learner model presented here is very simple - in its basic version containing representations only of a student's level of knowledge of various topics. While necessarily restricting the system's ability to adapt to an individual's misconceptions, it does allow it to be straightforwardly deployed more widely (as a practice environment) and, as stated above, the involvement of the teacher (or peers) can compensate for the system's inability to interact with reference to specific misconceptions. Moreover, multiple choice exercises can be created in text files, further allowing the system to be easily deployed in a variety of subjects, by a variety of tutors. The simplicity of both the student model and the method of exercise creation permits investigation of the potential of open student models to help learners to better understand themselves, enabling some of the general questions pertinent to inspectable student models to be investigated in a variety of contexts. Section 2 introduces the system. Section 3 describes the inspectable learner models from three perspectives: inspection by the student (modellee); inspection of the models of peers; and inspection of learner models by the instructor. In each case, the potential for helping learners to better understand themselves, is discussed. Conclusions are presented in Section 4. 2. The General Learning Environment The system is suitable for use in domains where multiple choice questions are appropriate as a means of practising the target material. The system presents ten questions at a time, each with a drop-down box where the correct answer should be selected. Figure 1 illustrates the interface for the domain of Japanese particles, where the grammatically correct particle has to be chosen for each of the sentences. 6 Figure 1: Multiple choice practice questions Tutors can advise students to use the system to practise material presented during classroom sessions, using textbooks or course notes for reference. Alternatively, instructors can add their own instructional materials, to be accessed from within the system. Questions for each concept (in this example, Japanese particles) are selected randomly. Candidate concepts are selected according to the student's previous performance. The aim is to provide students with opportunities to practise the areas with which they are experiencing most difficulty, thus each individual will receive exercises targeted in particular at a restricted set of concepts which are represented as least understood in their student model. (During the initial interaction, questions on all concepts are presented.) Standard questions can be loaded for specific domains. New questions can also be added by instructors, thus for particularly complex concepts many practice questions can be made available. These can be divided into various levels of difficulty, to enable the same instantiation of the system to be used by students at a range of levels. It also allows for the same concepts to be practised in different contexts within the same general domain - for example, Japanese particles in business or conversational Japanese. Moreover, it allows the system to be deployed in a range of subjects - not only different languages, but any domain for which appropriate multiple choice practice questions can be created. New questions are simply added to existing text files, or for new concepts (in the same or a new domain), new text files can be created. Thus the system can be easily deployed in a variety of courses, to investigate the research questions. 3. The Open Learner Model The basic student model is a very simple, numerical model. The total number of questions attempted for each concept is stored, as is the proportion of correct versus incorrect attempts. Greater weighting should be awarded to later answers to enable the system to use this information more accurately to infer where the student's current difficulties lie. (The weighting algorithm for the student model has not yet been implemented.) As described 7 below, these student models can be extended in particular cases, but for the simplest version of the system, the student model is a straightforward representation of knowledge levels in the various areas. The learner model is open to the individual student, to encourage them to reflect on their knowledge and misconceptions. Students can also access the models of peers, to help them further gauge their progress. Instructors can view the various learner models to enable them to better help individuals; to adapt their teaching for a particular group; or to help them set up optimal peer learning or tutoring groups. 3i. A Learner Model Open to the Student As described above, the student model contains representations of the student's performance on sets of questions related to different concepts in some domain. Remaining with the example of Japanese particles, Figure 2 illustrates the way in which the model is externalised to the student. Figure 2: The open learner model Currently the student model can be presented to the learner in one or both of two forms: tabular and graphical. The graphical representation is similar to the skill or knowledge meters of APT (Corbett & Bhatnagar, 1997), ADI (Specht et al., 1997) and ELM-ART (ELM Research Group, 1998), which display bars where the filled portions represent the student's attained knowledge or skill against the expert level; and OWL (Linton & Schaefer, 2000), which displays in bar form a user's knowledge level against the combined knowledge of other user groups. In contrast to the above, our system illustrates a comparison of correct versus incorrect attempts at questions. While, as pointed out by Linton and Schaefer, the skill meter approach focuses on achievements rather than shortcomings, our system aims to raise learner awareness of their performance in general - i.e. not only the degree to which they have mastered any particular aspect of the domain, but also concepts with which they are having difficulty (in contrast to areas they have not yet attempted). Other external representations for the student model will be considered at a later date. Nevertheless, as it is knowledge level that is represented, rather than specific concepts and misconceptions, these representations will likewise not be complex. 8 At present, a score of 1 is awarded for each correct answer for a given concept, and 1 is subtracted for an incorrect response. This can be seen in the tabular representation in Figure 2. (Representations are updated when students complete a set of ten questions.) The overall score, illustrated in both the tabular and the graphical representation implementation of of the the student model, mechanism to indicates award the greater student's weighting overall to later performance. answers is (As still stated above, required. Once implemented, the student model will better reflect the student's actual current knowledge levels.) The student can use the externalised learner model to see at a glance areas in which they are strongest and weakest, and use these as recommendations of what should be studied further, within or outside the system. If instructors choose to add domain content or explanations of common misconceptions to a particular instantiation of the system, these can be linked to the student model, and examined by the student. It can be seen that the information shown to students about themselves, is limited. An extension to the system, allowing the learner to negotiate the contents of their student model, is planned. This is based on the MR COLLINS approach of collaborative student modelling (Bull & Pain, 1995), where the student can argue with the system if they disagree with the contents of their learner model. Negotiating the content of the learner model is designed in part to enhance learner awareness of their learning. In the case of the new system, such negotiation will be less complex, focusing on the system offering the student a quick test if they claim to know something that they have not yet adequately demonstrated (with the system accepting the student's claim if they can demonstrate their knowledge in the test); and allowing the learner to inform the system if they believe its representations of their knowledge level of any concept is too high (for example, if they had guessed the answer to some of the multiple choice questions, and by chance got them correct; or if they have simply forgotten previously known material). This will enable the student to influence their learner model, and thereby also influence the subsequent selection of practice material. 3ii. A Learner Model Open to Peers In addition to inspecting their own learner model, a student can compare their performance to that of peers in their group (this occurs anonymously unless students choose to reveal their usernames to others). This enables learners to appreciate how their developing understanding of the target concepts compares to that of other learners, as suggested by Kay (1997). Students can retrieve the student models of good peers, to appreciate the level of knowledge for which they could aim; and also of weaker peers, which in some cases could help learners to realise that they are performing better than they had realised. Figures 3 and 4 show a comparison of the student models of different individuals. Figure 3: Comparing student models (graphical) 9 Figure 4: Comparing student models (tabular) A planned extension to the current system will also allow learners to compare their student model against the 'average' student model of all other learners (or a subset), in their cohort. The possibility of viewing peer models, in addition to enabling students to gauge their progress against other learners, also allows a student to collaborate with another learner, using their respective student models as a focus for discussion. In 2SM (Bull & Broady, 1997), displaying two student models to their co-present owners was designed to prompt peer tutoring, with each student individually completing an exercise, and then coming together to repeat the same exercise in the presence of their two inspectable learner models. In the new system, a pair (or small group) of learners can also come together, viewing their own and their partner's student models, and learning from collaborative interaction. In contrast to 2SM, each learner will have experienced a different set of questions tailored to their own needs, and the combined exercise will again be different. The pair will commence a fresh session, building a new student model reflecting their joint performance, drawing on the representations in their individual learner models to recognise whose explanation is more likely to be correct, in cases where they disagree about an answer when collaborating about their joint answers. (The system provides correct answers for student consultation after an exercise has been completed. Thus, should the pair agree on an incorrect answer, they are made aware of any inaccurate selection, and can see what the correct choice should have been.) The collaboration and peer tutoring expected to occur with the comparison of learner models (see Bull & Broady, 1997), will focus students' attention more directly on their knowledge and misconceptions. An interesting observation will be how students decide on the representation type to view, if their preferences differ. A planned extension to the current version of the system is to allow it to use the two student models of a collaborating pair of learners, to adapt the joint exercise to best utilise the relative strengths and weaknesses of the pair. 10 With the extensive sets of questions possible, students can potentially work with a variety of peers on a set of concepts if they prefer to learn with others. They can use the student models of others to help them select suitable collaborative partners or helpers who complement their own abilities. An extension to the system could even enable it to suggest suitable learning partners based on the contents of the various participants' learner models (see e.g. Collins et al., 1997). 3iii. A Learner Model Open to the Tutor The final use for the open student model is to help tutors to better understand their students. In the same way that students can compare their learner model to the models of others, teachers can do likewise with their students' learner models. This can allow instructors to help individuals with particular problems, or enable them to better target their teaching to the general difficulties of specific groups. The possibility of viewing a combined 'average' student model (see 3i) would facilitate the latter process. This is almost the reverse of the original aim of intelligent tutoring systems - to teach students as a human tutor might, using knowledge about a student contained in their student model. Here the student model is providing information for the human teacher, helping them to adapt their teaching appropriately. Although not part of the initial planned investigations, teachers could also view models in order to form learning groups (as in Mühlenbrock et al., 1998). Teachers might ultimately be able to provide some additional information to learner models based on, for example, student performance on assignments, and engage in system-mediated discussion with a student, about their learner model (see Bull, 1997). In the first deployment three instructors will use the system in three different courses: Japanese, physics and interactive systems. 4. Summary This paper has described a system with an open learner model designed to be viewed by the student modelled, their peers, and their tutor. The student model therefore has an even more central role than in the traditional intelligent tutoring system - as a learning resource for the student, to help them reflect on their beliefs in a student-system setting; as a means to prompt collaboration and peer tutoring; and as a source of information for instructors to aid the human teaching process. The system is currently in the early stages of development. While it could now be deployed in its most simple state, a few extensions will increase its utility as a focus for promoting learner reflection on their understanding. These extensions include a simple form of collaborative student modelling, and taking into account pair models to further support peer collaboration. Most importantly, weightings will be applied to the most recent answers given, in order to ensure that the student model appropriately updates as the student learns. Even with these extensions, the system will remain straightforward. It is a practice environment rather than a tutoring environment - though the instructor may provide links to course materials. Its primary aim is to investigate some of the questions relevant to the use of inspectable learner models in a range of contexts - both different subject areas, and different students. Such questions include the method by which learners prefer to view (and later interact with) their learner model - the individual's preferred method of accessing the learner model is likely to be more relevant than, for example, the influence of the domain, as it is knowledge level (rather than actual knowledge or misconceptions), that is represented. The goal here is to raise learner awareness of where their strengths and weaknesses lie, thus representations similar to those illustrated, could be usefully considered. Other issues include how much attention students pay to their own and to peer models; whether interaction with the learner model occurs with some students more than others; with some domain types more than others; or whether there are differences in the same student's acknowledgement of their learner model in different domains. Also relevant are instructors' use of their students' models. It is acknowledged that a more complex student model is ultimately likely to be most helpful, in particular in the context of the individual student viewing their own learner model. However, to investigate questions such as the above, it is useful to employ a simple learner model in an easy to deploy system. Of course, more complex 11 student models will have additional requirements for their externalisation - these can be investigated in parallel, or once some of the simpler questions have been addressed. An obvious question is how to show larger models the current type of display, while clear for small domains, may need to be modified if many interrelated concepts were to be included in a single exercise set. Nevertheless, even with our initial straightforward approach, it is intended that students will be able to benefit from interactions with the system, and that some initial guidelines about the effective externalisation of learner models can be obtained. References Barnard, Y.F. & Sandberg, J.A.C. (1996). Self-Explanations, do we get them from our students?, in P. Brna, A. Paiva & J. Self (eds), Proceedings of European Conference on Artificial Intelligence in Education, Lisbon, 115-121. Bull, S. (1997). See Yourself Write: A Simple Student Model to Make Students Think, in A. Jameson, C. Paris & C. Tasso (eds), User Modeling: Proceedings of the Sixth International Conference, Springer, Wien New York, 315-326. Also available from: http://w5.cs.uni-sb.de/~UM97/abstracts/BullS.html Bull, S. (1998). 'Do It Yourself' Student Models for Collaborative Student Modelling and Peer Interaction, in B.P. Goettl, H.M. Halff, C.L. Redfield & V.J. Shute (eds), Intelligent Tutoring Systems, Springer-Verlag, Berlin, Heidelberg, 176-185. Bull, S. & Broady, E. (1997). Spontaneous Peer Tutoring from Sharing Student Models in B. du Boulay & R. Mizoguchi (eds), Artificial Intelligence in Education, IOS Press, Amsterdam, 1997, 143 - 150. Also available from: http://www.eee.bham.ac.uk/bull/papers/AIED97.htm Bull, S. & Pain, H. (1995). 'Did I say what I think I said, and do you agree with me?': Inspecting and Questioning the Student Model, in J. Greer (ed), Proceedings of World Conference on Artificial Intelligence in Education, Association for the Advancement of Computing in Education (AACE), Charlottesville, VA, 1995, 501-508. Also available from: http://www.eee.bham.ac.uk/bull/papers/AIED95.htm Collins, J., Greer, J.E., Kumar, V.S., McCalla, G.I., Meagher, P. & Tkatch, R. (1997). Inspectable User Models for Just-In-Time Workplace Training, in A. Jameson, C. Paris & C. Tasso (eds), User Modeling: Proceedings of the Sixth International Conference, Springer Wien New York, 327-337. Also available from: http://w5.cs.uni-sb.de/~UM97//abstracts/CollinsJA.html Corbett, A.T. & Bhatnagar, A. (1997). Student Modeling in the ACT Programming Tutor: Adjusting a Procedural Learning Model with Declarative Knowledge, in A. Jameson, C. Paris & C. Tasso (eds), User Modeling: Proceedings of the Sixth International Conference, Springer Wien New York, 243-254. Also available from: http://w5.cs.uni-sb.de/~UM97/abstracts/CorbettA.html de Buen, P.R., Vadera, S. & Morales, E.F. (1999). A Collaborative Approach to User Modeling within a MultiFunctional Architecture, in J. Kay (ed), UM99: User Modeling, Proceedings of the Seventh International Conference, Springer Wien New York, 291-293. Also available from: http://www.cs.usask.ca/UM99/Proc/short/BuenRodriguez_032414.pdf Dimitrova, V., Self, J. & Brna, P. (2001). Applying Interactive Open Learner Models to Learning Technical Terminology, in M. Bauer, P.J. Gmytrasiewicz & J. Vassileva (eds), User Modeling 2001: 8th International Conference, Springer-Verlag, Berlin Heidelberg, 148-157. ELM Research Group (1998). ELM-ART Lisp Course. http://www.psychologie.uni-trier.de:8000/elmart Grigoriadou, M., Papanikolaou, K., kornilakis, H. & Magoulas, G. (2001). INSPIRE: An Intelligent System for Personalized Instruction in a Remote Environment, in P. De Bra, P. Brusilovsky & A. Kobsa (eds), PreWorkshop Proceedings: Third Workshop on Adaptive Hypertext and Hypermedia, User Modeling 2001, 3140. Also available from: http://wwwis.win.tue.nl/ah2001/papers/papanikolaou.pdf Kay, J. (1995). The UM Toolkit for Cooperative User Modelling, User Modeling and User Adapted Interaction 4, 149-196. Kay, J. (1997). Learner Know Thyself: Student Models to Give Learner Control and Responsibility, in Z. Halim, T. Ottomann & Z. Razak (eds), Proceedings of International Conference on Computers in Education, Association for the Advancement of Computing in Education (AACE), 17-24. 12 Also available from: http://www.cs.su.oz.au/~judy/Research_um/index.html Linton, F. & Schaefer, H-P. (2000). Recommender Systems for Learning: Building User and Expert Models through Long-Term Observation of Application Use, User Modeling and User-Adapted Interaction 10, 181207. Morales, R., Pain, H. & Conlon, T. (2000). Understandable Learner Models for a Sensorimotor Control Task, in G. Gauthier, C. Frasson & K. VanLehn (eds), Intelligent Tutoring Systems: Proceedings of the 5 th International Conference, Springer-Verlag, Berlin, Heidelberg, 222-231. Mühlenbrock, M., Tewissen, F. & Hoppe, H.U. (1998). A Framework System for Intelligent Support in Open Distributed Learning Environments, International Journal of Artificial Intelligence in Education 9(3-4), 256-274. Silva, A., Vale, Z.A. & Ramos, C. (2001). Inspectability and User Controlled Revision on Long Term User Models, in M. Bauer, P.J. Gmytrasiewicz & J. Vassileva (eds), User Modeling 2001: 8th International Conference, Springer-Verlag, Berlin Heidelberg, 254-256. Specht, M., Weber, G. & Schoech, V. (1997). ADI: Ein adaptiver Informations- und Lehragent im WWW, in R. Schaefer & M. Bauer (eds), Proceedings of ABIS'97: Workshop Adaptivitaet und Benutzermodellierung in interaktiven Softwaresystemen, Universitaet des Saarlandes, Saarbruecken, 53-60. Weber, G., Kuhl, H-C. & Weibelzahl, S. (2001). Developing Adaptive Internet Based Courses with the Authoring System NetCoach, in P. De Bra, P. Brusilovsky & A. Kobsa (eds), Pre-Workshop Proceedings: Third Workshop on Adaptive Hypertext and Hypermedia, User Modeling 2001, 41-53. Also available from: http://wwwis.win.tue.nl/ah2001/papers/GWeber-UM01.pdf Zapata-Rivera, J-D. & Greer, J.E. (2001). Externalising Learner Modelling Representations, Proceedings of Workshop on External Representations of AIED: Multiple Forms and Multiple Roles, International Conference on Artificial Intelligence in Education 2001, 71-76. 13 A Diagnostic Agent based on ZPD approach to improve Group Learning 1 2 Adja F. de Andrade 3 Paul Brna School of Computing Rosa Maria Vicari PGIE- UFRGS School of Computing and Mathematics University of Leeds Caixa Postal: 5071 University of Northumbria, NEI 8ST - UK LS2 9JT - UK Zip Code: 90041-970 Paul.Brna@unn.ac.uk adja@cbl.leeds.ac.uk Porto Alegre-RS rosa@inf.ufrgs.br Abstract: This paper explores the question of cognitive diagnosis based on the analysis of ZPD (Zone of Proximal Development) based on a Vygotskian perspective and Core concepts. The main aim is to present a new proposal of architecture, to model the diagnostic agent’s behaviour, describing some “scaffold tactics”. The paper also presents a scenario example to illustrate this proposal. This research demonstrates that the possibility of using skills, competencies and difficulties to parameterise the formation of a group can become an interesting approach for cognitive diagnosis. Although, we are conscious of the fact that to form groups does not guarantee that the learning will be better, we believe that it permits social interaction and exchange of knowledge, important pre-requisites in the group learning. Keywords: Diagnosis, Modelling Group, Zone of Proximal Development. Introduction If we don’t know each other, how can we interact? If we don’t know our own skills and difficulties, how can we share them, request or offer helping? The solution is not in building the “perfect” learning environment. But, perhaps, an environment able to describe our cognitive model and share this model with our peers. Perhaps, we are inserting a high responsibility for the technology. This certainly seems an ambitious goal regarding the actual “pedagogical role” of the technology. Motivated for this challenge, this paper presents the first sketches of the cognitive diagnosis using a Vygotskian perspective. The main aim is to present a new proposal of the architecture for the diagnostic agent and to model his behaviour. To explain this proposal, we begin classifying two important parameters: ZPD and Core. The first parameter is based on Vygotsky’s theory. This concept defines what we are not able to do alone but we can do with some scaffold. The second parameter identifies what we know, that is, which are our skills -Core level. The concept of core was firstly found in the work of Hansen (Hansen, 1999). Recent research (Kay, 2001b) has shown a tendency towards using the ZPD concept to build the learner model. In our approach, we will use the conception of ZPD to parameterise the formation of group model and not only a model of learner. About the kind of model utilised, we are investigating how to build an open learner model, where the student will be able to intervene in his own diagnostic process. The group model is inspired by some attributes presented in Paiva’s work (Paiva, 1997). Based on Luckin’s work (Luckin, 1996, 1999) we describe the use of the ZPD. 1 PhD Student in PGIE- Federal University of Rio Grande do Sul (UFRGS). Lecturer at the Pontifical Catholic University of Rio Grande do Sul Brasil (PUCRS). Currently in School of Computing at University of Leeds. 2 3 Supervisor in Leeds-UK. Supervisor in Graduate Program in Computer Science in Education-PGIE, Brasil. 14 The research problem refers to investigate the diagnosis process able to detect the zone of proximal development of the learner in a distance learning context. The motivation to use this approach is founded in Vygotsky theory. He defends that learners can improve their learning through mediated interaction with more capable peers. However, to detect the skills that needs support, firstly, we have to diagnostic them, and furthermore to identify the possible “scaffold tactics” adequate to the level of the learner. The approach adopted is based on the analysis of the beliefs and observation of the task done by the group. This task is monitored by a mediator agent and sent to the diagnostic agent. The proposal analyses the self-confidence model of the learner, which represents a kind of self-evaluation made by the learner. The evaluation of the diagnosis will be a dynamic evaluation, with focus in zone of proximal development of the learner. The first step in the evaluation process is to analyse the individual performance of the task, the second moment this evaluation is compared to the performance of the group, which involves mediation with other learners more able in this knowledge domain. The focus of our analyses privileges the process of interaction and not only the result (product) of task realised. One of the big difficulties to describe the diagnostic agent is due to need to interact with other agents (mediator, semiotic, collaboration), exchange message, register and evaluate the behaviour of the group to determine the diagnosis and take some support decision. This is a distributed task, that request a complex architecture involving communications protocols, access to group models, task performance module and still maintaining the individual characteristics of the learners. The actual stage of this PhD research is the formalisation and modelling of the behaviour of the diagnostic agent and definition of the diagnosis process. After this stage, we will start the implementation and the experimental study case. For the implementation of the diagnosis process, we are investigating several approaches based on model, mentalist, stereotype, overlay, bugs, and misconception approaches. Until the present moment the intentional approach based on the mental state seems to be the most adequate for our work. Although, the ZPD approach is a consolidated concept presented in Vygotsky’s theory and recently has been used by several researchers as a diagnostic function. It is important to highlight, even thought, we still do not have any concrete result, which would make a pedagogical validation possible to underline the significance of this approach. The Society of Pedagogical Agents Pedagogical Agents (Gürer, 1998) are described as intelligent agents that have an educational or pedagogical role to facilitate or improve learning. These agents can be modelled as personal and animated agents that interact with the user, or as cooperative agents who work in one background as part of the architecture of the educational system. The society of agents, a multiagent system is formed by four agents: Diagnostic, Mediator, Collaboration and Semiotic. This architecture is part of the project “A Computational Model of Distance Learning Based in the Socio-Cultural Approaches” (Andrade, 2001)(Jaques, 2002). The Mediating Agent is the interface agent of the society. The Diagnostic Agent has the function of diagnosis and updates the information in the group model, besides sending “scaffold tactics” to the Mediating Agent. The Semiotic Agent is a service agent that has the function to search for content in the knowledge base and send to the Mediating Agent to be shown to the student. The Collaboration Agent is an animated character agent who has the role of searching partners capable of assisting one student in his/her learning and to mediate the interaction between students using collaborative communication tools. The architecture of the multiagent system is shown in figure 1 below. 15 Semiotic Agent Request Pedagogical Content WWW Exercises Access Knowledge Examples Base Request Pedagogical Agents that aid in Student ZPD Content Pedagogical Tactic Student Behavior Diagnostic ... Mediating Agent Agent Cognitive Group Profile Model Diagnostic Mediating Agent Student Agent Model Actions Affective tool) Students Group (Collaboration Mediating Agent Profile User Pedagogical Content Pedagogical Tactic Pedagogical Content Student Message Message to the student Collaboration Group Group Model Agent Message to the student Student Student Profile Figure 1. Multiagent architecture (Andrade et al, 2001) Diagnostic Agent A Diagnostic Agent has the function of describing the cognitive diagnosis, modelling the group and suggesting “scaffold tactics”. Initially, when the task is proposed for the group, the diagnostic agent must create a mechanism to evaluate which skills are in the core region and which are in the ZPD region. To start the diagnosis process, the diagnostic agent must propose some task to the learner or group varying the degree of difficulty. This is represented as an action that is translated to communication speech act (FIPA, 1997): act (propose-task (TID, task(name(-)))) where task is associated to a definition as task (name(-), [skill-list]). The response expected for this message is a message of the sort: task-outcome (TID, [[core-skill], [ZPD-skill]]) where TID is a task identification. A task may have many instances, i.e., it may occur several times. These messages are generates by diagnostic agent and sent to the mediator agent, which provide adequate assistance for skills in ZPD. In figure 2, we can see the internal architecture of the Diagnostic Agent. 16 BDI Model Desires, Intentions and plans Beliefs Practical Reasoning Theoretical Reasoning Open Learner model Cognitive diagnosis Group Model Open Mediator Diagnosis Learner Agent model Validation of diagnosis Knowledge Update Module Open Learner Tactic of scaffold Scaffold model Module Message of Learner behaviour Teacher or Sensor Learner behaviour Module Other Level Mediator of support Tactics of Scaffold Message of tactic of scaffold Effec tor diagnose Colla borat Figure 2. Diagnostic agent architecture The Sensor Module is an internal part of the diagnostic agent. He has the role of communicating with mediator/collaboration agents. This module has the objective to interpret the message that arrives from the mediator and collaboration agents. The messages are described in ACL format (Fipa, 1997) and represent the input of the diagnosis process. The ACL is a communication language that permits one agent to send a demand to another agent to perform some task. The messages are received through the sensor and sent to the communication module for the disencapsulation or separation of messages in blocks. The next phase is the treatment or interpretation of this message. When the diagnostic agent needs send a new message, he has to set the encapsulation status to re-start the process again. This process is needed because the diagnostic agent needs to communicate with other agents in the architecture, and he is not able to make any decision about the diagnostic without this interaction. The choice of ACL format is because the message need to be stored in an adequate structure that becomes possible the agent to understand your content. One example of this message is outline follows: (inform :sender Mediator :receiver Diagnostic :content <create-web_page> :ontolgy <exercise> :reply-with <profile_interaction> ) 17 The Effector Module collaboration agents. is responsible to send messages with “scaffold tactics” to mediator and Besides, this module informs the diagnostic of the learners for the other agents for that they can facilitate the learning. The Open Learner Model is formed by the cognitive and emotional profile of the learner. The cognitive profile stores the information about beliefs, skills, difficulties and assistance. The emotional profile contains information about the personality of the learner, like introvert, extrovert, if he/she likes to work in group, and his/her level of motivation. These parameters are used as first reference, however they can change during the interaction. This model is considerate as an open learner model because it is inspectable by the learner (Dimitrova, 2001) (Kay, 2001b), which he/she can analyse and agree or disagree about his/her diagnosis. The Group Model is formed during the evaluation of the learner’s ZPD. When the diagnostic agent discovers that the learner has some skill in ZPD, he suggests forming a group with some expert that has knowledge in that domain area and can help that student. It maintains not only a cognitive status, but also an affective profile of the group sent by the collaboration agent. The Knowledge Update Module updates the agent beliefs about the learner’s performance, goals and skills. The Scaffold Module is formed by description of task, list of skill in ZPD, level of support (low, moderate, advanced) and the tactics suggested. The tactics of scaffold (for instance modelling, start solution, give clues) before to be applied must be observed the level of knowledge of the learner. The tactics have the role to help the students to perform some task, which skills are in ZPD level. These tactics are sent through the diagnostic agent to the mediator agent and the collaboration agent (Jaques, 2002) that interact directly with the learner. Diagnosis is the main module of the architecture. Its function is to diagnose what is in ZPD or Core level. The diagnosis starts when the diagnostic agent suggests some task to be performed by the group without any support. After, the group accomplished this task, the agent assess the task performance model and self-confidence model, which describe the level of knowledge and confidence of the learner to realise the given task. In function of this analysis the agent determine the ZPD_skills, in other words, the skills that the learner can not carry out alone and need some “tactics of support”. For the pedagogical validation of the diagnosis, the agent must communicate with the teacher. The diagnosis must be updated also in the group model. The Theoretical-Reasoning Module represents the agent’s beliefs. These beliefs modelling the knowledge about the domain (based on the BDI description to be described in the next section). However, Beliefs and desires are not enough to implement the behaviour of agent. They need of a “plan of actions” to achieve the goal and desires. The Practical Reasoning Module represents the planning module, in other words, this module describes the agent’s reasoning about what it should do. Diagnostic Agent Modelled with BDI Architecture When we desire to describe the human behaviour is common to use terms like “believe”, “want”, “desire”, “need”, etc. These terms are used by beeing human to explain the observable proprieties of the mind. With the objective to fundament these intentional explanations, also denominated intentional systems, we describe one set of mental states, which will represent the behaviour of the diagnostic agent. In the literature, a number of different approaches have emerged to modelling agent-oriented learning environment. The mentalist approach (Bratman, 1990) allows view the system as a rational or cognitive agent having certain mental attitudes of Belief, Desire and Intention (BDI), representing respectively, the information, motivational and deliberative state of the agent. Our goal is to model the diagnostic agent using a human metaphor. In the same way, the knowledge of learners also will be modelled through your beliefs and intentions. The choice to using the BDI approach is to facilitate the modelling of the diagnostic agent. Several intelligent tutoring systems have adopted this approach. But other approaches also could be used for 18 this function. BDI is only one form to describe the behaviour of diagnostic agent and your knowledge. It enables us to look at the importance of the social context and the believes, desires and intentions of the diagnostic agent involved. Nevertheless, the focus this research is not the formalisation in BDI, but the idea of ZPD to describe the diagnosis process. The beliefs represent the knowledge of the world. They are some way of representing the state of the world. BEL (a, p) means that the belief p has been ascribed to the agent a by some learner. The mental state Belief is visualised as the most basic epistemic concept usually considered in the form of preposition. Desires (DES) are commonly thought like other essential component of system state. They represent the motivational state of the system. Intentions are representations of possible actions (or chosen course of action) that the system may take to achieve its goals. The intentions capture the deliberative component of the agent. The diagnostic agent has two main desires: diagnostic and support. The first desire has the objective to identify the skills learned (skill-cor) and the skills that need support (skill-zpd). To achieve this desire the agent has the belief that the learner’s skills will be in ZPD level when the learner request help or the task is not performed with success. The second desire is to support the group using some strategy of managing of group. The agent will use the merge strategy when he desires join one learner (L1), expert in some knowledge domain (skill-cor), with some learner novice (L2) that has difficulties and his/her skills are in ZPD. One example of the beliefs and desire bases is outlined follows: DES (Diag, diagnostic_skills) BEL(Diag, skill_ZPD) if BEL(Diag, skill_cor) if BEL (Diag, diagnostic_skills) if BEL (Diag, next (BEL (group, BEL (Diag, next (BEL (group, BEL (Diag, skills_ZPD) or request_ajuda)) not-request_ajuda)), BEL (Diag, skills_cor) BEL (Diag, next (BEL BEL (group,competence_task_is competence_task_is_successful) unsuccessful)) BEL (Diag, next (BEL (group, (Diag, next (BEL (group, confident_knowledge)) DES (Diag, support_group) DES (Diag, support_group) DES (Diag, support_group) BEL (Diag, group_merge) if BEL (Diag, group_split) if BEL (Diag, group_cor) if BEL (Diag, skill_ZPD_L1) BEL (Diag, evaluate_subgroup), BEL (Diag, skill_cor_G1) Overlap with BEL (Diag, BEL (Diag, skill_cor_L1) overlap with (Diag, skill_cor_L2) BEL (Diag, skill_cor_L2) AND skill_cor_G2) Table 1. Beliefs and Desire base Formalisation of Core and ZPD The notion of the zone of proximal development (ZPD) was proposed by Vygotsky- as "the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers" (Vygotsky, 1978). The concept of core is mentioned in Lewis’ work (Lewis, 2000), where he mentions that “the knowledge of an individual has a central core that is “owned” by the individual who is able to use that knowledge in the autonomous performance of tasks”. The core i s a subset of the domain, which represents the knowledge internalised (learned) by the learner or group. Definition 4.1 (Domain): The domain is a structure < D, pre>, where: • D is a set of sentences in the form skill(name(…), type, pedagogical_goal, [support_list],context) and Pre is a relation D x D stating that a given skill is a pre-requisite of some other skill. 19 Definition 4.2 (core): Given a Knowledge Domain <D, pre>, the core of a learner is a structure <Cor,L> where: Cor • • ⊆ D. Cj ∈ Cor ∧ (Ci , Cj ) ∈ pre then Ci ∈ Cor If Ci, Cj are elements of core. The ZPD is also a subset of domain that describes skills that are not internalised, i.e., skills that the learner does not have yet, but that he/she can perform with some support or scaffold. Hansen defines ZPD as: “Surrounding that core is in ZPD region, in which the individual has some knowledge but not the full structure of capacities required, and thus needs help in performing tasks that depend upon that knowledge”(Hansen, 1999). Definition 4.3 (ZPD): Given a knowledge Domain <D, <ZPD,L> pre>, the ZPD of a learner L is a structure where: ZPD is a subset of D. The definition of core and ZPD (Dillenbourg, 1992b) are practically the same. In fact, the nature of the propositional content of the core and ZPD are very similar, although dynamic. In a given moment a skill can be in the core and in another can be is ZPD. The skill can move from ZPD to core (internalisation) or from core to ZPD (externalisation). core knowledge zone of proximal development individuals groups Figure 3. Core knowledge and zones of proximal development for individual and group (from Hansen et al, 1999) Figure 3 above shows the representation of core and ZPD. When a community or group is considered, some parts of each person’s core knowledge overlap with that of others. Besides this, that one person’s ZPD may overlap with the core of others as well. From this model, one might conclude that the collective core, union of cores, is greater than that of an individual; but also that each person can support cognitive development in the group by providing scaffolding for others (Wood, D., Bruner, J.C., & Ross, G, 1976). Group Diagnosis The Student Model is one of the important component in a traditional ITS. It is a collection of data about the knowledge’s of the student and is used by other components of ITS (in our case by others agents) to planning some sequence of instruction, feedback, explanation or scaffold. Sometimes the term student 20 model is called student diagnosis in the literature. For a literature review of cognitive diagnosis we suggest (Self, 1994) (Dillenbourg and Self, 1992a) and (Ragnemalm, 1996). Student diagnosis (Ragnemalm, 1996) is defined as “the abstract process of gathering information about the students and turning that information into the basis for instructional decisions”. In this paper we use the term group diagnosis for the process of cognitive modelling of the group, because we are interesting in the behaviour of the group and not only of the learner. As well as the student, group diagnosis has a similar purpose on how to adapt the tutoring, how to provide explanation or clues to provide the right level of coaching in the group. The group model is inspired by Paiva’s work (Paiva, 1997), where the notions of belief, action and group skills are discussed. This model is formed by a collection of data about the knowledge of the student and is used by other components of the environment (in our case by other agents) to plan some sequence of instruction, feedback or scaffold (support). The group model is also inspired by the work of Luckin (Luckin, 1996). The model known as VIS- The Vygotskian Instructional System - selects some factors that combined represent the learner’s ZPD. This model can be extended as for individual as groups, considering individual as instance of groups. The main attributes proposed are described as follows: group beliefs, social context of interaction, group skills, motivational and emotional characteristics, group difficulties and group relationship (i.e. Assistance required and offered). The approach used to evaluate the learner’s performance is based on Kay’s work (Kay, 2001a) through the use of stereotypes like novice, intermediate and expert. In her work, the author describes the essential elements of a stereotype, namely triggers, inferences and retraction. The stereotype approach seems an appropriate way to establish the initial model of the learner, before the diagnosis agent starts the interaction activity. It is important to notice that the use of these stereotypes will not be static. On the contrary, we are proposing a dynamic and ever changing configuration, where one learner can be in a given moment a novice at one task and in another task have an expert status. Continuing to follow the notion of stereotype, we adopt the community notion presented in Kay’s work. This notion adds some significant advantages to our proposal of group modelling because the data collected for a large number of users becomes feasible for the learners to classify their own stereotype inside the community. This community notion states that the learner can share some model of preferences and helps to facilitate the performance assistance in the learning process. The user-adapted interaction allows the system to “be able to cater for a diverse user population, with different users having different needs, preferences, interests, background, and so on” (Kay, 2001a). Some characteristics of the learner/group can also show some means or measure for evaluation if the skills are or are not in ZPD level. Some measures chosen for our evaluation are confidence, competence, motivation and help parameters of the learner in their interaction with learning environment (see section 3.1). The confidence level of the student refers to a belief of the learner about some knowledge of their level of performance in doing some task. The competence means the ability of the learner or group to perform some task successfully. The motivation is an important parameter in an interaction context. Sometimes, the learner can not perform some task or skill because he/she is not motivated, or there is no empathy between the members of your group. Finally, the capacity of helping means the capacity of the learner to accept and offer support. If a learner ignores help, and still cannot perform at the higher level of the ZPD as expected, then the agent needs to rethink the support. Perhaps the skill is outside this learner's ZPD, or the assistance provided is not useful and should be modified. The Scaffolding Process The “scaffolding” term was coined by Bruner (Wood, D., Bruner, J.C., & Ross, G, 1976) to specify types of assistance that make it possible for learners to function at the ZPD level. “Scaffolding” is currently used to describe how a more able mediator (other student, teacher, computational agent, etc) can facilitate the learner’s transition from assisted to independent performance. The Support (or scaffold) is a kind of assistance offered to the learner to perform some task that is at the ZPD level. This support is applied according to the level of the learner with relation to a given domain knowledge. The selection of support is based on the notions of stereotype and community 21 described by Kay (Kay, 2001a). Analysis of the group level, the agent that will perform the diagnosis detects which learners can be classified as novice, intermediate or expert. The support is associated with activities and actions. Each support must have a name, parameters, level and tactics and is represented as: support(name(…),level,tactic) where level is either low, moderate, or advanced, and tactic is one of the pedagogical strategies of support. When a task is selected is offered assistance to achieve a solution. This assistance also change with parameters of ZDP. Luckin (Luckin, 1996) argues that the “ZPD metric is the deciding factor in the choice of type and amount of teaching input”. We can notice that the ZPD is something dynamic that has been used for pedagogical decisions and scaffold and must be updated with the learner’s performance. In the perspective of assistance, the task is categorised initially into non-scaffold and scaffold approach. When a task is selected is offered assistance to achieve the solution. The assistance also change with the level of the learner (see table 2). a) The non-scaffold approach. The learner must realise the activity without any support. The goal is to identify, which abilities of the learner are at the ZPD level. When the agent starts the mediation process he has to consult the model of the learner to know about his background and competence in this domain. If he starts the scaffold approach without consulting this model, he can propose some task too hard and destroy the confidence of the learner, or to suggest some task very easy what can frustrate him/her. So, the support must be effectuated gradually of accord of learner’s level. b) The scaffold approach includes using some “scaffold tactics” to support the activity and discussions about how and when providing some help. We can use different tactics in several levels (see table 2). The assistance can be also interpreted as scaffold tactics, a “step-by-step” formation, where through mediation activity the gradual transfer of responsibility is transferred from the mediator for the learner. In the modelling of our interface, the own learner or group can help the diagnostic agent in the task of diagnosis. For this was planning a self-explanation space used by the learner to express their private speech about the interaction (Moll, 1990) or to share the understanding of the group, one kind of summary of group beliefs. There are three different levels of support. The low level is adequate for the group that needs maximum assistance, generally in the start of the activity. The moderate level is usually suggested in the middle of the learning process, during the performing of task. The advanced level is more used when the student has a high level of confidence of their knowledge and is able to express or explain their reasoning. The table below shows the group’s support level and the respective relation with some tactics for assisting performance. Self-explanation Level of support Tactics Private speech (individual Low Modeling commentaries) Start the solution Give clues Group beliefs Moderate Explanatory (commentaries of the group) Advanced Cognitive structuring Questioning Table 2. Ontology of tactics The agent is responsible to apply the tactics, but the learner or group must be able to choose which tactics they prefer as support. The modeling tactic is the process of offering behaviour for imitation or analogy, it is used when the group needs some example, tutorial or some demonstration before starts the activity. The start solution is used in the begin of the task, when the learner needs some initial support. To give clues is 22 an interactive approach, applied with the goal to guide a learner to improvement in performance on the next try. It must be used when the group has some difficulties and requests some information, the agent can send some clues to help, these tactics permit the learners fill gaps in their reasoning process. The explanatory tactic is employed when the group needs some explanation more textual or explanatory and not only tips. Cognitive structuring is more used at an advanced level, when the agent requests the group to structure their thinking or action, to summarise concepts, schemas or describe some mental operation. The questioning tactic is generally used to inquire mental operation and calls for a cognitive response, this tactic evaluate if the skill’s group is really in core level (learned). For a full exposition of the means of assisting performance see Moll (Moll, 1990). The group model described in this paper is being built using two perspectives: focus on the group beliefs and focus on the nature of assistance. This model looks at the interactions of the learners with the environment and the evolution in time, meaning that the beliefs and knowledge are subject to more frequent changes, where different tactics can be used as well as several support levels to improve the collaborative learning process. A Scenario Example In this section, we describe a scenario example to illustrate the process of diagnosis and support. We desire that the reader builds a mental picture of the system “in action” that helps to understand our proposal. The scenario is focusing on the role of the diagnostic agent and his interaction with other agents in the multiagent society. The context is distance learning, the task is to create a WebPage and the domain is the Internet. Firstly, one teacher asks the student or a group of students to create a webPage (task) in some appropriate software. The group begins to explore the software, without any scaffold (support). Some functions are known, others not, this will show us that only some skills are in the Core level. The mediator agent will be observing the behaviour and beliefs of this group, asking questions about the task. For example, if the group knows how to insert pictures in the page. Analysing the task performance and the level of confidence/motivation of the learner (see parameters in section 5), the mediator agent is able to identify which are the group’ difficulties and which part of the process they need some support. After the mediator agent monitor the task, he must send a message with the performance of group trough some protocol message ACL to the diagnostic agent. This message contains the identification of the group, task, assistance required, offered, overall performance, skills are not performed, cause of error, misconceptions, profile of interaction. In this moment, diagnostic agent analyses the difficulties of the group and he requests to the semiotic the agent some content (examples, concepts, etc) that can help the group. Continuing the process, the diagnostic agent sends the diagnosis result to the mediator agent informing that the group is in ZPD level and suggest some tactic of scaffolding, for instance, provide examples, clues, request some content about how to insert hyperlinks for example. In the same moment, an other agent, the collaborative agent is looking in the Internet for partners, who are more capable peers to support the group in this task. How does he do this task? The Collaborative agent communicate with the personal agents (diagnostic agent of the others students) to know about the cognitive model of other learners. Before the application the scaffold tactic, the diagnosis agent must communicate with the teacher for the final validation. After the implementation of the scaffold strategy (show example how to insert hyperlinks), the mediator agent must inform the diagnostic agent about the self-confidence evaluation of the group. Subsequently the diagnostic agent must update the group model with respect to the behaviour for this task. The model of the group will be formed for several attributes: group identification, group beliefs (general knowledge about the domain), group relationship (difficulties to interact), profile of interaction, the common interests (community notion), skills-core (abilities consolidate), like for instance change colours, insert separators, change colour of background. Beside, the skills-ZPD, abilities that need support like for instance (insert figures and hyperlinks), support required, support offered (no), cognitive diagnostic hypothesis. 23 Claim Create a webPage collaboratively Support Provide examples, to make questions Because Several students can collaborate around the internet Check-rule Evaluate the creation the WebPage by task analysis, Evaluate the level of To give tips or slots confidence of learner, Evaluate skill_core (learned) Evaluate skill-ZPD (need support) Open Issue Some conflicts between goals individual and collective Table 3. A scenario example Conclusions This research has examined how Intelligent Learning Environments can provide useful diagnostic information. The arguments presented consider the socio-interactionist approach of Vygotsky, at the level of pedagogic model, and using the agents' paradigm, at the level of computational model. This paper has provided the first sketches about cognitive diagnosis using the concepts of the ZPD and core. The status of this work is that the diagnostic process has been defined and is being formalised in a BDI Model (Bratman, 1990). The next step is the definition of requirements to describe the interface for the diagnostic agent. There is a lot of research to explore the diagnosis domain, specially, because this subject involves the question of “evaluation”, “support” and modelling of the learner, important discussion point in the ITS community. We parameterise the have seen formation that of the the possibility group can of be using an skills, interesting competencies approach to and build difficulties group to model. Although, we are conscious that to form groups does not guarantee that the learning will be better, we believe that it permits social interaction and exchange of knowledge, important pre-requisites in the learning process. In short, we described the learning environment like an efficient assistance channel. We believe that the question involving the designer of environments that effectively promote the support and collaboration between the learners, can be a gap with complexity and relevance to be explored in a thesis work. As a future result, we are optimists that the ideas presented in this work shows perspective to increase the group performance and improve learning. Acknowledgements This research is supported by Brazilian Government (CNPq/CAPES) and Northumbria University. Special thanks to Michael C. Mora and anonymous reviewers, all of whom provided many constructive comments and appointed open questions and problems on previous version of this paper. References Andrade, A. F.; Jaques, P. A.; Vicari, R. M.; Bordini, R.; Jung, J. (2001). A Computational Model of Distance Learning Based on Vygotsky's Socio-Cultural Approach. Proceedings of the MABLE Workshop (Saint Antonio, Texas, May 19-23). X International Conference on Artificial Intelligence on Education. Bratman, M (1990). What is Intention? In: Intentions in Communication. P.R. Cohen et al (Eds). MIT Press. Dillenbourg P., Baker M., Blaye A. and O’Malley C. (1994). The evolution of Research on Collaborative Learning. In Spada and Reimann (Eds.) Learning in Humans and Machines. Dillenbourg, P. and Self, J. A.(1992a). A Framework for learner modelling. Interactive Learning Environments, 2(2):111-137. 24 Dillenbourg, P. (1992b). The Language Shift: A Mechanism for Triggering Metacognitive Activities. In Adaptive Learning Environments Foundations and Frontiers. M. Jones and P. Winne (Eds.) Springer-Verlag, vol.85, 287-315. ISBN: 3 54055459-9. Dimitrova, V., Self, J. and Brna, P. (2001). Applying interactive open learner models to learning technical terminology, Proceedings of the 8th Conference on User Modelling, Berlin:Springer, LNAI. FIPA Specification. (1997). http://www.fipa.org/ Giraffa, L. M. M., Vicari, R. M.(1998). The use of agent techniques on Intelligent Tutoring Systems. In: 18th International Conference of Chilean Computer Science Society. Gürer, D. (1998). The Use of Distributed Agents in Intelligent Tutoring. Proceedings of Workshop On Pedagogical Agents, ITS’98 (San Antonio) p.20-25. Hansen, T; Holmfeld, L; Lewis, R.; Rugelj, J (1999). Using Telematics for Collaboration Knowledge Construction. In Collaborative Learning: cognitive and computational approaches. P. Dillenbourg (Eds.) Elsevier Science Ltd., Oxford, pp. 169-196. Jaques, A. P. Andrade, A. F.; Vicari, R. M.; Bordini, R.; Jung, J (2002). Pedagogical agents to support collaborative distance learning. Proceedings of Computer Support for Collaborative Learning CSCL2002, Colorado. http://newmedia.colorado.edu/cscl/275.html. Kay, J (2001a). User Modelling for Adaptation. In User Interfaces for all: concepts, methods and tools. Constantine Stephanides (Ed.), Lawrence Erlbaum Associates, Inc. EUA, pp. 271-293. Kay, J (2001b). Learner Control. In User Modelling and User-Adapted Interaction, Kluwer Academic Publishers, Netherlands, vol. 11, pp. 111-127 Lewis, R. (2000). Human activity in learning societies. Invited Paper. Proceeding of ICCE’00/ICAI, Taipei, Taiwan. Luckin, R. (1996). Trivar:Exploring the “zone of proximal development”. Proceedings of European Conference on Artificial Intelligence in Education. In P.Brna, A. Paiva, J. Self (Eds.) Colibri, Lisbon, 16-22. Luckin, R., & du Boulay, B. (1999). Capability, Potential and Collaborative Assistance. In User Modelling: Proceedings of the Seventh International Conference, UM99 J. Kay (Eds). Springer Wien New York, pp. 139-148. Moll, L. C. (1990). Vygotsky and Education: instructional implications and applications of sociohistorical psychology. Cambridge University Press, pp 177-183. Mora, M. C. (2000) Um Modelo Formal e Executavel de Agente BDI. PhD thesis, Porto Alegre: CPGC/UFRGS. Paiva, A.(1997) “Learner Modelling for Collaborative Learning Environments”, in Proceedings of the AI-ED 97 World Conference on Artificial Intelligence in Education. B. du Boulay & R. Mizoguchi (eds.), pp. 215-222. Ragnemalm, E. L. (1996) Student diagnosis in practice; Bridging a gap. User Modelling and User Adapted Interaction, 5(2):93-116. Self, J. A. (1993). Model Based Cognitive Diagnosis In User Modelling and User-Adapted Interaction, 3, 89-106. Self, J. A. (1994). Formal approaches to student modelling. In Greer, J. E. and McCalla, G. I., editors, Student Modelling: The Key to Individualised Knowledge-Based Instruction, NATO_ASI Series F, 295-352. Springer-Verlag. Vygotsky, L.S. (1978) Mind in society: The development of higher psychological processes. (M.Cole, V.J.Steiner, S.Scribner, & E. Souberman, Eds.) Cambridge, MA: Harvard University Press. Wood, D., Bruner, J.C., & Ross, G (1976). The role of tutoring in problem solving. Journal of Child Psycology and Psychiatry, 17, 89-100. 25 A Web-based Medical Case Simulation for Continuing Professional Education Douglas Chesher*, Judy Kay#, Nicholas King* *School of Pathology and #School of Information Technologies University of Sydney, Australia 2006 dougc@med.usyd.edu.au, judy@it.usyd.edu.au, nickk@pathology.usyd.edu.au Abstract: This paper describes the Simprac system, a simulation-based tool to support long-term learning by medical professionals and students in the area of management of chronic illness. Since it is intended to support independent learning, one of the important aspects of its design is the inclusion of several opportunities for learners to reflect on their knowledge and learning. This paper gives an overview of the system, and the motivation underlying its design. We then describe the opportunities it provides for learner reflection. One of these supports reflection at the end of each simulated consultation, with comparison of the learner’s performance against that of experts and peer groups. Another part supports reflection at the end of a whole case. Keywords: life-long learning, reflection, medical education Introduction In the rapidly changing field of health care, like many other professions, medical practice involves learning through the whole of working life (Brna P, 2000). Traditionally, this has involved modalities such as rounds, educational meetings, conferences, refresher courses, programs, seminars, lectures, workshops, and symposia (Davis D, 1999). However, in their review of the published literature on continuing medical education, Davis et al (1999) found that traditional formal didactic continuing medical education (CME) had little influence on physician behaviour and concluded that, "where performance change is the immediate goal of a CME activity, the exclusively didactic CME modality has little or no role to play". They did, on the other hand, find some evidence that, "interactive CME sessions that enhance participant activity and provide the opportunity to practice skills can effect change". This has led us to explore the potential role of simulation-based teaching systems to support long-term learning for medical practitioners. In the design of such system, we were influenced by the evidence cited by Anderson, Reder and Simon (Anderson JR, 1995). This suggests that optimal learning occurs when a combination of abstract and situation-specific training is provided and that abstraction promotes the transfer of knowledge and thus insight from one situation to another. In the context of a simulation-based learning system, this means that we should take care to go beyond a pure simulation of one or more cases. We should ensure that the fundamental design of the system includes elements which encourage the practitioner to reflect on each case, evaluate their performance in meaningful ways and abstract broadly from the particular learning experience provided by each simulation in order for knowledge transfer between similar or related cases to occur effectively. While most computer simulations have been developed with an emphasis on medical diagnosis and have generally involved a single patient encounter, much patient morbidity is associated with the diagnosis and management of chronic disease such as diabetes mellitus and cardiovascular disease. These chronic disorders evolve over time and involve multiple doctor – patient encounters. The simulation we have been developing explores this chronic disease model by enabling the user / health professional to review the patient over a number of consultations with the patient outcome for the following consultation being defined by the management strategy chosen by the practitioner. 26 Within a simulation, there is considerable scope for various styles of tutoring. For example, one might follow the model of Lajoie, Faremo and Wiseman (2001) in studying expert human tutoring as a foundation for the design of a teaching system. We have taken a rather different approach in creating a framework which can capture the diagnostic and management actions of a range of users, from novice to expert. From this, we can construct models of the expert’s behaviour as well as a variety of group models for learners at various levels, such as medical students and practicing general practitioners. The merit of a range of such models is that each can serve different purposes. For example, all learners would aspire to achieve expert levels of performance (and their patients would support this standard as the goal!). However, this might be an unrealistic and unfair comparison for medical students who would more reasonably compare themselves against their fellow medical students. It might also be valuable for them to compare their actions against the model for general practitioners since senior students should be aspiring to reach that level in the near term. Our goal is to design a simulation that includes natural points for reflection. That reflection can be based on the learner’s analysis of their actions. Importantly, it can also be supported by the presentation of comparisons between the learner’s actions and those in the expert model or one of the group models. Related Work Patient simulations have been used in teaching as well as assessing “clinical competence”. Simulations involving trained actors can provide the closest approximation to the real patient-doctor relationship. These have the potential to provide both verbal and non-verbal cues and can be accurately depicted in a standardised manner. Actors have been used in research into medical problem solving by clinicians (Elstein ES, 1978) but this form of simulation is clearly impractical for continuing education purposes. It seems desirable to build simulation-based systems which can offer some of the advantages of such human-actor-based simulations. Computer-based simulations have become increasingly common for both teaching (Hayes KA, 1996; Bryce DA, 1997) and assessment (Clyman SG, 1990). These are appearing as stand-alone simulations available on a CD-ROM as well as on the web. To-date, the web-based cases (Hayes KA, 1996) have been considerably (Friedman CP, less 1995) sophisticated has provided than an those excellent using CD-ROM outline of the (Bryce features DA, to be 1997). Friedman considered when developing a computer based clinical simulation. Combining this with the description by Melnick (Melnick DE, 1990) of the system used by the American National Board of Medical Examiners (NBME), a minimum feature set can be developed. Below is a compilation of the ideas and description by Friedman and Melnick. 27 Menu vs. Natural Using natural language creates a higher fidelity simulation and avoids Language Requests for cueing the student but requires more sophisticated programming. On the Data other hand, some users may become frustrated with trying to make their requests understood. With recent advances in computing power and language processing offering a natural language interface is becoming more practical. Interpreted vs. Again this can vary considerably between programs. One can provide the Uninterpreted Clinical raw results such as chest x-rays and pathology results without comment or Information one call provide a text report. Alternatively there may be some combination of the two. Deterministic vs. With deterministic progression, taking the same action always leads to the Probabilistic Progression same clinical result. In the probabilistic approach each action is associated with a set of probable outcomes and each outcome has a medically realistic probability of occurring. While the latter approach has the potential for a much richer simulation environment the initial development of the program is far more complicated. With computer technology, it is now feasible to generate cases from a knowledge base that stores information about the prevalence of disease and the probability of specific findings in the presence of that disease. Natural Feedback vs. Instructional intervention comes in at least two forms. The first is to provide Instructional Intervention feedback to the student on their progress through the case compared to some desired optimum. The second method requires that the student relate diagnostic hypotheses to clinical findings. This type of feedback has been criticised for being distracting and imposing a reasoning framework that is foreign to the student. Natural feedback involves the realistic progression of the case. If appropriate action is taken the patient's health improves. In contrast, if inappropriate action is taken the health of the patient will deteriorate and may necessitate action to restore the patient to good health. Scoring Many programs assess participants by; determining the errors of omission and errors of commission and derive indices from these. Single vs. Multiple Most computer simulations have been based on single patient encounters. Encounters Table 1 : Features of Computer Based Medical Case Simulations We will refer to these in the description of the design of our system. Overview of System We illustrate the operation of our system in terms of a case we have developed. This is an example of a reasonably challenging simulated task in diagnosis and management of hyperlipidaemia, which is a major independent risk factor for the development of cardiovascular disease. An overview of the consultation sequence for this first case is shown in Figure 1. The case has been chosen to require management over a period of several consultations which would typically run over six months. Notably, there is a real possibility that plausible but incorrect treatment would cause serious problems for the patient. This case is also interesting because it involves use of Fibrate, a drug which was once prescribed only by specialist practitioners. Recent changes mean that it is now widely prescribed by general practitioners. This is a classic case of the type of long-term workplace-based learning that arises in the medical profession. 28 Red: No action or cornstarch alone. Green: Fish oil alone or with cornstarch. Initial Blue: Statin alone or in combination with cornstarch or fish oil. First Presentation Purple: Fibrate alone or in combination with any other treatment. Consultation Thick line represents preferred path. First episode Further First episode pancreatitis. in triglyceride No change Predominant reduction in reduction in cholesterol in triglyceride Mild reduction from baseline [1] Predominant Mild reduction from baseline [1] Law suit. [2] Mild Reduction No Change pancreatitis. pancreatitis + No Change from baseline in triglycerides Second Consultation triglycerides Predominant Predominant reduction in reduction in cholesterol Predominant Predominant reduction in reduction in cholesterol triglycerides Side effect of fibrate. [3] triglycerides Third Consultation Fourth Consultation [1] Pancreatitis occurs if there have been two consecutive consultations without effective treatment. [2] Negligence if the patient has suffered pancreatitis and no effective treatment commenced. [3] Pancytopenia develops if fibrate used on two consecutive consultations. Figure 1: Overview of four consultation sequence in simulation This case involves four consultations. In the first, the medical practitioner takes a patient history and performs initial examinations. From these, a management action is proposed. For example, the optimal management action calls for a Fibrate-based treatment. The second consultation, illustrated by the four boxes in the second row, provides the practitioner with the outcome of their first management regime. As Figure 1 indicates, it is possible at this stage to move from a sub-optimal situation to the optimal one. (It is also possible to move away from the optimal situation.) This case exemplifies the tightly connected influence of the various management options with the ongoing disease outcomes. Most of the states at each consultation can lead to any of the states in the subsequent consultation, depending upon the treatment option chosen. The diagnostic component is supported through the user being initially presented with a short case vignette after which they have the option of • taking a medical history from the patient, • performing a physical examination, • ordering investigations, or • selecting management options. Questions are asked using free text entry and a series of “matching” questions are returned, that once selected, will elicit an appropriate response from the patient. The practitioner can perform a virtual physical examination by selecting a variety of “tools” or actions and applying these to different parts of the body ( Figure 2). A investigations, wide as is variety often of the investigations case with can clinical be requested, practice, will not however, always the be results available of the until a predetermined time later in the case. For example, the results of a CAT scan will not be available until the next consultation. 29 Figure 2: Sample examination screen Since an essential element of the system is to support learner reflection, there are two levels of review. At the first level, the user is able to review their progress at the end of each consultation. This enables the user to evaluate the questions they have asked, examinations, investigations and management options they have chosen. They can simply review this as a basis for reflecting about what they have done in this consultation. Our system has also been designed so that the user will be able to compare their own performance with their peers or an expert in the domain. The peer data will be built up over time as more and more individuals with that professional background attempt the case. The second level of review occurs once a case is completed and enables the user to assess their treatment path and patient outcomes through the case, again with the ability to compare their performance with both the optimum, as defined by the authors of the case, and against the performance of their peers. Summary and current status We have described the motivation for our design of a simulation-based learning environment designed to support the long-term learning of medical practitioners. We have also outlined the main elements of the current prototype and provided a high level view of one case that has been implemented. There are several distinctive aspects of the work. First, it tackles the domain of learning of long-term medical management in the context of chronic disease. This includes the need to perform diagnosis but it also requires the choice of treatment options which influence subsequent patient presentations. The second distinctive aspect is the focus on learner reflection, with learners being encouraged to review their performance and then compare it with various benchmark levels of performance: peer groups, more expert groups and the approach considered optimal by the designers of the case. The current prototype encodes the above case. However, it also implements generic elements for building further cases. These include the initial consultation interface, a keyword-based natural language understanding interface for collecting information from the history of the presenting illness, a graphical interactive interface for performing examinations, menu-based interfaces for ordering tests, recording tools for the learner to make notes about the case and aspects thereof, as well as to note 30 hypotheses. We still need to perform user evaluations and collect the data for the peer and expert comparisons. The current work represents an exploration of the ways that a simulation-based learning environment can support long-term learning by medical practitioners. It is also an exploration of ways to support reflection as a basis for abstracting the learning and for self-assessment and monitoring of learning progress. References Anderson JR, R. L., Simon HA (1995). Applications and misapplications of cognitive psychology to mathematics education. Barrows HS (1986). “A taxonomy of problem based learning.” Medical Education 20: 481-486. Brna P, I. P., Duncan D (2000). Support for Lifelong Self-Learners Using the Internet. Leeds, Computer Based Learning Unit, Leeds University. 2001. Bryce DA, K. N., Jeremy RW, Myers JH (1997). Facilitating clinical reasoning through computer based learning and feedback. Australian society for computers in learning in tertiary education. Clyman SG, O. N. (1990). “Status report on the NBME's computer-based testing.” Academic Medicine 65: 235-240. Davis D, O. B. M., Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisy A (1999). “Impact of formal continuing medical education: Do conferences, workshops, rounds and other traditional continuing education activities change physician behaviour or health care outcomes?” Journal of the American Medical Association 282(9): 867-874. Elstein ES, S. L., Sprafka SS (1978). Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, Harvard University Press. Elstein ES, S. L. (1990). “Continuing medical education: A ten year retrospective.” Evaluation and the Health Professions 13(1): 5-36. Fitzgerald JT, W. F., Davis WK, Barklay ML, Bozynski ME, Chamberlain KR, Clyman SG, Shope TC, Woolliscroft JO, Zelenock GB (1994). “A preliminary study of the impact of case specificity on computer-based assessment of medical student clinical performance.” Evaluation and the Health Professions 17(3): 307-321. Friedman CP (1995). “Anatomy of the clinical simulation.” Academic Medicine 70(3): 205-209. Hayes KA, L. C. (1996). “The interactive patient: a multimedia interactive educational tool on the world wide web.” MD Computing 13(4): 330-334. Lajoie, SP, S Faremo, and J Wiseman (2002). “Identifying Human Tutoring Strategies for Effective Instruction in Internal Medicine.” IJAIED, 12, to appear Melnick DE (1990). “Computer-based clinical simulation: State of the art.” Evaluation and the Health Professions 13(1): 104-120. Patel VL, G. G. (1993). “Reasoning and Instruction in Medical Curricula.” Cognition and Instruction 10(4): 335-378. Patel VL, K. D., Arocha JF (2000). Conceptual Change in the Biomedical and Health Science Domain. Advances in Instructional Psychology Volume 5. Educational Design and Cognitive Science. G. R. Mahwah, Lawrence Erlbaum Associates. 5: 329-392. 31 Knowledge-based Fuzzy Evaluation of Learners in Intelligent Educational Systems 1 2 3 Maria Grigoriadou , Maria Samarakou , Dionissis Mitropoulos , Michael Panagiotou 1 2 4 Department of Informatics, University of Athens, GR-157.71, Athens, Greece, E-mail: gregor@di.uoa.gr, Phone: +30 10 7275205, Fax: +30 10 7219561 Department of Energy Technology, Technological Institute of Athens, GR-122.61, Athens, Greece, E-mail: marsam@teiath.gr, Phone: +30 10 5385322,397, Fax: +30 10 5385306 3 Department of Informatics, University of Athens, GR-157.71, Athens, Greece 4 01-Pliroforiki S.A., 438 Acharnon Str., GR-111.43, Athens, Greece Abstract: An empirical approach that makes use of fuzzy logic to evaluate the learners towards greater learner control in the context of an intelligent educational system is presented. In this paper we propose a fuzzy logic-based decision making model that is able to store and analyse uncertain information regarding the various characteristics of the learner and evaluate his knowledge status, skills and cognitive abilities and reflect back the contents of the learner model to the learner. Teachers’ experience is incorporated in the definition of the fuzzy sets and in the final estimation of the learner characteristics using the knowledge-based system. In our approach, the evaluation of learner's knowledge level, the existence, or non-existence of a misconception and the estimation of his cognitive abilities, is based on processing qualitative and quantitative information. Thus, the proposed evaluation procedure, employing methods from fuzzy logic, evaluates the learner's progress, strengths and weaknesses by keeping learner performance parameters. In addition, learner-computer interaction provides the evaluation procedure with several measurements. The evaluation procedure aims at encouraging learners to engage in a variety of reflective activities and provides a feedback of events back to the learner so that he can decide if he is going to change his reasoning. At the same time, the learner is allowed to inspect the content of his/her learning model and, either follow the suggestions given, or decide to take responsibility for his/her own learning. Prototype educational software was developed in the following domains of Physics: Mechanics, Reflection-Refraction, Heat, Electricity, Models and Atoms. The fuzzy model has been tested on this prototype, in the domain of Mechanics (free fall of objects in the air or in vacuum) and the results have been very satisfactory. Keywords: Fuzzy model, intelligent system, physics, learner control Introduction The development of an Intelligent Tutoring System (ITS) requires the design of several modules that dynamically interact to provide individualised instruction towards greater learner control over the learning process (see Figure 1). A very important one is the module that evaluates the learner's knowledge, his misconceptions and his personal characteristics. The results of this evaluation, along with any background information, such as the learner's history, form a learner profile pattern upon which pedagogical decisions could be based (Georgouli, 2001). A typical criticism of the need for a detailed learner evaluation is addressed by Self (Self, 1990). However, he suggests that, with realistic aims, intelligent methods for learner evaluation will play a 32 significant role in an ITS provided they are closely linked with tutorial interaction, obtain their input directly from the learner and are open to teacher or learner inspection. Figure 1: Schematic diagram of an Intelligent Tutoring System. Fuzzy logic techniques have been used to improve the performance of an ITS due to their ability to handle imprecise information, such as learner's actions, and to provide human descriptions of knowledge and of learner's cognitive abilities (Kandel, 1992; Panagiotou and Grigoriadou, 1995). In the BSS1 tutoring system a general fuzzy logic engine was designed and implemented to support development of intelligent features, which can better manage the learner's learning (Warenford and Tsao 1997). Uncertainty of learner's performance in Sherlock II was managed with "fuzzy" distributions (Katz et al. 1993). A qualitative learner model was designed using fuzzy logic techniques for a tutoring system in the domain of physics inferring about learner's knowledge level and cognitive abilities from learner's behaviour (Panagiotou et al. 1994). In this paper we propose a fuzzy logic-based decision making model that is able to store and analyse uncertain information regarding the various characteristics of the learner and evaluate his knowledge status, skills and cognitive abilities and reflect back the contents of the learner model to the learner. The proposed learner model combines ideas from cognitive psychology with methods from computational intelligence. Teachers’ experience is incorporated in the definition of the fuzzy sets and in the final estimation of the learner characteristics using the knowledge-based system. The proposed model has been incorporated in a prototype Intelligent Educational System and evaluated by a group of teachers and students. Extracting information for learner’s evaluation A human tutor usually bases his pedagogical decisions on the information he collects regarding learner's knowledge, capability. The same beliefs, mistakes, information misconceptions, should be acquired cognitive and abilities analysed in an and ITS. problem To this solving end, the educational program should present the theory in different ways and use questions and exercises, organised into groups that allow us to evaluate the learner for specific knowledge, mistakes and misconceptions (Mandl and Lesgold, 1988). More specifically, as suggested in (Bertles, 1994), an ITS should extract and associate information regarding: • The knowledge status: knowledge level, mistakes, misconceptions. • The skill status: reading level, audio-visual ability, handling numeric computations. • The cognitive abilities: memory limitations, rate of learning, learning performance, attention, synthesis, abstraction, and generalisation. 33 • The meta-cognitive skills: the ability of linking together newly acquired with already existing knowledge, understanding. Information is extracted through learner's interrogation and monitoring, and can be used to formulate performance patterns that contain qualitative and quantitative data. Questions, or groups of questions, can be related to a subset of the domain knowledge that the learner should acquire and should have a weight representing their importance or complexity as regards the evaluation of the knowledge and the abilities of the learner (Nawrocki, 1987). For example, in order to facilitate information extraction in our prototype implementation we adopted an organisation similar to the one proposed by Mandl and Lesgold (1988). For the implementation of the prototype, the following actions took place: • Analysis of the knowledge domain: The analysis was done in co-operation with two high-school teachers (3 rd and 4 th rd grade), and tested in five 3 grade and seven 4 th grade learners, using the textbooks of these grades. • Decomposition of the knowledge domain, e.g. “free fall” in distinctive parts: free fall in the air, free fall in vacuum. • Development of a database containing different categories of questions and exercises, to be presented to the learner according to the evolution of the lesson. Each question is related to all the possible answers and reasons that a learner can give. • Recording possible misconceptions, and questions through which they were detected. Learner’s model for the level of learning, misconceptions and cognitive skills In our approach, the evaluation of learner's knowledge level, the existence, or non-existence of a misconception and the estimation of his cognitive abilities, is based on processing qualitative and quantitative information. Thus, the proposed evaluation procedure, employing methods from fuzzy logic, evaluates the learner's progress, strengths and weaknesses by keeping learner performance parameters. For example, the answers to the questions are compared with the typical answers and related reasons stored in a database, providing information about the number of correct or incorrect answers expressed as linguistic variable. In addition, learner-computer interaction provides the evaluation procedure with several measurements. The evaluation procedure aims at encouraging learners to engage in a variety of reflective activities and provides a feedback of events back to the learner so that he can decide if he is going to change his reasoning. The time spent to answer the questions and exercises, the number of learner attempts to find the correct answer to the questions and exercises are analysed to provide information about the learner’s characteristics. Figure 2: The three stages of the evaluation procedure. The evaluation procedure is realised in three stages (see Figure 2) using a set of decision-making systems, each one processing fuzzy information and evaluating a predefined learner's characteristic. Below, we provide an overview of the proposed approach. More specifically, a preliminary evaluation of the knowledge level and cognitive abilities is conducted depending on the learner's answer. To this end, learner's answers are compared with the right answers, recorded by the teachers in the ‘right answers data base’, and qualitative characterisations are assigned to the result. Thus, for the learner's answers to a category of questions that are related to his rate of learning we have used 9 discrete levels corresponding to ‘excellent, good, satisfactory, almost satisfactory, unknown, almost unsatisfactory, unsatisfactory, bad, very bad’ level of answers. This 34 evaluation corresponds to a discretisation of the universe of discourse according to the above- mentioned qualitative terms. In order to evaluate and extract information about the learner’s knowledge level, misconceptions and cognitive skills, we defined, for each distinctive part of knowledge, the following: Ei The categories of questions related to the distinctive part of knowledge under consideration. Eij The questions of the category Ei. Aij All the possible answers to the questions of the category Eij. Ai The percentage of the answers in the category of questions Ei, according to certain characterisations. _i The level of knowledge to be examined by each category of questions Ei. _n The misconceptions that the system can detect. Ik The skills that the system can detect. M_ The measurements performed by the system. Wik The weight of the category of questions Ei, or the measurement M_ related to the level of knowledge _i, or the misconception _n, or the skill Ik. U1={y11, y12, …, y1n} The discretisation of the universe of discourse of the knowledge level. U2={y21, y22, …, y2n} The discretisation of the universe of discourse of misconceptions. U3={y31, y32, …, y3n} The discretisation of the universe of discourse of the cognitive skills. Modeling teacher’s expertise in assessing learner’s knowledge, as well as modeling teacher’s personal way assessing, is based on the following resources: − The criteria that the teacher defines in order to assess learner’s knowledge level − Teacher’s estimations of the importance of different types of assessment questions that correspond to the above criteria, with respect to the learner’s knowledge level and the type of the topic under consideration, i.e. a theoretical concept or a procedure. − Teacher’s estimations of the relationship between learner’s correct answers and his/her proficiency of the topic. The assessment of the knowledge level _i, or the misconception _ n , or the cognitive skills Ik, is achieved via the formation of certain fuzzy sets, which are obtained from the correspondence of the learner’s answers and the measurements of the system to the experts’ fuzzy sets. The procedure for the assessment of the learner’s knowledge level is the following: When the learner answers all the questions E ij of a category Ei, the process of the answers provides a classification of these answers in certain categories, based on the percentages of the type of the answers (e.g. 30% sufficient, 50% rather sufficient and 20% insufficient answers). Suppose R the sufficient answers, AR rather sufficient, AW rather insufficient and W insufficient. Suppose also __ a percentage of the answers of the learner. Then we define 9 different categories of answers: 1. R≥_1 2. R≥_2 AND R+AR≥_3 3. R≥_4 AND R+AR≥_5 4. R+AR≥_6 AND R≥AR 5. R+AR≥_7 AND R≥AR 6. R+AR≥_8 AND AR≥R 7. W+AW≤_9 AND AW≥W 8. W+AW≤_10 AND W≥AW 9. W+AW≥_11 The knowledge level _i can be represented as a fuzzy set of the universe U1 as follows: _i = _1/y11 + _2/y12 + ... + _n/y1n where _i is the membership function, _i ∈ [0,1] and y1i is a characterization. 35 Table 1 shows a table of fuzzy sets for the assessment of the learner’s knowledge level, obtained from a certain category of questions. Table 1: y1 y2 . yn 1 0 0.3 . 0.1 2 0.8 0.1 . 0 . . . . . N 0.3 0.2 . 0.1 Table of fuzzy sets for the assessment of knowledge level for a certain category of answers, where 1, 2, ..., N are the categories of answers and yi are the characterisations of the knowledge level An example of such a table is shown in table 2. If now we suppose that the answers of the learner satisfy category 2 (R≥_2 AND R+AR≥_3), then the assessment of the knowledge level, for the certain category of questions, is: _i = 0.9/(sufficient)+0.1/(rather sufficient)+0/(medium)+0/(rather insufficient)+ 0/(insufficient) For the final assessment of the knowledge level the fuzzy sets obtained from all the categories of questions are taken into consideration. The assessment of misconceptions and cognitive skills follows similar procedure. sufficient rather sufficient medium rather insufficient insufficient 1 1 0 0 0 1 2 0.9 0.1 0 0 0 3 0.6 0.3 0.1 0 0 4 0.1 0.6 0.3 0 0 5 0 0.2 0.7 0.1 0 6 0 0 0.3 0.5 0.2 7 0 0 0.1 0.6 0.3 8 0 0 0 0.3 0.7 9 0 0 0 0 1 Table 2: Assessment of knowledge level for a certain category for questions Importance of different types of questions For the assessment of the knowledge level, or misconceptions, or the cognitive skills of the learner, there is a set of questions and measurements with different weights for each assessment. The proper weight used in each case, is based on the algorithm of T. L. Saaty (Petrushin and Sinitsa, 1993) and a table is created according to the following rule: If the importance of the question E i or the measurement Mi with respect to the question E k or the measurement M k is w then the importance of the question E k or the measurement M k with respect to the question Ei or the measurement Mi is 1/w. Description of the prototype Experiments have been performed to evaluate the performance of the proposed approach. The development of the hybrid model has been made for a prototype educational system in the following domains of Physics (M.Grigoriadou, M. Samarakou et al 1999): 36 Heat: -Expansion of solid materials, liquids, gases Optics: -Linear Propagation of Light, Shadow -Change in physical condition -Reflection and Refraction of Light -Analysis and Synthesis of Light Mechanics: -Free Fall on the Earth -Free Fall in vacuum Electricity: -Closed Circuits with an Electric Source and one or more Lamps/Resistors -Electric Current and Voltage measurements Atoms/Models: -The closed circuit Electron-based Model -The expansion Molecule-based Model -Geometric Models for Shadow, Reflection and Refraction -Newtonian Model for Free Fall In this prototype the knowledge domains have been analysed for Greek learners of 3 secondary education. In the following we will describe the actions that rd took and 4 th place grade of for the implementation of the part of the prototype that deals with The Free Fall of objects. Interface for the Free Fall of objects and recording of the students’ actions This part aims at helping teachers to teach and students to learn about free fall of objects and the role of the weight and the resistance of the air in free fall. Research in the domain of didactics in Science has pointed out that students develop the following misconceptions and difficulties while they try to study the fall of objects: “Heavier objects fall faster”. Students that have this idea do not take under consideration the influence of the resistance of the air to fall. They believe that heavier objects fall faster either in vacuum or in the atmosphere (Viennot L. 1979). “If there is no air, then there is no gravity – objects in vacuum have no weight”. This idea has its origin to the everyday experience that an object doesn’t fall only when it is supported (Whitelock 1991), i.e. when a force is applied by another object in contact. It seems that the wrong idea that forces are applied between objects only when they are in contact, comes from this perception. This wrong idea leads sometimes to the misconception that there are no forces applied in vacuum and the objects have no weight – the air and the atmospheric pressure are responsible for the gravity force (Mayer 1987). The misconception mentioned above is even stronger in “space”, i.e. when the object is not very close to the earth (e.g. a satellite): “Objects in space have no weight”. The example that follows illustrates the didactic approach and the interaction with the student, by using the scenarios “Fall of objects” of the relevant software. Each task is carried out in certain steps, where the activity of the student is recorded, as well as his/her choices. This data is used by the diagnostic system in order to detect specific cognitive difficulties, misconceptions, wrong ideas or lack of prerequisite knowledge. Consequently, this detection leads to the activation of an adapted instructional strategy, e.g. suggestion to read a part of theory or accomplish another task, presentation of some examples, etc. At the same time, the learner is allowed to inspect the content of his/her learning model and, either follow the suggestions given, or decide to take responsibility for his/her own learning. The above procedure is briefly described in the following. 37 Description of the interface: Figure 3: A typical screen of the educational prototype system. In the building on the right side of the screen (Fig. 3) there is a boy that can “let” an object free to fall on the ground. The user can change the altitude, using the slider in the control window seen on the left. S/he can also select one or more objects to fall (iron sphere, light or heavy ball). There are buttons to control the motion (start, pause, stop, reset), synchronized with a stopwatch and a distance recorder. Trails of motion, graphs of distance – speed – acceleration vs. time and the representation of the relevant vector quantities are also available. By clicking on the upper left corner of the image the user can “remove” the atmosphere and study the fall without the resistance of the air. The student can read in a separate window a description of the activities – tasks that she/he is asked to carry out. Detection of cognitive difficulties: 1. The student believes that heavier objects always fall faster 2. The student believes that if there is no atmosphere then there is no gravity force. Description of a students’ activity: Each activity is divided in three parts: − In the first part, the student reads a description of the problem / phenomenon and the relevant actions to be done. At this point, the student is asked to predict what is going to happen. − In the second part, the student is asked to carry out the simulation of the relevant experiment − In the third part, the student is asked to compare his/her prediction to the results of the and to record the results. experiment. 38 The answers in the first and third part activate the instructional strategy that proposes to the student either to revise a part of relevant theory, or to perform another task, or to observe some relevant simulations/ examples, etc. By the same time, the student can be informed about his/her learner model, through comments and remarks related to his/her actions and selections, and he/she can either follow the proposed tasks, or choose to control the progress of his/her activity. Example: Objective: Study of the factors that affect the fall of an object. In this series of activities you will study the fall of an object and the factors that affect it. You can select different objects and let them fall from various heights, simultaneously, or one at a time. (You can also “eliminate” the atmosphere from the earth an let the objects fall in vacuum.) Activity: Free fall of two objects in the air 1. Prediction If you let an iron sphere and a light ball fall simultaneously from the same height, which one will reach the ground first? Check the correct answer. 0 The iron sphere (correct) 0 The light ball (wrong) 0 They will reach the ground at the same time (wrong) Select one reason that justifies your answer. Depending on the previous answer, possible reasons are presented for each case: The iron sphere (correct) 0 o It is heavier o The resistance of the air has less effect on the iron sphere (correct) The light ball (wrong) 0 It is lighter and it moves faster o They will reach the ground at the same time (wrong) 0 All objects fall with the same acceleration o 2. Experiment Now, select the iron sphere and the light ball and let them fall simultaneously from the maximum available height. Observe which of them reaches the ground first. 0 The iron sphere 0 The light ball 0 They will reach the ground at the same time Is the result of the experiment in agreement with your prediction? 0 Yes 0 No At this point, the following feedback is given to the student, according to his/her learner model: In case of agreement between prediction and results, and depending on the reason selected in prediction, the following are proposed: − Selected reason “It is heavier”. The student is asked to “remove” the atmosphere and carry out the related activity. − Selected reason “The resistance of the air has less effect on the iron sphere (correct)”. The student is encouraged to go to the next activity. 39 In case of disagreement between prediction and results, it is proposed to the student to observe some real life falls (e.g. the fall of a sheet of paper and a book), observe the result and then repeat this activity. Representing Expert’s Knowledge According to the way the program is structured, the expert part contains the following databases: • Knowledge database, containing the theory to be presented to the learner. • Questions/exercises/activities database, containing the following categories: E1. Questions related to the knowledge of the relevant theory. E2. Activities aiming at checking the understanding of theory, the ability of the learner to solve equations and arithmetic formulae, and the ability to perform math operations. E3. Questions aiming to test the ability of the learner to associate new knowledge with real life situations, and to detect possible misconceptions. • Answers database, which contains all possible variations of the answers that the learner can give to the proposed questions/exercises. • Messages database, containing messages to the teacher for problematic situations that the system cannot handle. • Instructions/explanations database, containing messages to the learner to be used during the interaction with the ITS. The selection of the appropriate message depends on the recorded “qualitative model of learner’s knowledge and cognitive abilities”, or on the choices made by the teacher, or even on the choices made by the learner - if the teacher permits this. Exploiting learner’s record All the evaluations related to the level of knowledge, misconceptions and cognitive characteristics are recorded every time the learner goes through a distinctive part of knowledge. Furthermore, the topic, the way of presentation, the number of questions/exercises and the sequence of these questions are also recorded. These recordings compose the events in this incident of the distinctive part of knowledge, which belongs to the entire learner’s record, i.e. the knowledge that the system has about the learner. An example of such a recording of the events of an incident, during the instruction of free-fall is: 12/04/2002, free-fall, presentation of theory, five questions related to the kind of motion and the topics presented in theory, knowledge level of theory medium, four questions related to possible misconceptions about free fall, no misconception detected, …., knowledge level rather satisfactory, ability in maths average, rate of learning normal, no misconceptions. Learner’s record, during the presentation and exploitation of a distinctive part of knowledge, is mainly responsible for the activation of the counteractive rules included in the educational software and the adaptation of the instructional strategy, during the learner - computer interaction, within an incident. The recorded information is responsible for the way of initiation, of the program and the handling of repeated problematic situations. The program also uses the recorded information, so that it can exclude from further use approaches that were determined to be ineffective. If now the learner does not work out successfully the presented distinctive part of knowledge, then this ineffective approach is recorded, and the rules of the pedagogical part are activated. The part is repeated, until the learner finishes it successfully, or stops the program. Learner is always informed about the content of his/her record. In case that he/she prefers to deactivate the adaptation of the system, all the available topics are presented to him/her and he/she becomes responsible for his/her further learning process. 40 Conclusions The learners and teachers that participated in the analysis of the knowledge domain evaluated the prototype (Group 1). Three more teachers and their learners (Group 2) also evaluated the prototype by using it in two lessons during the instruction of free-fall. Altogether, 5 teachers and 27 learners evaluated the program. There has been a growing appreciation of the importance of purposeful learner control of the learners that need to be able to access and control them (Kay J. 1995, 1997). With regards to detecting learner’s misconceptions a success of 92% was achieved while the interaction with the system improved their knowledge. The experiments with the prototype verified that, by evaluating learner interaction it is possible for the system not only to adapt its educational strategy, but also the degree of learnercomputer interaction, provided of course that this option is supported by the system. Adaptive educational multimedia technology offers such possibilities, since it supports sound, video or animation and virtual laboratories. Moreover, the system responded to the anticipations of the teachers with respect to the estimations for their learners. These estimations were related to the evaluation of the level of knowledge and cognitive abilities of the learner, provided that the same answers were given in the class environment. The approach presented in this paper is an empirical one: the learner's evaluation depends on the designer's ability to analyse the knowledge domain suitably, define fuzzy variables and appropriate membership functions for their fuzzy sets, and relate learner response with appropriate knowledge and cognitive characteristics. The evaluation procedure is closely related to the knowledge scheme adopted for the representation of the particular knowledge and cognitive characteristics of the learner and can be used for the development of a qualitative model of the learner's knowledge and cognitive abilities. The model response can further be used for deciding about the appropriate teaching strategy. At the same time, the learner is allowed to inspect the content of his/her learning model and, either follow the suggestions given, or decide by his/her own to take responsibility for his own learning. The proposed hybrid approach realises a qualitative model of the learner, which stores and analyses information about the knowledge status and the cognitive abilities of each learner. We currently investigate techniques to enhance the percentage of success of our model by exploiting the training and generalisation capabilities of the neural networks to extract information from learner profiles that can further extend the applicability of the evaluation procedure. References Bertles K. (1994). A dynamic view on cognitive learner modelling in computer programming, Journal of Artificial Intelligence in Education, 5, 1. Georgouli K. (2001). Modelling a Versatile Mathematical Curriculum for Low-attainers. In Proceedings of the 8th Panhellenic Conference in Informatics, vol 1, Nicosia/Cyprus (2001) 463-472. Grigoriadou M., Samarakou M., Mitropoulos D., Rigoutsos A., Stavridou E., Solomonidou C. (1999). Vectors in Physics and Mathematics, Conference on Technology and Education, International Conference on Technology and Education, ICTE 99, March 1999, Edinburgh, p.71-73. Kandel A. (1992). Fuzzy expert systems, CRC Press, 1992. Katz S., Lesgold A., Eggan G. & Gordin M. (1993). Modelling the student in Sherlock II, Journal of Artificial Intelligence in Education, 3, 4, 495-518. Kay J. (1995). The UM toolkit for cooperative user modelling. User Modelling and User-Adapted Interaction, 4, 149-196. Kay J. (1997). Learner know thyself: a student models to give learner control and responsibility. In Proceedings of ICCE’97 Kuching, Australia. King R.E., Magoulas G.D. & Stathaki A.A. (1993). Multivariable fuzzy controller design, Control Engineering Practice, 2, 431-437. Mandl H. & Lesgold A. (1988). Learning issues for intelligent tutoring systems, Springer. 41 Nawrocki L. (1987). Maintenance in training. In Kearsley (ed.), Artificial intelligence and instruction: application and methods, Addisson Wesley. Panagiotou M. and Grigoriadou M. (1995). An application of fuzzy logic to learner modeling. In Proceedings of the IFIP World Conference on Computers in Education, Birmigham. Panagiotou M., Grigoriadou M. & Philokiprou G. (1994). Knowledge and learning learner model. In Vosniadou S, Corte E & Mandl H (eds), Technology-Based Learning Environments, NATO ASI series F., Springer, 243-249. Petrushin V., Sinitsa K. (1993). Using Probabilistic Techniques for Learning Modelling. Proceedings of (AI-ED93) World Conference on Artificial Intelligence in Education, Edinburgh 1993. Saaty T.L. (1978). Exploring the interface between hierarchies, multiple objectives and fuzzy sets. Fuzzy sets and systems, 1, 57-68. Self (1990). Bypassing the intractable problem of learner modelling. In Frasson and Gauthier (eds.), Intelligent tutoring systems: at the crossroads of artificial intelligence and education, Norwood, NJ: Ablex. Viennot L. (1979). Spontaneous reasoning in elememtary dynamics. Eur. J. Sci. Ed. 1,205-221 (1979). Warendorf K. and Tsao S.J. (1997). Application of fuzzy logic techniques in the BSS1 tutoring system. Journal of Artificial Intelligence in Education, 8, 1, 113-146. Wenger, E. (1987) AI and Tutoring Systems: Computational and Cognitive Approaches to the Communication Knowledge, California: M. Kaufmann Publishers, Inc. 42 Modeling the Learner Preferences for Embodied Agents: Experimenting with the Control of Humor Aude Dufresne, Martin Hudon Department of Communication, University of Montreal. C.P. 6128 Succ Centre-Ville, Montreal, Qc, Canada, H3C 3J7, dufresne@com.umontreal.ca Abstract: This research presents a model and an experiment on the integration of personality preferences in support systems for learning. We will present briefly the context of the research on the access to learner and group models, than the theoretical background on personalization of interface and more specifically of the functions of humor in general and how it can be used to integrate affective dimensions in tutoring interaction. This research stems partly from the Reeves and Nass [1] postulate that people will react the same way to a computer mediated interaction, then they do to a real interaction, and thus that their reaction to an humoristic tutor would be generally more positive than to the non-humoristic tutor: the tutor attracts more attention, the perceived usability, social presence and personalization are improved. We will present the design and experimentation of an open model support system, where feedback is given to learners on their progression, but also where preferences for support may be defined. It was experimented under two conditions “with” and “without humor” and qualitative attitudes measures where taken. Though results are only preliminary, this study of the impact of humor suggests various considerations on how personality aspects can be integrated and their impact studied in ITS systems. Keywords: Personalization, embodied agents, humor, evaluation, distance education. Context of the research This research stems from a preoccupation for interface and communication design in ITS especially in the context of distance education, in very open domains of learning. To design appropriate interaction in the context of learning it is necessary to have access to rich tasks and learners models. Thus the majority of ITS models depend on a highly detailed representation of the domain to be learned, where knowledge can be tested at micro level and inference can be made amongst knowledge elements. It is then possible to design natural language understanding, specific explanations or deitic demonstration for a given problem or scene layout. Our challenge was to generalize the principles of support, to make them accessible when the models of the domains are more shallow; when the programming of support has to be done by a professor with no special training. In such context, evaluations cannot be as detailed and more self assessment is necessary to access the learner’s model. The support functions were to be part of a course editor. Principles that could be applied to extract constraints in tasks and to give advice; to present demonstration and suggest content element using a generic editor. In order to enrich the contextual model and also the means of support, we developed a dynamic and adaptive interface in which we experimented various dimensions of personalized support. ExploraGraph a dynamic interface to support the task and learner models The ExploraGraph© Navigator (figure 1) was designed to facilitate the visualization of learning structures, that present task scenarios, knowledge or document structure. A conceptual model, with typed nodes and typed links, is used to simulate the relationships between the elements and organize the global representation to serve as a front to the course content. The graphs follow a physical model, where links express semantical relations among elements, and which reacts with zoom and fish eye effects when they are explored by the learner (see previous description [2-4]). 43 Figure 1: ExploraGraph© Navigator, showing the graphical structure of activities with individual and group completion levels and the Hacker MsAgent contextual help. As in other kinds of maps, adaptive annotations were used to give the learners feedback on their progression in the content, how often nodes have been visited. They can also edit their learner model by editing the degree of "completion" for each node displayed in graphs. The learner's model is thus presented as an overlay on the task's model, in structures of activities or concepts, giving feedback to what as been explored (visited), and what the learner consider as being finished. The hierarchy amongst nodes makes it possible to de propagate, both exploration and completion levels in the learner model . Individual learner models are kept both locally and on a database server, so the learner may access his profile from different computers (e.g. home or university). It is thus possible to compile group models so a feedback can be given on the activity of the class. Feedback on the group of learners [5] can also be used to motivate the learner as a passive feedback or to support more active type of feedback (e.g. “ All the others have finished”). Isolation is a problem in distant learning and graphs can become a transparent way to provide learners with information on what is happening to the group. Thus, learners can also display the levels of "visit" and "completion" of fellow learners. The postulate was that this visibility and easy access to learner models and group models would encourage learners to update them, thus improving the information the ITS system was using. In ExploraGraph©, since graphs are dynamically generated, further adaptive functions may be added, so graphs are contextually arranged in order to cue the learner toward more relevant areas, using zoom functions (this corresponds to Brusilovsky's [6] principle of maps adaptation). Other support functions are included in the environment, like MsAgents avatar animations or messages, control of the environment when the learner specify intentions. Microsoft Agents were introduced in the environment partly “to fulfill the need for social context” when no other learner is on-line, but also to serve as the interface to the advisory system. They are driven by the rule-based support system as defined in the database. We evaluated a first version of the system with 9 learners over a 6-weeks course. We used observation, trace analysis and questionnaires to evaluate the usability of different modalities of help in the environment [7]. We found, as expected, that the pacing of the help was critical, that physical (force feedback) guidance seemed to provide better retention, and that prolonged support improved motivation. 44 Comments (video taped observation, focus group and open questionnaires) showed a lot of variation in the preferences of learners for support: modality, animation of graphs, agents, timing and other personality aspects of support. It appeared important to introduce more parameters in the learner model linked to his preferences for support. It was important to investigate the attitudes toward various personalization factors, that could be taken into account while defining the rules. Those personalization factors should eventually appears as ways for the learner to control his environment (adaptation) or eventually for the system to adjust to the learner’s reactions (adaptive). We are planning to give the learner the possibility to choose from various coaches, with different personalities, to ask more or less support and finally to choose whether he want humor or not. We wanted to experiment and evaluate the impact of those personalization factors separately. We describe here an experiment on the attitudes toward humor used by embodied agents in pedagogical interaction. This was a controlled experiment were humor was either present or not present, we wanted to investigate how learners reacted and whether that could be an interesting variable to include in an ITS. Embodied agent and the personalization of support Embodied agents are a seen as an interesting paradigm in AIED to support learning, because they offer a better integration between personalization and the spatial context of explanation [8-11]. As the term “embodied” suggest, they are more expressive and thus appear to be more effective to simulate real tutorial interactions. Embodied agents can be used to attract attention, guide and demonstrate using deitic gestures, suggest emotional context using the expression of emotion. Studies were made on the integration of affective dimensions in embodied agents. As Elliott, Rickel, & Lester [12] suggest, the affective dimensions are important, because they make the coach appear to care about the learner, to be with him, and because it may communicate enthousiasm about the task. Okonkwo and Vassileva [13] also studied the impact of having coaches express emotions while giving advices. They found that non-verbal expression of emotions did not influence performance but had a positive effect on the perception of the help. But it is difficult to make an agent to be emotionally appropriate in his interaction. To do so the system’s model should be more elaborate to better sense the learner emotional state, and to better react to it, both at the content and emotional level, “bridging between sensory input and action generation” [14]. As Cassel and Thórisson [15] have found it is not as much emotions per se, as the envelope role of non-verbal expressions that accompany dialogue, that are important to give a lifelike impression and to ensure fluidity in interaction. But emotion is but one aspect of embodied agents. As André et al. [16] affirmed “the next major step in the evolution of interfaces is very likely to focus on highly personalized interfaces”. So the new undergoing challenge in interface design in general and in ITS is to try to enrich the support model with more dimensions. human But like the dimensions, integration of incorporating personalized aspects of embodied personality, agents poses affective the and problem social of the investigation, the simulation and the evaluation of complex dimensions of affective and personality aspects of learning. Inside the ITS, the personalization of support appears to be an enrichment of the communication model, that uses the specifications of the context of the activity of the learner, but more importantly his learning model and preferences, to intervene or shape the communication model. The integration of personalized support in education appears more and more essential on a pedagogical point of view, but also to make supportive agents more believable and trustable. As André [8]explains: “A growing number of research projects in academia and industry have recently started to develop lifelike agents as a new metaphor for highly personalised human-machine communication. A strong argument in favour of using such characters in the interface is the fact that they make human-computer interaction more enjoyable and allow for communication styles common in human-human dialogue”. She presents the Presence system which “uses affect to enhance the believability of a virtual character, and produce a more natural conversational manner”. In this context we thought that humor could be an interesting dimension to integrate [17], first because of its potential for creating more personalized embodied agents, displaying emotions and social personalities depending on the context. Just as the simulation of affective reactions might make 45 embodied agents more believable, we postulated that humor might add to the impression of intelligence and complicity of supportive agents and should make them more acceptable to humans. Humor could be an efficient communication strategy used to attract attention, diminish stress or stimulate affective and motivational reactions of learners. But humor is highly tinted with personality aspects and thus is difficult to integrate in a learning environment. In fact, the development of personalization dimensions asks for new methodologies of research and evaluation in AIED. Should the ontology be defined theoretically and its usefulness assessed empirically ? Should it be extracted from the observation of interaction in comparison to pretest or posttest of psychological dimensions? Experimenting with the control of humor The perception and impact of humor in communication Humor is similar to a game, it is possible only when the participants are capable of a certain degree of meta-communication, stating that “this is a game”. As Eco [18] describes it breaks the links between signs and signification, introducing an incongruity between them and thus a second level meaning. This signification game mixes the expectations of the receiver, which experience first an interrogation and then surprise when he is confronted with the unexpected meaning. Humor has been said to increase the perceived social presence in a medium. According to Lombard and Ditton [19, p. 9] “The presence is [20] the perceptual illusion of non-mediation […] occurs when a person fails to perceive or acknowledge the existence of a media during a technologically mediated experience”. The definition suggests that this illusive experience is at the perceptual, cognitive and emotional level of the user interaction. According to them the media is not only perceived as transparent but it naturally suggests the possibility to support and simulate “real” social interactions. According to Short et al. [21] a high degree of social presence is important for specific types of interactions, for example persuasion and problem solving are difficult when the level of social presence is low. In a context where a learner has difficulties, and where the advisory system tries to influence him or to stimulate his motivation, social presence might be especially critical. ¨For Biocca [22] the perceived “social presence” is linked to the perception of intelligence and of intention expressed through the mediation of the artefact. Theories on humor may be grouped under three dimensions: the superiority theory (humor presumes and places the receptor into an inferior position), the relaxation theory, the incongruity theory. Those dimensions of humor may each be used for a specific purpose in the context of tele-learning: to exert authority to bring students into a different behavior; to unleash tensions associated with learning or with the tutoring interaction; to destabilize students and provoke new understanding. But humor is accompanied by a high level of noise in the interaction. The intent meaning might be unclear, the underlying model of social interaction might be inappropriate to the context, or to the learner personality. Humor is highly cultural, and its meaning is negotiated as the interaction evolve between participants. If humor is to be used, some coherent common codes must be developed between the system and users; some means of communication must be designed so users may understand, learn and adjust the models that lay behind the system; and so the system can be influenced by the reaction and preferences users have for humor. It is important to diminish distance and noise in the supportive interaction. So the messages the system is giving are understood and efficient in promoting understanding and efficient learning on the part of the participant. 46 Possible impact of using humor in an ITS What could be the impact of using humor in ITS ? Research on embodied agents and research on humor suggest that humor might make embodied agents more believable; that it may make the computer look more intelligent, since it would seems as though he not only communicate, but also metacommunicate about the situation. Humor might help alleviate tensions associated with the interaction with a computer, with the isolation and stress of distance education. Finally, in a way, the noise in communication associated with humor might be a way to hide or dilute inappropriate interventions of the ITS. Methodology and hypothesis What could be the impact of humor and how is it possible to experiment using it in the context of ITS ? As Reeves and Nass [1] proposed, it is possible to experiment interaction with computers the same way we are evaluating real interactions, in this case having a condition ”with” and a condition “without humor”, and comparing impact on the usability and attitudes toward the system. But if we generalize the objectives of the system which were to integrate personalization of the interaction in the system; it is important to integrate personalization parameters of the support system, which could affect the support system, and eventually be controlled either with direct adaptation by the learner (I want more humor) or by adaptive adjustment by the system (He does seem to like humor). Though the system was designed so the learner could control it using the control panel, in the context of this research the conditions were fixed by the experimental set up, ie the learner could not change them. As a first step we transformed the ExploraGraph rule editor so it would be possible to define rules “with” or “without humor”. We experimented the system in the context of a course on learning the Flash software with undergraduate students at University of Montreal. The experiment was reduced to a three hours period, where the students were to explore the content, do some exercises and then pass a small test. Support rules were designed both with and without humor, using the different dimensions of humor – incongruity, superiority, relaxing, etc. The design was a split group experiment, where half of the subjects had first a version with humor and then without, and the other half started without humor and then with humor; the switch was time based and blinded to them. Twelve subjects participated in the experiment but only eight filled both questionnaires and were kept for the analysis. Two questionnaires (after each experimental period) and interviews were used to collect data on attitudes of learners. As suggested by research on humor, the hypothesis were that humor would bring a more favorable evaluation of the support system, having more “social presence”; that agents would be perceived as more sensible, intelligent, more credible. Results Though we cannot use the results to confirm our hypothesis, because of the limited number of subjects and the short duration of the experiment. But they can serve as indication not only of the possible impact of humor on ITS, but also on the complexity of its interweaving with personality and context. In general the learners had difficulty getting to know and use the ExploraGraph environment in such a short period. They found the graphs very different from what they are accustomed to, like regular hierarchical hypertext pages. In fact even with simple hypertext course, we had found that students suffered from the “lost in hypertext” syndrome and that they lack directions on how to organize their learning activity [7, 23]. In a previous experiment ExploraGraph had been used for 6 weeks, and it had taken some time, before learners got accustomed to it, to understand the semantic of the links and to organize their activity using it. Also in this experiment, the students had to pass a test at the end, and this might have influenced their evaluation of the environment and of the agents’ support. In general the evaluation of the agents was better for the humorous version and the appreciation of the agents was lower for the second evaluation (see Table 2). In fact, the difference between the first and the second evaluation was more important when the learners passed from an humorous to an non humorous condition (3.9 to 3.3 vs 3.6 to 3.5). 47 First Second evaluation evaluation Group Humor No Humor Mean Group Mean Mean SD G1 3.9 G2 3.5 3.7 0.99 G2 3.6 G1 3.3 3.45 1.08 Mean 3.75 3.4 SD 1.03 1.03 Table 1: Mean attitudes and standard deviation toward the agents in the support system forconditions with or without humor (scale 1 to 6), for the two groups at the first and second evaluations. Figure 2 presents the agglomeration of results to the questionnaires grouped by category according to the hypothesis. In general the attitudes toward the agent were more positive, when the condition was “with humor”: the agent was found to be more funny, more clear, more intelligent; it was found to attract more attention, to be more relaxing. As for attributes associated with the “social presence” dimension, the humorous agent was found more friendly, sensible, expressive, and social; though he was found less personal (but the meaning of this question might not have been understood clearly (more personality vs more personalized to learner). On the contrary, the humorous version of the agent was found not to be as good a support to motivation, orientation and learning. This might be due first to the stress associated with the test at the end of the experimentation. It might also be due to the very short experience with the course and the agent - only three hours, which limited the number of possible support interventions. It would be interesting to see if the attitudes would be the same for a course lasting many weeks, when the isolation and motivation might become a problem. 48 toward agents Believable, intelligent attract attention Relaxing Like humour friendly Presence Attitudes Social Clear, appropriate sensible, expressive social Learning personal Support motivation Support learning Orientation, guiding 0.00 1.00 2.00 3.00 Mean 4.00 5.00 attitudes Humor No humor Figure 2 – Mean attitudes toward the agent and the help provided in the ITS for conditions with or without humor on a scale from 0 to 6. Discussion In general results and interviews showed that the appreciation of humor was much dependent on context: How difficult is the course and the learning environment? Whether there is an evaluation at stake. Specific situations where the agent interventions where out of place. While some students liked the agent, other expressed reserves on his personality, they did not like some of his remarks which they found inappropriate. The agent, the “Hacker”, had been designed to address “resistant” students [24]; his humor was found by some students to be aggressive (using a superiority strategy, the agent was teasing the student), his relaxing and incongruous behavior was also found misplaced, by some students since there was an evaluation and the students were stressed. Some students expressed the need to stop the agent at one point or they wished they could have chosen another one, when proposed so at the interview. So, even though the theory and general empirical results suggest humorous agents might preferred to non-humorous one, in some conditions and for some students they were found disturbing. In general students wished they could have more control on supportive interventions. Another interesting result, was that humorous interventions were perceived as less supportive for the task, orientation and motivation. Even though this might be linked to the very short experiment, it might be important when adjusting the degree of humor to take into consideration both task and personality factors. May be keep humor for when a task has just been completed (reinforcement) or no task is urgent (beginning of the course). This experiment on humor is part of a more general research, where students will eventually be able to choose the personality of the supportive agent. Not only would they be able to choose humor or not, more or less support, but they will be able to choose amongst a set of coaches with different personalities. Following Martinez [24] research on learning styles, we will offer them four archetypal coaches designed to match the four kinds of learners – transforming or intentional, performing, conforming, and resistant. She describes how individuals follow a complex mix of beliefs, desires, emotions, intentional effort, and cognitive and social styles to learn, which must be taken into account 49 6.00 by the supportive environment. In fact learning styles and humor theories suggest ontologies for preferences, which may be included in general rules of support systems. For example for intentional or resistant learner, incongruity type of humor might be more appropriate; for p e r f o r m i n g learner superiority type of humor might be more supportive. So the apparent personality of the coach can be used to represent a style of support – timing, parameters of the situation where help will be offered, content of the advices. We had designed four coaches for the different type of learners, but only one the Hacker was used in the context of this research. Eventually with multiple coaches, we will experiment the differential models of personalities of help in relation to style of learners, in order to measure the spontaneous use of the coaches, and their impact on attitudes toward help (perception and reaction to help). Conclusion Even though research on embodied agents with models of affective reactions are interesting, few evaluations have been made of their acceptance by learners in real educational context. More so, it appears interesting to study how we can design general models to use them in context where the knowledge of the domain is limited to interdependencies in tasks or concepts, like what is described by Paquette & Tchounikine. [25]. In this direction, the ExploraGraph system was built to externalize the structure of the task and the learner and group models and to facilitate its access to learners. In it, support rules may be designed to use adaptive interface and MsAgent animations and advices. Parameters were added in the support system to take into account preferences of the learners (humor, chosen coaches, level of support). We have used it to make this control experiment on the attitudes toward humor in embodied agents. Though with only a limited number of subjects, we found that attitudes toward agents displaying humor was generally more positive, that it makes them appear more intelligent, sensible, believable, that their social presence is higher. We also found that embodied agents with humor were more distracting and not perceived as being as good support for learning. As Eco[18] suggests, the learner must look twice to understand the second level in humorous intervention. This appears to attract his attention, but also to distract him from what he is doing. The learners reacted negatively to such disturbance. Learners comments suggested that their affective reaction is highly dependent on their personality and that of the agent. More research is needed to describe in more details how humor and personality could be linked to define situations and actions to support learners based on cognitive or learning styles [24]. It is also important to analyze and model the control and reaction of learners to the help provided and to compare this to their attitudes toward help and the justification they see for preferring one coach over the other. An environment where the learner could control the degree and style of coaching would be interesting to study, but it must rely on strong and generic theoretical models of coaching, linking the diagnosis of situations and the types of support actions, to generic models of tasks as in ExploraGraph [2]. It would be interesting to do observation and to ask learners to characterize the personalities of the coaches and their interventions to precisely understand the reaction of learners to different personalities, and types of humor in agents. We may also find gender-based differences in the use and control of the support system as in Okonkwo and Vassileva [13]. But letting the learner adjust the support preferences is not enough, since even though it is important to let the learner control his environment; it might be a tedious task for him. So we intend to integrate in the environment adaptive features taking into account reactions to support (agent is stopped, advice are not followed). So general rules could be adjusted to students in general and to a specific learner using learning mechanisms. References 1. Reeves, B. and Nass, C. The Media Equation. CSLI Publications, New York, 1996. 2. Dufresne, A., ExploraGraph : Improving interfaces to improve adaptive support. in AIED'2001, (San Antonio, Texas, 2001), IOS Press: Amsterdam, 306-313. 50 3. Dufresne, A., Model of an Adaptive Support Interface for Distance Learning. in ITS'2000, 4. Dufresne, A., Cosmova, V., LeTran, T. and Ramstein, C. An interface to support the learner (Montréal, 2000), Springer Verlag, 334-343. with dynamic graphs and multimodal goal driven explanations. in Lajoie, S.P. and Vivet, M. eds. AIED'99, IOS Press, Amsterdam, 1999, 660- 662. 5. Dufresne, A. Conception d'une interface adaptée aux activités de l’éducation à distance - 6. Brusilovsky, P. Methods and Techniques of Adaptive Hypermedia. User Modeling and User- 7. Dufresne, A., Interface design for Distant Education : Experimenting Different Environments. ExploraGraph. Sciences et Techniques Éducatives, 8 (3) 2001. 301-320. Adapted Interaction, 6 1996. 87-129. in EDICT'2000 International conference on information a communication technologies for education, (Vienne, Austria, 2000), 363-372. 8. André, E., Klesen, M., Gebhard, P., Allen, S. and Rist, T., Integrating Models of Personality and Emotions into Lifelike Characters. in workshop on Affect in Interactions – Towards a new Generation of Interfaces in conjunction with the 3 rd i3 Annual Conference, (Siena, Italy, 1999), 136-149. 9. Nkambou, R. and Laporte, Y. (eds.). Producing Non-Verbal Output for an Embodied Agent in 10. Rickel, J. and Lewis Johnson, W. Task Oriented Collaboration with Embodied Agents in an Intelligent Tutoring System. Springer-Verlag, Berlin, 2001. Virtual Worlds. in Justine Cassell, J.S., Scott Prevost and Elizabeth Churchill editors ed. Embodied conversational agents, MIT Press, Cambridge, Mass., 2000. 11. Johnson, W.L., Rickel, J.W. and Lester, J.C. Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education, 11 2000. 47-78. 12. Elliott, C., Rickel, J. and Lester, J.C., Integrating affective computing into animated tutoring agents. in IJCAI Workshop on Animated Interface Agents: Making Them Intelligent, (Nagoya, Japan, 1997), 113-121. 13. Okonkwo, C., Vassileva, J., Affective Pedagogical Agents and User Persuasion. in Universal Access in Human - Computer Interaction (UAHCI), (New Orleans, USA, 2001), 397-401. 14. Thórisson, K.R. Communicative Humanoids - A Computational Model of Psychosocial Dialogue Skills Media Arts & Sciences, School of Architecture & Planning, MIT, Boston, 1996. 15. Cassell, J. and Thórisson, K.R. The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Converstional Agents. Applied Artifical Intelligence, 13 (4-5) 1999. 519-538. 16. André, E., Muller, J. and Rist, T., The PPP Persona: A Multipurpose Animated Presentation 17. Hudon, M. Modélisation des stratégies d'intégration de l'humour au sein de l'environnement 18. Eco, U. Le signe. Labors, Bruxelles, 1988. Agent. in Advanced Visual Interfaces, (1996), ACM. ExploraGraph. Département de Communication, Université de Montréal, Montréal, 2002. 19. Lombard, M. and Ditton, T.B. At the Heart of It All: The Concept of Presence. J. of Computer Mediated Communication, 3 (2) 1997. 20. 21. Bertrand, D. Dire et faire au travail. Érès, Toulouse, 1995. Short, J., William, E. and Christie, B. The social Psychology of Telecommunication. Wiley, London, 1976. 22. Biocca, F. Cyborg's dilemna: Progressive embodiment in virtual environments. J. of Computer Mediated Communication, 3 (2) 1997. 23. Dufresne, A. Conception d’interfaces pour l’apprentissage à distance. La Revue de l'Éducation à Distance, XII (1/2) 1997. 177-200. 24. Martinez, M. and Bunderson, C.V. Foundations for Personalized Web Learning Environments. ALN Magazine, 4 (2) 2000. 25. Paquette, G. and Tchounikine, P. Contribution à l'ingénierie des systèmes conseillers : une approche méthodologique fondée sur l'analyse du modèle de la tâche. Sciences et Techniques Éducatives, 9 2002. (to be published). 51 Interactive Cognitive Modelling Agents - Potential and Challenges Vania Dimitrova School of Computing, Leeds University, Leeds, LS2 9JT, UK, vania@comp.leeds.ac.uk Abstract: Interactive cognitive modelling agents are defined here as diagnostic agents that involve human learners in diagnostic dialogues and extract a picture of the learner's cognition in terms of beliefs, misunderstandings, misconceptions, and reasoning. This paper is written both as a reflection on our recent work on interactive open learner modelling, which is a specific and fairly simplified interactive cognitive modelling method, and as a proposal for developing a framework for interactive cognitive modelling agents. We discuss advantages of the approach and outline pitfalls with the initial architecture suggesting possible research techniques to tackle these problems. Key words: interactive cognitive modelling, meta-cognition, evaluation. 1 Introduction Learners expect to be understood when they ask for advice, assistance, explanation, guidance, tutoring, etc. Effective adaptive learning environments require robust learner models (LMs) (Self, 1999) that represent learners’ preferences, needs, knowledge, misconceptions, skills. Diagnosis is a mutual process – it depends on the diagnosee's involvement and the diagnoser's ability to encourage this involvement (Dillenbourg, 1996). Learners would expect the computer teachers with whom they are interacting to be willing to participate in a discussion about their problems rather than providing quick, incomplete, and incomprehensible responses. The result of such diagnostic interactions is eliciting a picture of the learner's cognition with the active participation both of the learner and the teacher. Not only is interactive diagnosis likely to be more accurate and to enable effective personalisation, but when it does take place in educational situations, it can bring deeper insights for both the teacher in terms of reflection on their own practice and the learner in terms of promoting important metacognitive skills. Such wealth diagnostic interactions are increasingly needed nowadays in many advanced learning environments – particularly systems that require sophisticated learner models and promote meta-cognition to help learners understand themselves what their problems and needs are. In contrast, traditional computer diagnostic systems seek to infer the reasons for the learners' behaviour without directly involving the learners. Recently, approaches that involve learners in diagnosis have been proposed (Bull et al., 1995; Bull & Brna 1999; Kay, 1995; McCalla et al., 2000; Morales et al., 2000; Paiva & Self, 1995; Zapata-Rivera & Greer, 2001). Most of these methods are concerned with open learner modelling where the learners are provided with the means to inspect and change the models the systems build of them. Commonly, these systems externalise the LMs in some viewers and provide menu options for the learners to change the content of their models. The users can sometimes ask for explanations and justifications of the computer's opinions. The approach proposed by Bull (1997) suggests an enhanced method of interaction in a menu-based environment for negotiating the learner model. When inconsistencies between the student and the computer’s views about the student’s beliefs are identified, negotiative dialogue is triggered. However, naturally occurring human-human diagnostic interactions, e.g. peer diagnostic systems (Bull & Brna, 1999), accommodate a richer set of communicative activities – the dialogue comprises multiple exchanges, no one controls or restricts what another may say (at the same time some participants, e.g. the teacher, might be attributed a guidance role), and all parties can change the topic and initiate a new discussion. Such rich interactions that aim at extracting a model of the learner’s cognition are addressed in this paper. 52 This position paper will discuss the design of a new diagnostic architecture - artificial agents that understand human users by talking and listening to them. More specifically, we will present Interactive Cognitive Modelling Agents (ICOMAs) - computer diagnosers capable of involving learners in an ongoing dialogue that results in eliciting models of the learners’ cognition with the active participation of the learners. We will present an initial exploration of the approach - interactive open learner modelling, illustrated in the system STyLE-OLM. Advantages and possible applications of ICOMAs will then be discussed. Finally, we will sketch out further research towards extending STyLE-OLM to provide a framework for interactive cognitive modelling. We will outline challenges and will speculate about possible solutions in order to accommodate more advanced domain reasoning, richer communication means, extended dialogue management, and enhanced learner model maintenance. 2 Interactive Cognitive Modelling Agents Human teachers follow a variety of diagnostic tactics when they guide diagnostic dialogue. To illustrate, let us consider an excerpt from the communication between a teacher and a language student who studies for technical translator and faces problems with some domain terms. Similar dialogues have been observed in early studies we have conducted with university students in Bulgaria (Dimitrova & Dicheva, 1998). Note, that the actual dialogue was in Bulgarian, the translation below is done by the author. The domain concepts mentioned in the dialogue are given in small capital letters. [1] Learner. about [2] it. Is I Teacher. your We question. [3] Learner. [4] Teacher. OPERATING [5] is [6] an a No. Teacher. discuss of some OPERATING Isn't example for PASCAL but I know nothing of other terms before I can answer SYSTEMS? this MSDOS? an You While with OPERATING the icons. [9] Teacher. to SYSTEM. OPERATING [10] Learner. BASIC at [11] Teacher. meant SYSTEM WINDOWS to the WINDOWS return You about mean run example the Do MsDos have the to your PROGRAMMING languages an as you know any other and am confused now. it. maintain MS D O S OPERATING is the one communication example of an SYSTEM. MSDOS? OPERATING OPERATING write a that on Different with to of same provides back PROGRAMS that same. people SYSTEM, also is contact MS D O S Let’s COMPUTER PROGRAMS another exactly in talk the is mean Not OPERATING are SYSTEMS and people and thought WINDOWS COMPUTERS. a about SYSTEM? OPERATING SYSTEM. [8] P ASCAL . to so. one talking WINDOWS? heard think I COMPUTER Learner. need need you is OPERATING [7] we I MSDOS Teacher. means will people to SYSTEMS? between OPERATING heard related Have Yes, Learner. What have PASCAL in COMMANDS GRAPHICAL question. SYSTEMS USER To provide and SYSTEM to order INTERFACE compare different control to the communicate which WINDOWS uses and menus PASCAL, LANGUAGES. used to write programs? My friends did some school. That PROGRAMMING COMPUTER with is right. LANGUAGE the help BASIC allow of an is an example programmers OPERATING SYSTEM, to of write for a PROGRAMMING PROGRAMS , example LANGUAGE. which are So is run on WINDOWS. This example shows that diagnostic dialogues are naturally embedded into the whole teaching process. We propose that these dialogues are conducted by Interactive Cognitive Modelling Agents which aim at extracting a picture of the learner’s cognition. Such agents will have to share resources with other parts of an interactive learning environment, for example, domain expertise and learner model. These agents may serve as main diagnostic components in learning environments. They could also be used together with other diagnostic methods (e.g. assessing learners’ drill performance) to enhance the quality of the learner models by addressing aspects that may well have been missed or diagnosed wrongly by the traditional diagnostic methods. To build models of ICOMAs, we will make the following assumptions: • ICOMAs share common communication means with the learners where domain facts are discussed and models of the learners' conceptual understanding are extracted. The communication language should allow effective diagnostic interactions where both a diagnosee and a diagnoser participate actively. 53 • The interaction comprises a sequence of episodes which span over multiple turns and follow specific diagnostic tactics. ICOMAs plan the content of the interaction and take diagnostic decisions based on their domain expertise. These agents are empowered by discourse knowledge that enables them to lead a coherent interaction aimed at eliciting a picture of the diagnosee’s conceptual understanding • ICOMAs' aim is to elicit a picture of the learner’s conceptual understanding in terms of beliefs (as in the example above), misunderstandings, misconceptions, and reasoning. These agents must incorporate appropriate reasoning capabilities that enable the extraction of an interactively constructed learner model. Computational frameworks of interactive cognitive modelling agents will allow understanding the process and will provide vehicles for building robust computer simulations of interactive teachers capable of understanding the learners' problems and needs. Moreover, formalisations will aid the implementation of diagnostic agents in various domains. 3 Initial Exploration – STyLE-OLM We have examined a specific interactive cognitive modelling method, called interactive open learner modelling (IOLM), where a learner is provided with the means to inspect and discuss the conceptual models that computer systems build of them (Dimitrova, 2001). Despite the fact that IOLM agents focus mainly on discussing the content of the learner model and demonstrate a fairly simplified case of the interaction discussed above, it has confirmed the feasibility of the assumptions discussed in section 2. Many techniques from this specific method appear fruitful in the more advanced diagnostic model the ICOMAs address. A formal framework for interactive open learner modelling has been developed (Dimitrova et al., 1999; Dimitrova et al., 2000; Dimitrova 2001). It includes distinctive components: a discourse model based on an approach known as dialogue games manages diagnostic interactions and provides both a diagnoser and a diagnosee with a common communication method and symmetrical power in dialogue maintenance (Dimitrova et al. 1999, Dimitrova et al., 2002) while a formally defined mechanism based on a belief modal operator adopts nonmonotonic reasoning to maintain a jointly constructed LM (Dimitrova et al., 2000). The framework has been demonstrated in STyLE-OLM - an IOLM system in a terminological domain (Dimitrova, 2002). Two instantiations of STyLE-OLM – in Computer Science and in Finance - have been developed, the example here is from an experimental study with the system conducted in a Finance domain (topic – Financial Markets). The study involved seven post graduate students from the author's department. More details about the study are given in Dimitrova (2001). The excerpt below shows a fragment of an interaction with STyLE-OLM where the system challenges the robustness of a learner's conceptualisation, which provides for reflection and scaffolding of the learner’s domain knowledge. Initially, the learner model contains information that the learner knows some aspects of the terms MONEY MARKET and CAPITAL MARKET but often mixes these terms. STyLE- OLM initiates a dialogue to explore the learner's knowledge about these terms and to find possible reasons for the confusion. [1] STyLE-OLM[inquire] [2] Learner[agree] [3] STyLE-OLM[inquire] [4] Learner[inform] I DO YOU AGREE I THINK that CAPITAL THINK MONEY MARKET MARKET that MONEY MARKET raises operates operates CAPITAL CAPITAL MARKET and with with SHORT TERM INVESTMENTS. SHORT TERM INVESTMENTS. operates operates with with WHAT? SHORT TERM INVESTMENTS. The dialogue follows a game tactic that elaborates on the similarities and differences between the two terms. Move [4], however, reveals an erroneous aspect of a learner's knowledge: the proposition 'CAPITAL MARKET operates with SHORT TERM INVESTMENTS', which the learner is assumed to believe, is not confirmed by the diagnoser’s domain knowledge. A new dialogue episode is initiated searching for some explanation for the learner's error. [5] STyLE-OLM[challenge] WHY DO YOU THINK CAPITAL MARKET operates with SHORT TERM INVESTMENTS? [6] Learner[justify] are BECAUSE CAPITAL MARKET operates with TREASURY BILLS and TREASURY BILLS SHORT TERM INVESTMENTS . 54 The screen shot in Figure 1 depicts the environment at the moment when the learner constructs his justification. The system now registers a possible reason for misattribution: the student may believe wrongly that 'CAPITAL MARKET operates with SHORT TERM INVESTMENTS' because he believes that 'CAPITAL MARKET operates with TREASURY BILLS and TREASURY BILLS are SHORT TERM INVESTMENTS'. Since the last belief has also not been confirmed by its domain expertise, STyLE-OLM initiates a new dialogue episode for explaining this error: [7] STyLE-OLM[challenge] and WHY TREASURY BILLS are DO YOU THINK CAPITAL MARKET operates with TREASURY BILLS SHORT TERM INVESTMENTS ? Buttons for Graphical browsing tools for through the constructing dialogue CGs history Button to switch to BROWSE mode The learner constructs and selects a CG with the proposition of his justification Textual To justify the representation learner selects of the system's a justify move challenge Dialogue history Figure 1. Learner's justification in STyLE-OLM (move [6]). At this time, the student is confused. The robustness of his domain beliefs is challenged. He switches to a BROWSE mode where he browses through the beliefs in the LM and sees the level of correctness the system assignes to them. Having seen that the belief presented in Figure 2 has been classified as known wrongly by the system, TREASURY BILLS the learner realises that his claim that CAPITAL MARKET operates with and TREASURY BILLS are SHORT TERM INVESTMENTS might be wrong. 55 Buttons for browsing through the beliefs in the LM Button to switch to DISCUSS mode A belief from the LM presented with a CG Degree of correctness the system assigns to the belief, represented with the CG History of the preceding discussion Figure 2. The STyLE-OLM environment in BROWSE – the learner browses the beliefs in the LM provoked by the system's challenges in move [7]. He now needs the system's help to explore facts about these terms. He switches back to DISCUSS mode to continue the discussion of the LM. He asks a question: [8] Learner[inquire] IS [9] STyLE-OLM[inform] IT I TRUE KNOW TREASURY BILLS that that TREASURY BILLS are are SHORT TERM INVESTMENTS ? SHORT TERM INVESTMENTS . The learner realises that one part of his statement in [6] is correct and challenges the other: [10] Learner[inquire] IS IT TRUE that CAPITAL MARKET operates with SHORT TERM INVESTMENTS? [11]STyLE-OLM[inform] I DO NOT KNOW that CAPITAL MARKET operates with SHORT TERM INVESTMENTS. The learner has clarified the part of his beliefs which is wrong. He withdraws his claim that CAPITAL MARKET operates with SHORT TERM INVESTMENTS. [12]Learner[disagree] I DISAGREE that CAPITAL MARKET operates with SHORT TERM INVESTMENTS? The interaction continues with the comparison between MONEY MARKET and CAPITAL MARKET. Due to space restrictions, we will not present the whole dialogue here. A full transcript of this dialogue as well as some other example interactions with STyLE-OLM are given in Dimitrova (2001). After the dialogue is terminated, a jointly constructed learner model that takes into account what has been expressed by the two agents during the interaction is obtained (e.g. the beief 'C APITAL MARKET operates with SHORT TERM INVESTMENTS' will be deleted from the initial LM). STyLE-OLM allows inspecting and discussing the learner model in a relatively expressive graphical manner which fosters the articulation of domain knowledge and can lead to conceptual understanding. A constructive dialogue guided by the system enables the exploration of aspects of a learner's domain knowledge and the extension of the scope of beliefs in the learner model. Learners are provided with a symmetrical role in maintaining the dialogue. A flexible diagnostic mechanism allows the management of a learner model jointly constructed by the computer system and the learner with the latter being provided with equal power to influence the diagnosis. 56 3.1 Potential of Interactive Cognitive Modelling Agents Interactive cognitive modelling agents have a strong potential in advanced learning environments capable of evaluative tailoring study to with the needs of STyLE-OLM the learners has and demonstrated promoting meta-cognitive advantages of the processes. approach in The terms of improving the quality of the learner model and providing the means for reflective activities (Dimitrova et al., 2001, Dimitrova, 2002). Improving the quality of the learner model. We observed fewer inconsistencies in the resulting LM, • a larger scope of learner's beliefs, and some explanations of the learner's errors. The obtained LM included a higher proportion of valid assertions about the learner's knowledge and minimised the number of not valid assertions about the learner's knowledge. Providing means for reflective activities. The study allowed us to monitor the following reflective • activities with STyLE-OLM: the students rendered statements about their domain beliefs, they went back to claims about their beliefs and (sometimes) changed these claims, and they investigated arguments to support their beliefs. While more knowledgeable learners were engaged in reflective dialogues about the domain, less knowledgeable learners were provoked to inspect their models and challenge the robustness of these models. ICOMAs can be embedded in advanced e-learning systems to enable better understanding of the learners and to help learners understand themselves what their accomplishments and problems are. There is a growing interest in finding robust and computationally tractable methods for eliciting models of the users’ cognitive states to aid the development of personalised systems in various domains, especially in the increasingly popular Internet applications. ICOMAs developing sophisticated personalised Internet agents, for example can be personal used as a basis e-consultants or for e- metntors. Personal e-consultants are interactive agents that offer personalised advice tailored to the users' • problems and needs. Such agents may be incorporated in modern e-commerce or e-banking systems. For example, a non expert in financial planning seeking to understand the concept of an "ISA" may be provided with a personal e-consultant that discusses the domain terminology with the user, infers a model of the user's conceptual understanding, and offers adaptive explanations tailored to the user's understanding of the terminology. E-mentors are agents that act as personal mentors. Mentoring is a relationship in which one person - • usually someone more experienced - helps another to discover more about themselves, their potential and their capability. The mentor's role is to listen, ask questions, probe for facts and understand its mentee and to act as a source of information, experience, and advice. Artificial mentors could be embedded in new generation e-training systems to provide the means to understand the trainees, offer personalised help, and help the trainees identify themselves what their needs are. While the evaluation of STyLE-OLM outlined potentials of the IOLM framework, it also revealed unsolved aspects that led to pitfalls of the architecture. We will sketch out these aspects next and will draw speculations about how they may be addressed in a more sophisticated framework for ICOMAs. 4 Challenges In this section, based on examining learner interactions with STyLE-OLM, we outline further improvements of the IOLM framework in order to maintain enhanced diagnostic interactions required in interactive cognitive modelling. 4.1 Exploring advanced domain inference Interactive cognitive modelling requires high level logic in order to develop appropriate tactics to reveal reasons for users’ misconceptions. We have experimented with conceptual graphs, which have been found a suitable formalisation for the purposes of interactive diagnosis. However, some commonsense reasoners, such as modus tollens, which are often applied by humans have been difficult to capture. This led sometimes to missing student reasoners and interrupting profitable dialogue episodes. In order to address negations of domain propositions represented with conceptual graphs, an 57 additional modal operator not shall be considered (Sowa, 1984), a methodology how this can be implemented in computer applications is discussed in Dau (2000). ICOMAs need to analyse propositions composed by the users. It is difficult to predict how the learners will express their propositions. Even in a highly structured graphical communication language exploited in STyLE-OLM (Dimitrova et al., 2002), computational problems with ambiguity of domain propositions became apparent. Firstly, many relations have overlapping of their meanings in everyday language. When learner mixed such relations (e.g. "agent" and "actor", see (Sowa, 1984)) StyLE-OLM assigned erroneous conceptualisation while the learners had simply confused very similar words. Dealing with mixed relations requires some representation of interdependencies between relations and suitable reasoning to find relation similarities. For example, λ-definitions of relations used in conceptual graphs theory (Sowa, 1984) may empower such reasoning. Secondly, often a proposition is a re-phrase of another, which, if not captured by the domain reasoning mechanism, may lead to misdiagnosis or obscure dialogue moves such as repetition or inappropriate challenging. Extended comparison techniques to allow for different perspectives of the same knowledge to be captured are needed. Mechanisms, similar to those presented in (Dieng & Hug, 1998; Martin, 2000) seem applicable in ICOMAs. Dealing with ambiguity of domain propositions requires not only discovering potential ambiguous situations but also addressing them in the dialogue. Meta-dialogue for dealing with mis- communications and grounding, e.g. (Traum & Dillenbourg, 1996), need to be incorporated in the dialogue maintenance mechanism. The study with STyLE-OLM showed the need to handle reasoning under incomplete domain expertise. When the system did not have information about a domain fact, it simply assumed that this was an erroneous belief and challenged it, which led to inappropriate system behaviour at times (e.g. a learner's statement "A bank operates with money" was not confirmed by the system's domain expertise and challenged "Why do you think a bank operates with money", which frustrated the learner). A less knowledge-centred behaviour of the diagnostic agent is required. We may envisage that at times the diagnostic agent behaves as a peer who may extend its competence to incorporate information provided by the learner depending on its trust in the learner and its own domain reasoning. Planning diagnostic dialogue when the diagnoser's domain expertise is incomplete seems to relate to decision making under uncertainty which deals with reasoning that require information not available at the time it is needed. One way to tackle the problem is to employ some form of defeasible reasoning, i.e. to make some assumptions from which some conclusions may be drawn and withdraw the assumptions later on if the assumption is proven invalid (Davis, 1990; Parsons, 2001). In this case, ICOMAs will need to have a mechanism for dealing with the degree of certainty about the truth of domain propositions. Another possible method to deal with incomplete domain expertise is argumentation (Krause & Clark, 1993; Parsons, 2001). For instance, an ICOMA may accept a proposition suggested by a student if it cannot find a rebuttal for it. In this case, the agent needs to be able to incorporate some kind of argumentative reasoning in its dialogue planning. 4.2 Providing rich communication means STyLE-OLM provided a graphical communication medium combining propositions represented as conceptual graphs and illocutions represented with sentence openers. While, such environment was found favourable for diagnostic interactions (Dimitrova et al., 2002), some problems were also identified. Firstly, mixing textual and graphical representations requires keeping the meaning of both representations coherent. When graphics is utilised for constructing dialogue utterances, a sophisticated mechanism for linguistically generating coherent text linguistic that expressions represents the from graphics meaning of the is needed graphical in order to provide expressions. When communication is based on conceptual graphs, natural language processing approaches that generate text from graphs, e.g. (Angelova & Bontcheva, 1996; Nikolov et al., 1995), may be employed. Secondly, there is no comprehensive study of the type of operations needed when communication is done with graphics, for example how to facilitate the construction of graphical utterances, the modification of graphical "propositions", the search through graphical expressions, etc. We adopted a 58 rather heuristic approach following conventional operations used in graphical packages but it became apparent that a more systematic approach is needed to examine the effectiveness and pitfalls of these operations. In this line, approaches from Human-Computer Interaction seem favourable, for example (Green & Petre, 1996). Thirdly, the participants in the evaluation of STyLE-OLM did not agree regarding their choice of graphics or text for communication. The study was too limited to discuss this issue deeply, and further exploration is needed. In this line, (Cox, 1999) provides possible directions highlighting the difference between situations in which a presented external representation is interpreted and situations in which participants construct external representation (both types of situations are present in communicating with diagrams). As Bull et al. (2001) argue, differences in the learners' cognitive styles impose a variety of communication means to be combined in a single system. Since the associations between cognitive style and presentation format are not straightforward, Bull et al. propose that learners' should be given the choice of a textual or graphical environment (in domains for which either may be appropriate) for discussing their cognitive model. Providing text input would require student diagnosis based on a free text, which is a challenging computational task at present, e.g. the learners' statements may not make sense according to the system's domain model. Recent research in natural language processing is addressing relevant aspects (e.g. Ramsay & Seville, 2000) and one would expect in due course interactive cognitive modelling to accommodate communication in a free natural language. This would open a new research issue of how to accommodate misunderstanding, repair and grounding (e.g. Traum & Dillenbourg, 1996) in a dialogue which is aimed at student diagnosis. 4.3 Maintaining a coherent diagnostic dialogue Maintaining a coherent diagnostic dialogue requires dealing with vague and incomplete information about possible learner's misconceptions and suitable diagnostic tactics. Reasons for people’s cognitive errors are generally difficult to define. There is a fair bit of work done on defining theories for concept learning that deal with the correct application of rules such as generalisation, specification, similarity (Thagard, 1982). However, very little has been done to precisely define types of rules that people apply wrongly or fail to apply, which leads to building wrong or incomplete conceptual models. In this line, in order to deal with vagueness of defining diagnostic tactics, ICOMs would need to appropriately incorporate methods for reasoning under uncertainty (Krause & Clark, 1993; Parsons, 2001). An important role of ICOMAs is to scaffold learners' conceptual understanding. This can be achieved via adopting a variety of strategies to discuss wrong beliefs, e.g. comparison (Milosavlevic, 1997), explanation (Bontcheva & Wilks, 1999; McCoy, 1989), inquiry dialogue (Wong et al., 1998). In addition, learners need to be navigated through the dialogue in order to help them understand the semantic structure of the conversation. At times, the learners in the study with STyLE-OLM felt that the system was jumping between different propositions without any obvious reason for this, and the learners failed to follow what was going on. Diagnostic dialogue can be improved if the diagnoser makes apparent the rhetorical structure of the conversation, for instance what the current dialogue tactic is, why the system is changing the tactic, when the system goes back to a previously initiated dialogue episode, etc. To illustrate this, we present below an improvement of the dialogue example given above with added phrases to explain the rhetorical structure of the conversation. [1] STyLE-OLM[inquire] MARKET. MARKET We It can operates [2] Learner[agree] [3] STyLE-OLM[inquire] trying I to seems try to with AGREE We of SHORT that just compare definition that you compare MONEY MARKET MARKET MARKET. the two terms terms MONEY here. DO MARKET and YOU THINK CAPITAL MONEY INVESTMENTS. mentioned MONEY CAPITAL TERM confuse these operates the and CAPITAL CAPITAL with definition SHORT of MARKET, MARKET so raises TERM MONEY let CAPITAL INVESTMENTS. MARKET. us and We are consider operates the with WHAT? [4] Learner[inform] [5] STyLE-OLM[challenge] operate that [6] I with CAPITAL Learner[justify] TREASURY THINK see, SHORT MARKET IT BILLS I IS are that CAPITAL you TERM think TRUE with BECAUSE TERM operates that INVESTMENTS. operates SHORT MARKET Can SHORT CAPITAL both MONEY you TERM with SHORT MARKET explain TERM and what INVESTMENTS. CAPITAL makes MARKET you think INVESTMENTS? MARKET operates with TREASURY BILLS and INVESTMENTS. 59 [7] STyLE-OLM[challenge] you and to TREASURY Learner[inquire] IS [9] STyLE-OLM[inform] are IT Learner[inquire] IS [11] STyLE-OLM[inform] IT To operates a SHORT to my I TERM CAPITAL of TREASURY know you believed. operates MARKET with Can I TREASURY ask BILLS INVESTMENTS? TREASURY that didn't CAPITAL BILLS knowledge, best with that believe that TRUE the fact you TRUE Well, [10] is that BILLS [8] MARKET This confirm my are TREASURY MARKET SHORT BILLS TERM are operates knowledge, I INVESTMENTS? SHORT with can't TERM INVESTMENTS. TREASURY confirm BILLS? that CAPITAL BILLS. Sessions with STyLE-OLM have revealed that both the diagnoser and the diagnosee may need to express uncertainty in their dialogue utterances. Fir example, the diagnoser may deal with information that is not available and may need to make assumptions about the student's knowledge, while the student may not be completely sure about the validity of their statements. Consequently, diagnostic dialogue has to accommodate different verbal expressions of uncertainty, such as definite, likely, possible, unlikely, and impossible (Krause & Clark, 1993). 4.4 Eliciting a learner model under uncertain conditions The result of a method for student modelling is a model of the student’s cognition. Following the discussion above, it is apparent that the mechanism for eliciting a resultant student model has to accommodate reasoning under uncertainty. While some level of uncertainty might be handled via interaction enabling agents to challenge or withdraw their beliefs and to clarify their statements, a more elaborated notion of uncertainty would be sensible. For example, representing some strength of beliefs in the learner model (e.g. 'entirely sure', 'not very sure', 'guessing') and making plausible inferences that incorporate degree of belief and nonmonotonic reasoning (Parsons, 2001). The study with STyLE-OLM confirmed that inconsistency is often a case in students' beliefs. Although clarification dialogue in STyLE-OLM did enable us to handle inconsistency in student beliefs and avoid extensive belief revision (Giangrandi & Tasso, 1995), we found that some contradicting propositions were left due to limitations of the system's reasoning (e.g. a learner was thought to believe both ‘CAPITAL operates BILLS MARKET with are not LONG operates TREASURY TERM with BILLS ’ LONG TERM INVESTMENTS’ (correct) and ‘CAPITAL MARKET (wrong), which are actually contradictory because TREASURY INVESTMENTS b u t SHORT TERM INVESTMENTS). Therefore, a reasoning mechanism that explores deeper all consequences of the student's claims is required. A feasible approach seems advanced nonmonotonic reasoning (Davis, 1990). The mechanism for ascribing participants' beliefs from their communicative acts would have to adjust the belief ascription according to the agents' goals. In STyLE-OLM, when asking questions learners were diagnosed that they did not know a domain fact. However, a question may not always indicate missing knowledge, but may sometimes mean that the students seek for confirmations of domain aspects they know. Further extensions need to take into account theories that deal with the repair of mistaken ascriptions, e.g. (Lee & Wilks, 1997). 5 Conclusions This paper is written both as a reflection on our recent work on interactive open learner modelling and as a proposal for further research on interactive cognitive modelling. Our long term goal is to develop robust and efficient models of computer agents that can conduct diagnostic dialogues with a learner (or a group of learners) in order to understand the learners and help themselves understand what their problems and needs are. One type of such diagnostic agents - Interactive Cognitive Modelling Agents have been discussed in this paper. ICOMAs are interactive diagnostic agents that involve human learners in diagnostic dialogues and extracts a picture of the learner's cognition in terms of beliefs, misunderstandings, misconceptions, and reasoning. As an initial exploration of the approach we have examined a method called interactive open learner modelling where a computer diagnoser enables a human learner to inspect and discuss the model the diagnoser builds of him/her. STyLE-OLM - the system we have built to illustrate our IOLM framework - is a rather simplified demonstration of ICOMA. However, it did allow us to observe some advantages of the approach, which were outlined in 60 the paper. We have also sketched out potential problems with STyLE-OLM and have pointed to possible methods to tackle these problems in an enhanced architecture of interactive cognitive modelling agents. References Angelova, G., & Bontcheva, K. (1996). DB-MAT: Knowledge acquisition, processing and NL generation using conceptual graphs. In P.W. Eklund, G. Ellis & G. Mann (Eds.), Conceptual structures: knowledge representation as interlingua: 4th international conference on conceptual structures (pp. 115-128). Springer, LNAI 115. Bontcheva, K., & Wilks, Y. (1999). Generation of adaptive (hyper)text explanations with an agent In Proceedings of the European Workshop on Language Generation (EWNLG'99), model. Toulouse, France. Bull, S. (1997). Collaborative student modelling in foreign language learning. PhD Thesis. University of Edinburgh. Bull, S., & Brna, P. (1999). Enhancing peer interaction in the Solar system. In P. Brna, M. Baker & K. Stenning (Eds.). Roles of communicative interaction in learning to model in Mathematics and Science: Proceedings of the C-LEMMAS conference, Ajaccio, Corsica. Bull, S., Brna, P., & Pain, H. (1995). Extending the scope of student models. User Modeling and UserAdapted Interaction, 5(1), 45-65. Bull, S., Dimitrova, V., & Brna, P. (2001). Enhancing reflective modelling through communicative interaction in learning environments. In P. Brna, K. Stenning, M. Baker & A. Tiberghien (Eds.), The role of communication in learning to model. Mahwah, NJ: Lawrence Erlbaum Associates, in press. Cox, R. (1999). Representation construction, externalised cognition and individual differences. Learning and Instruction, 9, 343-363. Dau, F. (2000). Negations Conceptual in structures: simple conceptual logical, linguistic, graphs. and In B. Ganter computational & G. issues: W. Mineau (Eds.), 8th International conference on conceptual structures. Berlin London: Springer, 2000. Davis, E. (1990). Representations of commonsense knowledge. San Mateo: Morgan Kaufmann. Davis, E. (1990). Representations of commonsense knowledge. San Mateo: Morgan Kaufmann. Dieng, R., & Hug, S. (1998). MULTIKAT, a Tool for comparing knowledge of multiple experts. In ML. Mugnier & M. Chein (Eds.), Conceptual structures: theory, tools, and applications: 6th International conference on conceptual structures (pp. 139-153). Springer, LNAI 1453. Dillenbourg, P. (1996). Some technical implications of distributed cognition on the design on interactive learning environments. Journal of Artificial Intelligence in Education, 7 (2), 161180. Dimitrova, V. (2001) Interactive open learner modelling. PhD thesis, University of Leeds. Dimitrova, V. (2002). STyLE-OLM: Interactive open learner modelling. Technical report, School of Computing, Leeds University (submitted to the International Journal of Artificial Intelligenec in Education). Dimitrova, V., & Dicheva, D. (1998). "Who is Who": The Roles in an Intelligent System for Foreign Dimitrova, V., Brna, P., Self, J.A. (2002). The design and implementation of a graphical communication medium for interactive open learner modelling. In Proceedings of ITS2002 – Intelligent Tutoring Systems Conference, Springer, LNCS. Dimitrova, V., Self, J.A. & Brna, P. (2001). Applying interactive open learner models to learning technical terminology. In Proceedings of UM2001 – User Modelling conference. Springer, LNCS. Dimitrova, V., Self, J.A., & Brna, P. (1999). The interactive maintenance of open learner models. In S.P.Lajoie & M. Vivet (Eds.), Artificial intelligence in education (pp. 405-412). Amsterdam: IOS Press. Dimitrova, V., Self, J.A., & Brna, P. (2000). Maintaining a jointly constructed student model. In S.A. Cerri & D. Dochev (Eds.), Artificial intelligence: Methodology, systems, and applications (pp. 221-231). Berlin: Springer, Lecture Notes of Artificial Intelligence. Giangrandi, P. & Tasso, P. (1995). Truth maintenance techniques for modeling student’s behavior. Journal of Artificial Intelligence in Education, 6, 153-202. Green, T.G., & Petre, M. (1996). Usability analysis of visual programming environments: a 'cognitive dimensions' framework. J of Visual Languages and Computing, 7, 131-174. 61 Kay, J. (1995). The UM toolkit for cooperative user modelling. User Modeling and User-Adapted Interaction, 4, 149-196. Krause, K., & Clark, D. (1993). Representing uncertain knowledge: An Artificial Intelligence approach. Intellect. Oxford. Language Terminology Learning. British Journal of Educational Technology, Vol. 29 (1), 47-57. Martin, Ph. (2000). Conventions and notations for knowledge representation and retrieval. In B. Ganter & G. W. Mineau (Eds.), Conceptual structures: logical, linguistic, and computational issues: 8th International conference on conceptual structures. Berlin London: Springer. McCalla, G., Vassileva, J., Greer, J., & Bull, S. (2000). Active learner modelling. In G. Gauthier, C. Frasson & K. VanLehn (Eds.), Intelligent tutoring systems (pp. 53-62). Berlin Heidelberg: Springer-Verlag. McCoy, K. (1989). Highlighting a user model to respond to misconceptions. In A. Kobsa & W. Wahlster (Eds.), User models in dialogue systems (pp. 233-253). Springer-Verlag, Berlin. Milosavljevic, M. (1997). Augmenting the user’s knowledge via comparison. In A. Jameson, C. Paris & C. Tasso (Eds.). Proceedings of the 6th International conference on User Modeling. Available on-line from http://um.org. Morales, R., Pain, H., & Conlon, T. (2000). Understandable learner models for a sensorimotor control task. In G. Gauthier, C. Frasson & K. VanLehn (Eds), Proceedings of the 5th International conference on Intelligent Tutoring Systems (pp. 222-231). Springer, LNCS. Nicolov, N., Mellish, C., & Ritchie, G. (1995). Sentence generation from conceptual graphs. In G. Ellis et al. (Eds.), Conceptual structures: Applications, implementation and theory: 3rd International conference on conceptual structures (pp. 74-88). Springer, LNAI 954. Paiva, A., & Self, J.A. (1995). TAGUS - a user and learner modelling workbench. User Modeling and User-Adapted Interaction, 4, 197-226. Parsons, S. (2001). Qualitative methods for reasoning under uncertainty. The MIT Press. Cambridge. Ramsay, A., & Seville, H. (2000). What did he mean by that?. In S. Cerri & D. Dochev (Eds.), Artificial Intelligence: Methodology, systems and applications, 9th International AIMSA Conference (pp. 199-209). Springer, LNAI 1904. Self, J. A. (1999). The defining characteristics of intelligent tutoring systems research: ITSs care, precisely. International Journal of Artificial Intelligence in Education, 10, 350-364. Sowa, J. (1984). Conceptual structures: Information processing in mind and machine. MA: AddisonWesley. Thagard, P. (1992). Conceptual revolutions. Princeton University Press. Traum, D., & Dillenbourg, P. (1996). Miscommunication in multi-modal collaboration. In Working Notes of the AAAI Workshop on Detecting, Repairing, And Preventing Human--Machine Miscommunication. Walton, D. (1984). Logical dialogue games and fallacies. University Press of America, Lanham. Wong, L.H., Quek, C., & Looi, C.K. (1998). TAP-2: a framework for an inquiry dialogue based tutoring system. International Journal of Artificial Intelligence in Education, 9, 88-110. Zapata-Rivera, J-D., & Greer, J. (2000). Inspecting and visualising distributed Bayesian student models. In G. Gauthier, C. Frasson & K. VanLehn (Eds), Proceedings of the 5th International conference on Intelligent Tutoring Systems (pp. 544-553). Springer, LNCS. 62 Authors (in order of their appearance) Susan Bull Electronic, Electrical and Computer Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK Theson Nghiem Electronic, Electrical and Computer Engineering, University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK Douglas Chesher School of Pathology, University of Sydney, Australia 2006 Judy Kay School of Information Technologies, University of Sydney, Australia 2006 Nicholas King School of Pathology, University of Sydney, Australia 2006 Maria Grigoriadou Department of Informatics, University of Athens, GR-157.71, Athens, Greece Maria Samarakou Department of Energy Technology, Technological Institute of Athens, GR- 122.61, Athens Greece, Dionissis Mitropoulos Department of Informatics, University of Athens, GR-157.71, Athens, Greece Michael Panagiotou 01-Pliroforiki S.A., 438 Acharnon Str., GR-111.43, Athens, Greece Adja F. de Andrade Computer Based Learning Unit, Leeds University, Leeds, LS2 9JT, UK Paul Brna School of Computing and Mathematics, Northumbria University, Newcastle upon Tyne NE1 8ST, UK Rosa Maria Vicari PGIE- UFRGS, Caixa Postal: 5071 - Zip Code: 90041-970, Porto Alegre-RS, Brasil Aude Dufresne Département de Communication, Université de Montréal, C.P. 6128 Succ CentreVille, Montréal, Qc, Canada Martin Hudon Département de Communication, Université de Montréal, C.P. 6128 Succ Centre- Vania Dimitrova School of Computing, Leeds University, Leeds, LS2 9JT, UK Ville, Montréal, Qc, Canada 63