Student Models that Invite the Learner In: The SMILI Open Learner Modelling Framework Susan Bull, Electronic, Electrical and Computer Engineering, University of Birmingham, UK s.bull@bham.ac.uk http://www.eee.bham.ac.uk/bull Judy Kay, School of Information Technologies, University of Sydney, Australia judy@it.usyd.edu.au http://www.cs.usyd.edu.au/~judy Abstract. In recent years, the learner models of some adaptive learning environments have been opened to the learners they represent. However, as yet there is no standard way of describing and analysing these ‘open learner models’. This is, in part, due to the variety of issues that can be important or relevant in any particular learner model. The lack of a framework to discuss open learner models poses several difficulties: there is no systematic way to analyse and describe the open learner models of any one system; there is no systematic way to compare the features of open learner models in different systems; and the designers of each new adaptive learning system must repeatedly tread the same path of studying the many diverse uses and approaches of open learner modelling so that they might determine how to make use of open learner modelling in their system. We believe this is a serious barrier to the effective use of open learner models. This paper presents such a framework, and gives examples of its use to describe and compare adaptive educational systems. Keywords. Open learner model, framework for open learner modelling INTRODUCTION Open learner models are learner models that are accessible to the user – usually the learner being modelled, but sometimes also to other users. Self (1999) points out that various components of learning environments can be usefully made open, stating that “it is only the fact that learner models have traditionally been ‘closed’ that opening them up is such a big deal”. Nevertheless, this does not diminish the importance of open learner models as a means to promote educational benefits such as encouraging learner reflection and metacognition; and to consider implementation or technical issues such as achieving an accurate learner model. In recent years there has been increasing interest in opening the learner model for these and for other purposes. The externalisation of the learner model can be in a simple form such as: a skill meter that is a part-shaded bar showing learner progress as a subset of expert knowledge (Papanikolaou et al., 2003; Weber & Brusilovsky, 2001); or the probability that a learner knows a concept (Corbett & Anderson, 1995; Corbett & Bhatnagar, 1997); or a user’s knowledge level compared to the combined knowledge of other groups of users (Linton & Schaefer, 2000). Skill meters have been extended to show progress as a subset of material covered which is, in turn, a subset of expert knowledge (Mitrovic & Martin, 2002); or a further extension also allowing the existence of misconceptions and size of topic to be included in the skill meter (Bull & McEvoy, 2003). Other simple representations have been used, such as: a graphical format illustrating the extent of knowledge above the ‘neutral line’, and areas of difficulty below the line (Bull & Nghiem, 2002); arrows in a target, where the number of arrows represents the level of knowledge of a topic, and the colour intensity of the target represents the relevance of the topic to the current learning goal (Brusilovsky et al., 2004); a haptic learner model using a haptic feedback device enabling a learner to ‘feel’ the extent of their knowledge, where known concepts feel hard, difficulties are soft, and misconceptions are soft and sticky (Lloyd & Bull, 2006); and graphical representations to externalise the learner model for a sensorimotor control task, using arcs, boxes, arrows and animations at different velocities (Morales et al., 2000). Open learner models for young children need to be simple in format in order that the children can understand them. Examples include knowledge level represented as coloured magic wands (aiming for gold, silver and bronze) for 7-8 year old children (Bull et al., 2005); and different smiling faces representing different levels of knowledge for 8-9 year olds (Bull & McKay, 2004). Alternatively, the information presented to the learner can be more complex, such as: a graphical externalisation of a Bayesian network (Zapata-Rivera & Greer, 2001); a hierarchical tree structure (Kay, 1997); a conceptual graph (Dimitrova, 2003); textual descriptions of knowledge and misconceptions (Bull & Pain, 1995); or textual explanations of a fuzzy logic model (Mohanarajah et al., 2005). Multiple views of learner model data, offering learners a choice of how to access the model contents, are also possible in both complex open learner models (Mabbott & Bull, 2006), and simple open learner models (Bull & Mabbott, 2006). Skill meters to indicate knowledge level were amongst the earliest open learner models (Corbett & Anderson, 1995), and remain the most common form of open learner model due to their simplicity, ease of integration in an interface, and relative ease of implementation. At the same time, work was becoming quite advanced on the more complex interactive open learner models, allowing the learner to add evidence to a learner model and receive evidence of the source of learner model data to support their decision making in relation to actions in their model (Kay, 1995); or negotiate the contents of their learner model through a student-system collaborative learner modelling process where both parties aim to reach agreement on the learner model data (Bull & Pain, 1995). A wide range of issues relating to open learner modelling have been investigated more recently in both simple and complex open learner models, as described below. Colour is a common way of indicating knowledge level in both simple and more complex open learner models. Adaptive link annotations as found in many adaptive educational hypermedia environments are a simple form of open learner model using, for example, coloured bullet points in a contents list to indicate a learner’s readiness to attempt a topic (Weber & Brusilovsky, 2001; Weber et al., 2001). Colour has also been used in concept maps and hierarchical knowledge structures to indicate knowledge level (Mabbott & Bull, 2006); and text labels in combination with the size of the text (Uther & Kay, 2003; Kay & Lum, 2003; Li & Kay, 2005). Systems may provide explanations of how the learner model information was used in adaptation (Czarkowski & Kay, 2006; Czarkowski et al., 2005), or where the evidence in the learner model came from (Kay, 1997; Kay & Lum, 2005). Some systems also allow the learner to look back at previous knowledge states or anticipated future states (Bull et al., 1995). As stated above, students can also be involved in the maintenance of their learner model, for example: by providing information explicitly requested by the system (Beck et al., 1997); by adding evidence for consideration (Czarkowski et al., 2005); by editing the model in order to correct the contents (Bull & McEvoy, 2003; Kay, 1997); by negotiating the contents of the learner model through some kind of discussion with the system, where the two partners come to an agreement on the model contents (Bull & Pain, 1995; Dimitrova, 2003); by negotiating the learner model, the result of which is used as a formal assessment of the learner (Brna et al., 1999); by persuading the system through requesting/demanding to demonstrate their viewpoint – e.g. by taking a system-generated test, the result of which may or may not modify the learner model (Mabbott & Bull, 2006); or through dialogue where the system aims to encourage learners to change their learner model if their reasoning does not match that of an expert (Grigoriadou et al., 2003). Interaction about the learner model need not necessarily be directly between the learner and the adaptive tutoring system, but could be mediated by an artificial agent or chatbot (Kerly et al., 2007). It is not only knowledge-related attributes that can be externalised to the learner. For example, learning strategy use and language transfer – the effect of an already-known (native or other) language on the learner’s acquisition of their target language – have been modelled and discussed with the learner (Bull et al., 1995); trust meters (similar in form to skill meters) have been used to indicate other characters’ level of trust in the learner based on their actions, in educational games (Johnson et al., 2005); and virtual pets have reflected social and affective learner attributes in their responses to the learner (Chen et al., 2005). Furthermore, learner models can be opened not only to the learner modelled, but also to other users, as suggested by Hansen and McCalla (2003) and Kay (1997). Learner models open to instructors can be used to allow the instructor to follow the evolution of a student’s knowledge (Rueda et al., 2003); to help instructors adapt their teaching to the individual or the group (Grigoriadou et al., 2001; Zapata-Rivera & Greer, 2001; Yacef, 2005); to help instructors organise learning groups (Mühlenbrock et al., 1998); or to allow the teacher to combine the learner data with other information about the learner, obtained from outside the system (Jean-Daubias & Eyssautier-Bavay, 2005). Learner model information can also be provided to parents, to enable them to follow their child’s progress (Zapata-Rivera et al., 2005). Some systems allow the user to control who has access to their learner model, as in the UMPTEEN approach where users can open their model to peers and/or instructors in named or anonymous form, and view individual peer models to which they have been given access, and the group model (Bull et al., 2007). Learner models can also be externalised to system designers, to help them test the learner modelling process (Paiva et al., 1995). However, while important during design and implementation, in this paper, we focus on the role of the learner model when it is open to learners, peers and teachers within completed systems. Recent technological advances have led to the development of the field of ‘mobile learning’. Open learner models have been developed for handheld computers, allowing the learner to review their knowledge on the move with small-scale learner models that can be transferred between the desktop PC and mobile device (Bull & McEvoy, 2003; Bull & Reid, 2004); and methods of externalising larger scale user models for small-screen devices, that could be adapted to the education context (Kay et al., 2003) as discussed by Uther and Kay (2004). Open learner models have also been deployed in distance learning contexts to lessen the feeling of isolation, and to allow students to compare their progress to others (Dufresne & Hudon, 2002); and to externalise learner models to instructors in distance learning situations (Mazza & Dimitrova, 2003). Figure 1 shows some of the ways that open learner models are presented to users: it illustrates some of the diversity in the appearance of open learner models in addition to the diversity of uses described above. As indicated by the forms shown in Figure 1, interactions with an open learner model, and presentation formats of open learner models, can differ quite widely. Other differences include: the extent to which a learner model is accessible to the learner; who may initiate access to the learner model (the learner and/or the system?); whether the learner has access to information regarding uncertainty in the model; and the flexibility of access to the model. In fact, the seemingly simple matter of making a learner model open turns out to involve a quite complex range of choices, each with significant implications. This has important implications for the designers of new personalised teaching systems with open learner models. It also makes it harder to see the key differences between ways that existing systems have made use of the openness of their learner models. Fig. 1. Some Open Learner Models.1 1 Thanks to the following for their work on the open learner models illustrated in Figure 1: Inderdip Gakhal, Mohammed Ghani, Piyush Kathuria, Lisa Ko, Stella Lee, Luke Lim, Andrew Mabbott, Manveer Mangat, Tom Marianczak, Josie Marsh, Adam McEvoy, Mark McKay, Theson Nghiem and Harpreet Pabla. The current situation is that there has been a blossoming of work on open learner models, where these take many forms and can play many roles. This creates difficulties for the ITS designer who wants to make best use of open learner models, learning from the existing systems and generalising from them. It also makes it difficult to know what should be described about an open learner model and how to compare this aspect of two systems. This paper addresses the need for an open learner modelling framework for use by researchers, for the description and analysis of open learner models, and to support design involving open learner models in new systems. It should also be a useful introduction for those who are new to open learner modelling. The next section presents the SMILI Open Learner Modelling Framework, discussing purposes for opening the learner model, and methods of achieving openness, and the following section gives examples of SMILI in use. The last section concludes with a summary. THE SMILI OPEN LEARNER MODELLING FRAMEWORK Figure 2 shows the four parts of the SMILI framework for Student Models that Invite the Learner In. In this section, we introduce the framework showing how each aspect relates to the purposes that open learner modelling can serve. The full table is quite complex, reflecting the range of purposes of open learner modelling and the means to achieve them. In practice, if one were describing a system or systems, as we do in the next section, one would simply use the elements of the framework as a checklist, indicating which aspects apply in each system described. In this section, we show how the framework elements interact with each of the main purposes we have identified for open learner modelling. The framework’s four parts are: a) b) c) d) Context and evaluation: How does the open learner model fit into the overall interaction and how was it evaluated? WHAT is open? HOW is it presented? WHO controls access? Starting with part (a), the general aspects are: General I – Centrality of Openness of the Learner Model: Openness of the learner model may be more or less central to a system’s aims. This section is used to describe the ways that the learner makes use of the model, and how it fits into their broader learning activities. General II – Evidence or Evaluation of the Open Learner Model: Evidence for the utility of the open learner model includes the type of study undertaken – for example: experimental, labbased, authentic context of a system in normal use, qualitative evaluation, statistical evaluation, anecdotal evidence. Through evaluation it is possible to discover a variety of information about use of an open learner model. Do students actually use the open learner model? How do they use it in experimental conditions and in authentic (deployed) situations? Do they use it as intended, or do they perhaps misuse the open learner model, for example by changing it inappropriately, as suggested by Tanimoto (2005)? Given the diversity of work on open learner modelling to date, it is difficult to identify general ideas or approaches that have worked well, or that have been less successful; or settled research issues. However, as the field continues to grow, if authors can be encouraged to describe their results in sufficient detail, we will hopefully be able to address some of these questions. There are many possible dimensions for evaluation. The first and critical evaluation must assess those aspects related to usability. This means that people must be able to understand what is presented at the learner model interface. To evaluate this, it is necessary to conduct evaluations with the relevant target user population, using standard usability evaluation techniques, such as the think-aloud (Nielsen, 2004) in the laboratory. This can assess whether users can understand the model as presented and whether they can control the interface to the model. In addition, it is valuable to have authentic field trials that determine whether users can still make effective use of the openness within their real learning activities. This constitutes a tougher test of learnability as well as understandability of the interface to the learner model. Many of the systems described in the previous section have reported laboratory trials. There have been fewer reports of field evaluations, although Kay (1995) and Czarkowski and Kay (2006) report both laboratory and field trials and Bull et al. (2006) reports field trials. Another important part of usability evaluations involves affective issues, with questionnaires or interviews to learn about user attitudes to the open learner model. Another critical dimension relates to the effectiveness of the openness for the intended purpose. We discuss those purposes below. However, we now note that they fall into two main categories. The first concerns aspects such as improving the accuracy of and control over the model: these are achieved so long as the user can interact effectively with the learner model. In such cases, evaluating the usability of the learner model is key to assessing the effectiveness of the openness. The other main group of purposes is motivated by a body of theory and empirical evidence about the learning merits of opening the learner model. For these purposes, the evaluation may begin with a usability evaluation and then rely upon the existing literature indicating the value of goals, such as reflection. However, a more rigorous evaluation of the value of the open learner model should assess whether there are better learning outcomes when the open learner model is available. For example, Mitrovic and Martin (2002) reported differential learning outcomes for students who had access to the learner model and those who did not. Moreover, this form of evaluation can assess the relative value of openness for different groups of learners, such as high ability compared with lower ability. A third major form of evaluation should collect information about use of the open learner model. This goes beyond usability since it assesses the ways that learners engaged with the learner model. We will gain different classes of insights into actual use of openness, and the ways that it really fits into learning, from both laboratory and field trials (see, for example, Bull and Mabbott (2006) for a description of how students chose to view their learner model when different learner model views were available, and the kind of information they chose to access – their knowledge level, misconceptions, comparison to instructor expectations and comparison to the group’s knowledge). We now move to parts (b) - (d) of Figure 2, shown in three subfigures. In this section, we show how each element interacts with the purposes or goals of the openness of the learner model: these are the reasons why a system would make the model open. In Figure 2, an ‘X’ indicates the row element is critical for the purpose in that column, ‘=‘ means its importance is arguable and a blank indicates the element does not play a significant role for that purpose. We discuss these in the remainder of this section. First, however, we introduce the purposes for openness. In designing a new system, one should first establish these purposes and this will determine which columns are relevant: for example, if the goal is to support reflection, only that column is relevant. To describe the framework, we begin with a brief overview of the purposes (columns) and the issues in the rows. Then we will briefly review the interactions with these, as summarised in the table. The purposes for opening the learner model are: • • • • • • • Improving accuracy of the learner model by allowing learners to contribute information to their learner model; Promoting learner reflection on knowledge and understanding, an important metacognitive foundation for learning; Helping learners to plan and/or monitor their learning based on the information available in the learner model; Facilitating collaboration and/or competition because partners can improve understanding of themselves and each other by gaining information from their respective learner model(s); Facilitating navigation of the learning system by enabling navigation directly from the learner model, as exemplified by the ELM-ART (Weber & Specht, 1997) display of the learner model as the main course elements coded according to a traffic light metaphor with green for aspects the learner is ready to learn, red where the learner model suggests that the learner lacks the required pre-requisite knowledge; The collection of related issues of addressing the learner’s right of access to data about themselves, supporting control over their learning through greater control over their learner model and increasing trust in a system by making transparent the information used for adaptation, where all of these relate to broader principles of management of personal information (Kobsa, 2002); Using the learner model, or part of it, as an assessment of the learner. We have placed accuracy first because it is of a different order from the other elements. This is perhaps the most obvious purpose for opening the learner model. It can help address the cold start problem where the system initially knows nothing about the user. Also, since learner models are typically built from evidence that may be noisy and uncertain, this aspect of openness may enable system builders to make use of whatever evidence is available, with the possibility that the user can correct problems. StyLE-OLM (Dimitrova et al., 1999; Dimitrova 2003) exemplifies this goal, with an interface to create a more accurate learner model, based upon a combination of dialogue and the display of a graphical form of the model. The next three purposes make use of educational theory and evidence for the potential learning benefits from supporting reflection, monitoring and planning, collaboration or competition. For example, Boud (1985) describes the benefits of learners reflecting on their knowledge state and learning processes. The ways that an open learner model can support this are varied. For example, Schon (1983) distinguishes three classes of reflection: reflection-in-action, reflection-on-action and reflection-on-reflection. He points to the benefits associated with meta-cognition, where increasing understanding of one’s learning effectiveness improves learning by improving the ability to learn. An open learner model has the potential to support all three of these forms of reflection. In Figures 2b) - 2d) rows have the elements constituting the learner model aspects that may be made available. We show how these interact with the major purposes for openness. As stated above, critical elements for a purpose are marked with X, arguable ones with =, and others are blank. Broad Descriptors Centrality and context of the Open Learner Modelling Characterisation of the ways the learner interacts with the open learner model, and the prominence and role of the open learner model within the system Evaluations Overview of the evaluations conducted and evidence collected about the effectiveness of the open learner model Right of access, control, trust = X X X X X X X X X = = X X X X X = = = = = = X X X X X = = = = = = X = = = = = = = X = = X X X X X X X = X = X = X = X = X X X X X X X = = = = = X = = = X = = = = = = = X X X X = = X = = = = X X = = = = X X X = = X = = = = X X = X X X = X X X X X X X X X X = = X Fig. 2. b) SMILI Framework: WHAT is available? X X X X X X X Assessment Navigation Complete Partial Knowledge level Knowledge Difficulties Misconceptions Learning issues Preferences Other Other users' LM 2. Match underlying Similar representation 3. Access to Complete uncertainty Partial 4. Role of time Previous Current Future 5. Access to sources Complete of input Partial System Self Peer Instructor Other 6. Access to model Complete effect on personal’n Partial Collab/ Comp 1. Extent of model accessible Descrip tion Properties Plan/ Monitor Elements Reflection Purpose Accuracy Fig. 2. a) The SMILI Open Learner Modelling Framework Overall: these two elements capture two overall aspects of the open learner modelling. X X X X X X X X X X X X Right of access, control, trust Assessment X X = X X X X = X X = X = X X X X X X X X = X X = = = = X = X X X X X = X = X X X = X X X = X Assessment 9. Flexibility of access X X = X X = X X = X = X Right of access, control, trust 8. Access method Navigation Textual (i.e...) Graphical (i.e...) Overview Targeted/all Details All Details Support to use Inspectable Editable Addition Student persuade System encourage Negotiated Complete Partial Collab/ Comp 7. Presentation Descrip tion Plan/ Monitor Properties Reflection Elements Accuracy Purpose X X = X Navigation 10. Access initiative System comes from User Peer Instructor Other 11. Control over Complete accessibility (to Partial others) System Peer Instructor Other Descrip tion Collab/ Comp Properties Plan/ Monitor Elements Reflection Purpose Accuracy Fig. 2. c) SMILI Framework: HOW is the model presented? X X X X = = X = = = = X = = = = = X = = X X = X = = = = X X X X X X X X X Fig. 2. d) SMILI Framework: WHO controls access? The rows, which identify the means of achieving these goals, indicate: WHAT is available? 1. 2. 3. Extent of model accessible defines how much of the learner model is available to the user. Which attributes can the user access? Do they have access only to the level of knowledge of the various topics or concepts, or can they access representations of specific concepts known? Can they access information about their difficulties and/or misconceptions? Can they access other users’ learner models? How similar is the open learner model to the underlying representation and structure of the learner model? Does the learner model represent uncertainty? If so, does the user have access to this information? 4. 5. 6. If the system maintains models of more than just the current knowledge, etc., does the user have access to historical information or predicted future states? What level of access does the user have to the various sources of input to their learner model? Where multiple users or programs may contribute to the learner model, does the learner have access to information about which data comes from where? Is this general (e.g. from ‘a peer’), or is the specific person (or program) named? Is the learner able to determine the effect their learner model has on the way the interaction is personalised for them? Do they know simply that it is an adaptive system, with adaptations occurring in some way according to their learner model, or do they have access to the details of the adaptivity at a deeper level? One of the commonest forms of openness has been via a small set of skill meters. For example, Corbett and Anderson (1995) incorporated a fourteen line meter showing the learner’s progress on key learning goals. This represented a very small part of the underlying detailed learner model which is cognitively based and quite complex. Similarly, Mitrovic and Martin (2002) present a skill summary of the high level elements of the learner’s demonstrated knowledge of SQL even though the underlying constraint-based model has hundreds of constraints. In both these cases, the learner sees a partial model, covering only knowledge. It does not match closely to the underlying representation used by the system. In both cases, the skill meter shows the degree to which the system models the learner’s mastery and in the case of Mitrovic and Martin, the display shows a summary of achievement, errors and the lack of information in relation to one class of skills. At the other end of the spectrum, Kay (1995) explicitly designed the learner model so that it could be made available to the learner, and so the full model is available. This includes misconceptions as well as correct knowledge. The interface presents the model in a form that is very close to the underlying representation and the learner can scrutinise this to identify the underlying reasoning and uncertainty, including details of all sources of input to the learner model. There are several knowledge representation and reasoning approaches that seem amenable to display of the learner model in a form that matches the underlying representation: Dimitrova (1999; 2003) demonstrated this with conceptual graphs and Zapata-Rivera et al. (2001) with Bayesian nets. Making the effect of the learner model available to the learner adds an additional layer of difficulty to the interface design: the learner needs to be able to understand both the learner model and the way that it drives the personalisation. A very simple form of this is generally possible at some level because the learner who can alter their learner model can use this as a way to see the effects. Indeed, just this was reported in even very early work (Burton & Brown, 1979): learners experimented with the system to see the effect of their actions. Unfortunately, when learners purposely perform actions to see their effect on the personalisation, this has the side effect of corrupting the learner model. Czarkowski et al. (2005; 2006) explicitly attacked this problem by making the personalisation explicit at the interface, with links to the learner model. HOW is the model presented? 7. How is the learner model presented? At what level of detail can the learner model be accessed? Is there an overview, which can serve as a starting point for identifying details to explore? Can the learner access all details, or does the system present a targeted set of just a selection of appropriate ones? Is there any system support to understand or use the open learner model, for example, explanations of what the various components of the open learner 8. 9. model mean, or an artificial guiding agent to help learners understand or interact with the contents of their learner model? How is the learner model accessed? Is it available for viewing only, or can the user interact with it in some way? Can they directly change or edit the learner model? Can they request to prove their level of understanding, etc., in order to indirectly influence the model, or can they provide additional evidence? Does the system encourage them to change the learner model, or can they discuss or negotiate with the system about the learner model, hoping that the system will accept their viewpoint as a result of this interaction? How flexible is access to the learner model? Can it be viewed in different formats? Can users choose the level of detail to view? Can the model be accessed and/or interacted with, by different methods? These aspects deal with the nature of the interface to the learner model. We have indicated some of the scope of learner model interfaces in a previous section, with the overview of some of the many forms that they take as well as screen shots of some examples in Figure 1. Notably, these aspects interact with the nature of the model. For example, if the model is small and simple, it can be fully displayed without taking much screen real estate. However, if the model is large or complex, this is impossible. Then, if the whole model is to be made available, it may be necessary to provide some form of overview plus a facility to focus on selected parts so that the learner can more readily navigate it. This was the focus of Uther and Kay (2003, 2004) for learner models with several hundred components. WHO controls access? 10. Who initiates access to the learner model? Is it the system or the learner? Does the instructor decide at which point a learner’s learner model will be open to them? (Note – a full description should distinguish learner access from teacher access.) Can a learner decide, for example, the level a peer must have achieved before they can have access to their learner model? 11. If the learner model is available to other users or programs, how much control does the user have over the release of their model to these others? These aspects have had relatively little exploration in existing open learner modelling. In most systems, the learner model has been designed for use by the learner. Indeed, the skill meters, being compact, are readily incorporated into the interface of a learning system so that they are continuously present. There has also been some progress in exploring other ways to make the learner model available for people other than the learner. For example, Collins et al. (1997) made the learner model available to workplace colleagues so that they could seek help from the most suitable person. Linton and Schaefer (2000) used the comparison between an individual’s and the group model available to motivate learning. Bull et al. (2007) allow students to optionally release their learner model to peers in order to promote collaboration amongst learners. More recently, there have been explorations of open learner models explicitly designed for use by teachers (Mazza & Dimitrova, 2003; Yacef, 2005) and for parents and teachers (Zapata-Rivera et al., 2005; Bull & McKay, 2004). In some systems, the same elements of openness will apply in a system for more than one purpose. For example, a system may be designed for learners to reflect on their knowledge and use this as a starting point for planning. Then the two columns, reflection and planning, may look very similar. It may also be the case that in some systems, elements that we have defined as important (or not important) in general for a particular purpose of opening the learner model, will differ. Thus the pattern of X and = in a column may vary. We now review each of the purposes of opening the learner model, discussing how row elements relate to the purposes in each column and give our reasons for marking certain cells as critical, or arguably so. The first section is rather longer than the rest since it helps clarify the framework elements and it includes some discussion of our design of SMILI, the choice of elements to include and distinguish. Accuracy of the Model: The accuracy of the learner model is the first purpose considered, because it is key in adaptive learning environments. By ‘accuracy’ we mean the extent to which the model represents the learner’s own cognitive model or knowledge level, or its validity. In interactive open learner models (i.e. open learner models that allow learners to directly influence the model contents through, for example, editing them or discussing the model with the system), accuracy of the learner model will usually be an important consideration. We suggest this means that any designer of an interactive open learner model should use this column as part of their design process. However, in learner models that are open for inspection only, the accuracy of the learner model will not usually be related to the openness of the model. The extent (Row 1) of openness has three aspects. The first relates to the completeness of accessibility. It is obvious that at least partial access is critical for accuracy: if a learner is to help improve model accuracy, they need to be able to find out what is there. It may be surprising that we argue that complete access is not critical. Particularly if the model is small or easy to understand, complete access may improve accuracy. However, for large or complex models, complete access may be so overwhelming that the learner could not easily correct problems. In this case, partial access to the learner model data may be more effective. The next aspect of extent of openness deals with the accessibility of various forms of knowledge, lack of knowledge and misconceptions. We show all of these are important for accuracy because some form of access is critical for awareness as a foundation for improving correctness. We have distinguished some of the forms of knowledge because they play different roles for open learner models. Knowledge level is generally a small set of information: it is amenable to display with, for example, simple skill meters that can be part of the basic learner interface. By contrast, access to the knowledge modelling means either that the model is very simple, so that it is easy for learners to deal with all of it, or it provides an interface to a larger amount of information. We explicitly show difficulties because these are important. Of course, difficulties might also be modelled simply as lack of knowledge. However, we believe that the designer of an open learner model should think carefully to ensure that learners can readily find this information, if it is appropriate to the particular learning context. This may range from an indication of the broad areas of difficulty, to specific problems. Information about misconceptions has also been separated for similar reasons. The design of interaction to improve the accuracy of models of misconceptions poses special challenges because a learner may not realise they have a misconception – it is in the nature of misconceptions that the holder does not realise that they are misconceptions. Fortunately, learners may be able to state that they do believe an incorrect ‘fact’ shown in their learner model, or that they do not believe it. Of course, widely used overlay models have only ‘known’ concepts, or an estimate of the knowledge level. Increasing the accuracy of such a model will probably be focussed around knowledge only. Notably, a learner can often provide information about aspects they do not know. The last aspect of extent of openness deals with all modelled aspects other than knowledge. We have distinguished learning issues such as the learner’s goals and aspirations. These can play an important role in a teaching system and, therefore, deserve careful consideration in design or description of a system. We show preferences because so many systems make use of these, ranging from learning preferences to presentation preferences. SMILI then has a category for all other aspects of the learner’s personal model, such as social or emotional issues for systems that model such attributes. We include other users’ learner models as a separate category because these can be important in collaborative and other group learning. Figure 2 shows that all these aspects of the learner should be available to support improvement of the model accuracy. Of course, even a partially open model may present several types of information. We show that similarity to the underlying representation (Row 2) is not necessarily important. This may seem surprising. For simple models, a similar presentation may be better for accuracy. However, the core issue is that the presentation be understandable: for complex learner models, a different presentation may be clearer than one that is more similar to the underlying representation. In a broad sense, access to uncertainty in the learner model (Row 3) is important for accuracy. For example, suppose that a learner is aware they do not fully understand a particular concept. If the model shows that they do understand it, but they see an indication of the system’s uncertainty, they can make more sense of what the model means. We therefore show complete access as possibly important and partial access as critical: this is for the same reasons discussed above, that this interacts with the complexity of the model. Current data is most important with reference to the role of time (Row 4) for accuracy of the learner model. The relevance of previous model information is arguable, especially as a user’s memories may be unreliable. A learner may provide invaluable input to predictions of future knowledge states but we show its accuracy as debatable. As we see increasing use of large-scale classwide historic data for predicting user performance, this promises to be statistically more accurate than the learner’s predictions. Row 5 deals with access to the sources of evidence for the learner model. These are the foundation for the system’s conclusions about the user, as described in Row 1. Essentially, this aspect enables the learner to determine why the system holds the beliefs it does. This is also a foundation for control, with the learner potentially defining whether a particular evidence source is allowed to contribute to their model. Awareness of the effect of the model on personalisation (Row 6) has a subtle, but important role in ensuring accuracy. This is because the meaning of the model might be best defined by the way it is interpreted by an application system. For example, suppose the learner is modelled as preferring easy examples. Since there is no absolute meaning of easy, the learner may only see the real meaning of this part of the model by its effect. Only then can they really judge its accuracy. We mark this as an arguable need for accuracy in Figure 2. We also note that such awareness may, potentially, raise motivation for a learner to strive to reach an exact model. We now consider the role of model presentation (Row 7) for accuracy. SMILI distinguishes two forms, textual and graphical, both widely used, but with a strong trend towards the latter. As long as it is understandable and useable, the format is not usually important for increasing model accuracy. Figure 2 shows that an overview and targeted details (as required/requested by the learner or inferred as important by the system) are crucial for accuracy because these provide both orientation and then the opportunity to focus on selected parts of the learner model. It is arguable whether complete details are helpful for accuracy, especially where the model is complex or large. The final aspect of presentation is support for using or interpreting the open learner model. We mark this as critical, although for very simply and clearly presented models, the simplicity may itself constitute the support. More typically, especially for more complex models, additional assistance may be essential for understanding the model and contributing to supporting maintenance of its accuracy. With regard to the access method, the learner must have some kind of interactive access (Row 8) to the model in order to improve its accuracy. The fundamental and critical level allows the learner to inspect the model (though this alone will not increase model accuracy). It is arguable whether accuracy is enhanced where the learner can edit the model: certainly learners can purposely make the model inaccurate; or they may not have adequate knowledge to edit their learner model appropriately. However, some learners may be able to provide useful information. For non-knowledge aspects, such as preferences, there is a stronger case for editing improving accuracy. Addition refers to the learner adding evidence to be considered alongside the system’s own inferences – i.e. not changing or correcting as in an editable model, but provision of further evidence by the learner. Similar to editing, addition may or may not lead to greater model accuracy (though addition will usually result in a lower level of changes in the model, as other evidence will also be taken into account). SMILI marks support for the learner to persuade the system as a critical mechanism for accuracy. A simple and natural way to do this is where the learner can request a test to persuade the system of their knowledge, or negotiation where both system and user can argue their case, with the aim of coming to an agreement over the model contents. This requires the maintenance of different viewpoints on knowledge – the system’s and the student’s own beliefs about their knowledge – in order to facilitate discussion and negotiation. (Multiple viewpoints also allow greater control over how evidence is combined, and what changes have been made in the model.) SMILI distinguishes another category of access: encouraging, which is where the system might try to entice the learner to take action in their model. We also consider the flexibility of access to the learner model information (Row 9) – i.e. are there choices about how to access the model? This, too, is subtly relevant to accuracy. One of the foundations of personalised teaching is that different learners respond best to different learning experiences. So, too, different people may perform better and prefer different ways to interact with their learner model. Therefore, we argue that this aspect may be important for accuracy, to ensure the learner can access precisely those, and possibly only those, parts of the learner model that are relevant to the current concerns. We show it as possibly important for the complete model, but critical for partial access. Row 10 of SMILI indicates the access initiative supported. For accuracy, it is critical that any party who recognises a potential problem with the learner model can initiate action to give the learner access to the model. For example, it is important that the learner can choose to interact with their model – openness motivated by the goal of accuracy is based on the expectation that the model could be wrong (or at least, incomplete), and that the student may be able to help. Of course, a combination of these approaches constitutes mixed-initiative access to the model. Control over who (or what) else can access the information in a student’s learner model (Row 11) will not necessarily improve the model accuracy. For example, if a learner withheld model information from an instructor, they may not be able to play their role as instructor effectively and hence they may corrupt the model inadvertently. This more detailed discussion of the elements that are important for accuracy of the model is intended to introduce the elements that characterise openness, and support the various motivations for opening learner models. For the remainder of this section, we will focus on the less obvious parts of the framework. Reflection: We now discuss the role of open learner models in supporting reflection. In essence, the most critical elements for supporting reflection are probably that the learner be able to inspect (Row 8) the system’s view (5) of their knowledge, difficulties and misconceptions (1) at the current time (4). Overall, reflection has fewer critical elements than accuracy. Arguably, reflection might be supported by other model contents, such as social issues modelled in a collaborative learning environment. For presentation, it is critical that the model be readily understood, although it is arguable whether this needs to be similar to the underlying representation (2). It may not be necessary to have access to uncertainty in the learner model (3). If the learner notices disparities, this may lead them to reflect further! However, there is a question of trust in the value of the model if learners perceive it to be withholding information. The role of time (4) for reflection on knowledge would normally focus on the current model, as this gives the learner the clearest indication of where they are. However, there is potential value in reference forwards and backwards in time, for example to highlight progress and achievements over time; or to raise learner awareness of the next stage of the learning process. To reflect on their learner model, the learner must be able to see the contents (8) – so an inspectable model is crucial. Other access methods are not critical, but it may be argued that interaction such as persuading or negotiating the model may facilitate reflection by focussing the learner’s attention. The learner must be able to initiate the access (10) and perhaps others might well draw the learner’s attention to the model when an event, such as the learner making a mistake, creates a good opportunity for reflection. The argument for the importance of access to sources of input (5) is based on the fact that the learner may benefit from reflection both on the state of their model as well as the evidence that causes the system to hold its beliefs. For example, suppose the system has evidence of many incorrect attempts at a class of problem and this is the basis for its belief that the learner has difficulties. The learner may need to see this evidence to appreciate why they have a problem. Similarly, it can be argued that access to the effect of the model on personalisation (6) may be important for reflection. For example, if parts of the model cause the system to present only basic information, realising this might cause the learner to reflect on the reason they are missing out on interesting additional information. As in the case of accuracy, flexibility of access (9) may be important in enabling learners to access their learner model in a manner most useful to them; this may be critical to supporting reflection on knowledge. Planning/Monitoring Learning: This goal is very similar to that of promoting reflection, in that it involves the learner in assessing how they are doing, and how to use this information to decide on future learning goals. Accordingly, this discussion deals with the differences between this column and that for reflection. One difference is the likely greater importance of at least partial access to uncertainty (3). This information supports decisions about the reliability of information in the model, which can be used to help plan and monitor progress. It is likely that partial access will be sufficient, as it is only knowledge of the existence of uncertainty, or an indication of the extent of uncertainty, that the learner really requires for this purpose. To monitor or plan learning, learners need to access the current model. This may be inspectable only. Users must be able to initiate inspection (10) to plan and monitor effectively, as autonomous learners. Most important is likely to be access to system inferences, as the system will have been designed to facilitate learning in an effective manner. Similarly, the role of time (4) is different than for reflection. Monitoring involves tracking past performance and progress and this makes the historic model more important. Planning could benefit from projections into the future, perhaps exploring scenarios of the potential effect of different plans. It is important that the learner is aware of the effect of their learner model on the interaction (6), otherwise they will not be able to use their model data effectively in planning, where this planning relates to further interaction with the system (as opposed to planning learning episodes to occur outside the system). Collaboration/Competition: In this paper we focus primarily on individual learning, while noting that open learner models can also be effective in supporting collaboration or competition by opening (parts of) the learner model to other users. Thus a main difference between this column and the previous ones is the increased attention on peers, with lesser consideration of other issues. Supporting Navigation: Navigation to/through information is key in adaptive educational hypermedia. An open learner model can support easy navigation in such systems. This is related to planning, as the learner may identify their needs and decide what to do next, navigating directly from the open learner model. However, planning is not necessarily restricted to activities within the system, but is a metacognitive activity. Navigation need not be a metacognitive activity. The similarities (and differences) can be seen in the correspondence between this column in Figure 2 and that for planning. Right of access, control and trust: This aspect has three related goals. We have combined them because they are different from our core concerns about open learner modelling for educational goals. Notably, this set of goals has a distinctive profile in Figure 2, as it is quite different from the goal of reflection and the associated monitoring and planning. Assessment: The purpose of assessment refers to the learner model being used in some way to assess the user. It is usual to distinguish two classes of assessment: formative and summative. Formative assessment is largely covered by the discussion of reflection, monitoring and planning. This column is reserved for assessment that is, at least partly, summative: for example, where it contributes to a student’s grade. Current (4) knowledge (1) will usually be the main attribute relevant for assessment purposes as well as uncertainty of model data (3). Inspection (8) is necessary and the model might be transformed into a more traditional assessment format – relevant to the presentation of learner model contents (7). Interactive methods of accessing the learner model (8) might also be used, in particular, persuasion and negotiation, as these allow the user to demonstrate their understanding if they disagree with the system’s assessment. We exclude direct editing of the learner model since that would allow the learner to play the system for an undeserved high mark. To a lesser extent, addition of evidence by the learner could have a similar effect. Instructors will need to be able to initiate model access (10). Peer assessment may also be supported. EXAMPLES OF USE OF THE SMILI OPEN LEARNER MODELLING FRAMEWORK The various aspects of open learner models are not usually described in sufficient detail to analyse them in the SMILI framework. For example, the extent to which uncertainty is explicitly noted in the system may not be reported, so it may be difficult to judge whether the student has complete or partial access to this information. Similarly, the underlying learner model representation may not be fully reported, preventing comparison to the learner model information presented to the learner. Some descriptions do not make it clear whether there is some learner model information that is not available in the open learner model. This is related to the extent of the model accessible and the question of whether the presentation contains an overview only, or all or targeted details. Editing the model and additions to the model need to be clearly distinguished from each other, as do persuading and negotiating the model contents. Often the extent to which the learner is aware of the effect of the learner model on system adaptivity is not reported. i. Centrality of the open learner model Mr Collins: Interaction is focussed around the open learner model. sam-coach: Coach mails advice, and invites scrutiny of learner model. Subtraction Master: Users can interact with minimal or with extensive inspection of the learner model. SASY: Users can scrutinise the personalisation and the learner model. XX XX X XX X Subtraction Master - Child X XX XX XX XX XX X X x Subtraction Master - Teacher SASY-unix X X X XX Assessment Plan/ Monitor XX sam-coach Right of access, control, trust Reflection Mr Collins Navigation Accuracy Purpose Collab/ Comp ii. Evidence Mr Collins: lab-based study with 9 users; a case study; interaction logs; questionnaires; structured / evaluation interviews. sam-coach: lab-based study of accuracy of learner model with 7 users; field study over 8 weeks with 352 users modelled, 82 in the user model viewer experimental group. Subtraction Master: lab-based study with a small number of users (11 children and 2 teachers); interaction logs; video recordings; teacher questionnaires; interviews with children and teachers. SASY: series of small qualitative lab evaluations mainly with 3-5 users; 82 users in field trial. x Fig. 3. Mr Collins, sam-coach, Subtraction Master and SASY in the SMILI Open Learner Modelling Framework. The upper part shows the general issues. The lower part shows the goals of openness of the learner model: XX for central goals; X for lesser goals and x for minor concerns. We therefore use four of our own systems to illustrate use of SMILI: Mr Collins (Bull et al., 1995; Bull & Pain, 1995); sam-coach (Benyon et al., 1992; Cook & Kay 1994), Subtraction Master (Bull & McKay, 2004) and SASY (Czarkowski et al., 2005; Czarkowski & Kay, 2006). The first pair (Mr Collins and sam-coach) represents two of the earliest detailed open learner models, chosen here because they have previously been described, but not as fully as can be achieved with the SMILI framework. Subtraction Master and SASY are more recent, giving a current perspective. Figure 3 summarises the centrality of openness of the learner model in each of the systems, the ways in which they were evaluated (the first two elements of Figure 2) and the primary goals (columns of Figure 2). Mr Collins Mr Collins – COLLaboratively maintained, INSpectable learner model (Bull et al., 1995) is a system with an open learner model for foreign language learning, aimed at university level students. Students place the object pronoun in the correct position, and give translations of sentences, in 12 clause types in European Portuguese. On inputting their answer, the learner states how sure they are about its correctness. These self-assessments form the student’s belief measures in the learner model (a-d, with ‘a’ being high). A parallel set of system confidence measures is based on the correctness (or problems in) the answers given (1-4, with ‘1’ being high). The primary aim of Mr Collins is to improve the accuracy of learner modelling using a negotiated model, while also promoting learner reflection through this negotiation process. Students can also look forwards and backwards in time – i.e. inspect but not negotiate previous and predicted future states of their learner model to facilitate reflection, and inspect information about language transfer and learning strategies used. While other purposes are relevant (e.g. learner trust in the adaptive decisions of the system; formative assessment), these are not central purposes for opening the model. Because the interaction is designed around the open learner model, the openness of the model is vital to the success of the system. A lab-based study with 9 users found that students understood their learner model, and would argue with the system if they disagreed with the contents of their model (Bull & Pain, 1995). Figure 4 shows an excerpt illustrating the learner model negotiation process (illustrated using two clause types), where the learner first changes their belief about their knowledge of a language rule, and then challenges the system’s confidence in their knowledge of the rule. It should be noted that, although the system disagrees with the learner’s action of changing their belief in the learner model, and it is able to justify its representation, because it is the student’s belief the system cannot override it if the learner insists on the change. However, when the learner challenges the system about its belief about their knowledge, the system requires them to prove their point of view, amending its belief only if evidence is shown. Thus Mr Collins maintains two sets of belief measures which is important for negotiation, both being taken into account during an interaction. In the example given, the negotiation was initiated by the student. In cases where the two belief measures for a language rule are incompatible (i.e. not identical or close – 1 and c/d; 2 and d; 3 and a; 4 and a/b), the system initiates discussion. student (from menu): system: View student model. YOUR CONFIDENCE (a - d) almost sure (b) almost sure (b) THE PRONOUN IS: Pre-verbal in negatives (e.g. Não os compra) Post-verbal in positive main clauses (e.g. Compra-os) student (from menu): SYSTEM CONFIDENCE (1 - 4) very sure (1) unsure (3) Change own belief (positive main clause statements). system: Currently your confidence in your use of the rule applicable in positive main clause statements is: almost sure / b. (System’s confidence is: unsure / 3) What is your new confidence level? • very sure (a) student (selects): • unsure (c) • very unsure (d) • (cancel) very sure (a). system: You have changed your confidence measure from ‘b’ to ‘a’. This new confidence value is a lot higher than that of the system. The two confidence measures are incompatible. Your last five attempts to place the pronoun in positive main clause statements demonstrate that you are still having difficulties. Confidence measure ‘a’ is too high for your actual proficiency. What do you wish to do? • system justify itself student (selects): • change belief to match system • view student model • keep new belief keep new belief. student (from menu): Change system’s belief (positive main clause statements). system: The system’s confidence in your use of the rule for positive main clauses is currently: unsure / 3. What do you believe the system’s confidence should be? • very sure (1) student (selects): system: • almost sure (2) • very unsure (4) • (cancel) very sure (1). Your choice of ‘1’ is different from the system’s assessment. What do you wish to do? • accept system’s decision student (selects): • system justify itself • view student model • accept compromise (2) • justify myself system justify itself. system: You are trying to change the system’s confidence measure from ‘3’ to ‘1’. This new confidence value is too high. Your last five attempts to place the pronoun in positive main clause statements were the following: *O Manuel a mostrou no mapa. *O Manuel mostrou a no mapa. O Manuel mostrou-a no mapa. *O homem o comeu rapidamente. O homem comeu-o rapidamente. These recent attempts demonstrate that you are having difficulties. You have probable transfer from Spanish for pronoun placement twice. You have omitted the hyphen once. What do you wish to do? • accept system’s decision student (selects): • view student model • accept compromise (2) • justify myself justify myself. system: Here is a test. If you give the right answer, the system’s confidence will be changed to ‘1’. Please place the pronoun ‘a’ into the correct position in the following sentence: • O João confirmou (João confirmed / it) Fig. 4. Negotiating the Learner Model in Mr Collins (changing own and challenging system beliefs). um-based sam-coach The sam-coach was based upon a learner model constructed primarily by monitoring (Benyon et al., 1992). Primary goals were reflection (Crawford & Kay, 1992), monitoring and planning, and to give learners the power to scrutinise and control their model, enhancing its accuracy. The model used the accretion-resolution representation (Kay, 1995), for scrutable long-term modelling: the user can scrutinise their model to see what the system believes about them, the evidence for those beliefs, the meaning of the model components and, if they wish, to use the interface to tell the system their own assessment of their knowledge, preferences or other aspects, and this is added to the model without removing any of the existing evidence. Learners could also use the learner model as a starting point for navigation within the environment. Learners worked with the sam text editor as part of their university computing course work. The sam-coach sent weekly advice on how to make better use of the text editor. For example, a user who had not learnt the undo command might be emailed advice about it. For students in the experimental group, the email included details of how to view and interact with their learner model. Openness of the learner model was central to this work. Indeed, the learner model was built and evaluated qualitatively independently of its use in the sam-coach. (Parts of it were also used for a unix tutor.) At the time of the sam-coach evaluation, learner models were built over eighteen months, from monitoring editor use. During the eight weeks of the experiment, additional evidence came from the coach and student interaction with the model. The evaluation indicated some learners carefully scrutinised their models and some learnt about sam from this, because part of the model interface provided customised explanations of the components of the model: for example, there was an explanation of the undo command and the value of learning it. There were several parts to the interface to the learner model (Cook & Kay, 1994). Figure 5 shows qv (quick overview). Black nodes indicate the learner knows this aspect, as in the case of ‘quit_k’, the quit command, near the top of the figure. White node symbols indicate lack of knowledge. The default presentation collapsed parts of the model tree as in Figure 5, where the node labelled ‘powerful’ is shown as unknown and represents a collection of concepts. Left-clicking on such a node expands it. Right-clicking on it gives access to full details of the model and allows the learner to indicate the value of a component and this caused evidence to be added to the model. In addition, as the model was stored as a set of text files within the user’s file space, the user could edit it. It also provides a personalised explanation of the concept. The scrutiny support presented the full set of evidence for a component and an explanation of how this was interpreted. Fig. 5. Example of an overview screen of a sam-coach learner model. Subtraction Master Subtraction Master is a system to support the learning of subtraction by 8-9 year old children by providing tutorials, adaptive help and multiple choice questioning. The main aim of inspecting the learner model is to encourage children to think about their learning, thus reflection is the main relevant column for the learner’s use of the system. Subtraction Master also has a version of the open learner model for teachers, for two purposes – first to allow teachers to edit the learner models of children (for example, according to the results of a test taken outside the system, or after individual coaching from the teacher), to update the learner model and therefore increasing its accuracy; and second, to monitor the progress of individuals and the group. The centrality of the openness of the learner model to the interaction will vary depending on the user. Teachers can use it extensively if they wish, or they may use it only if seeking clarification of a learner’s understanding while helping an individual at the computer. Children vary in their level of use, from minimal (the one time that the system automatically shows the model), to only when the system suggests it (when the student is having problems), to extensively (watching the model update after each answer). A lab-based study with 11 users found that the children could understand their learner model, and were also able to understand a representation showing their progress in comparison to that of their peers (Bull & McKay, 2004). Figure 6 shows the child’s view of their learner model in two forms: (1) smiling faces to represent their level of knowledge; and (2) smiling faces to contrast their knowledge with the knowledge of their peers. The teacher’s view shows in addition, the misconceptions held – shaded dark, and the misconceptions that could have been demonstrated given the questions answered, but where no misconception was evident – shaded light. Strength of evidence for the learner model representations is shown in numerical form, and an individual’s knowledge can be displayed against the average knowledge of the group. The child’s view is necessarily simple (for example, the children have not yet learnt how to interpret graphs, so these cannot be used); the teacher’s view provides additional information which may help the teacher to identify a child’s difficulties. The Child’s View: their own learner model and comparison to the group The Teacher’s View: individual misconceptions and comparison of the individual against the group Fig. 6. Subtraction Master open learner model for children and teachers. SASY-unix The driving motivation for SASY was to enable users to see how an adaptive hypertext was personalised to them. The goals for this were both to encourage reflection and to provide a foundation for user access to the whole of the personalisation process: this includes access to the user model (or learner model) but also enables the user to see what has been presented to them, or omitted, in the personalisation. SASY provides a generic framework that has evolved through a series of versions, starting with Tutor (Czarkowski & Kay, 2000) which was used to build a tutor about parsing, to the current SASY, which was used to create multiple hypertexts, including a tutorial about unix file permissions. We will focus on this and call it SASY-unix. An example screen from this is shown in Figure 7. This tutorial was designed for university computing students but adapts to the different needs of different courses as well as preferences such as the number of quiz questions. Fig.7. A SASY-unix page teaching about unix file permissions, with open learner model information. Openness is central to SASY. Its predecessor, Tutor (Czarkowski & Kay, 2000), was similar to a conventional adaptive hypertext, with the difference that there was a single link at the bottom of each page: this link invited the user to explore how the page was adapted. After a large scale field trial and a series of small-scale qualitative evaluations, we concluded that people could not work out how to scrutinise using this link. This led to a shift to making the details of personalisation much more obvious (Czarkowski et al., 2005): a small-scale qualitative evaluation showed this to be successful and led to the current version of SASY as shown in Figure 7 Some important features of a SASY page are the list of the components from the user model at the right of the screen. The list includes just those components that played a role in the personalisation. Each of these has a link labelled ‘why?’ and this takes the user to details of the user model. Note also, that at the top of each page is a link labelled ‘Your profile’. This takes the user to their user model. The model itself shows the components of the model, explanations of what they mean and details of how they are interpreted and used. The user can edit their user model at this page to increase the accuracy of the model. Users can also navigate to relevant topics or questions from the main part of the screen. Using the SMILI Open Learner Modelling Framework to compare systems Figure 3 has already summarised the centrality of the openness, the evaluations and the central goals of our open learner models. Figure 8 now provides a more detailed comparison of these four systems. For each element that is relevant to any of the systems, there is a set of five symbols. They refer to the systems in the following order (and the comparison column uses these abbreviations): • • • • • Mr Collins (MrC); Sam learner model as used in conjunction with the sam-coach (Sam); Subtraction Master Child’s view (SM-C); Subtraction Master Teacher’s view (SM-T); SASY-Unix as used for the unix tutor (SU). For each, the symbol O refers to elements that are open/present in that system, ^ for those that are not. In Rows 1, 3 and 4, there are two reasons that an aspect may not be open to the user. One is that the aspect is part of the learner model, but it is not open to the user (shown as ^). The other reason is that the learner model representation does not have this aspect and so there is no meaning to opening it; this is indicated with the - symbol. For example, SASY-unix does not model misconceptions, so clearly it does not make any available either. Since we are using the framework to compare these systems, we have omitted all rows that do not apply to any of them. (If we were using the framework to describe a single system, we would include all the elements shown in Table 2 so that omissions would be clear.) So, for example, the first description cell indicates that all but the child’s view of the Subtraction Master open learner model give access to the complete model. Of course, each of these systems was created before the SMILI framework existed. It is interesting that even in the first elements (Row 1), completeness of the extent of accessibility, all but the child’s view of Subtraction Master give complete access to the full model. This was to be expected for sam-coach and SASY-unix tutor, given their goals of user control. As we saw in Figure 2, full access to the model is critical for that goal, which is not the case for the other goals of these systems. It is important that Subtraction Master gives the teacher complete access, a reflection of their complete control, while it limits children to partial access. It omits negative information to children for two reasons (i) to avoid discouraging children and (ii) to reduce complexity. Interestingly, sam-coach and SASY-unix tutor do not provide access to the knowledge level of the learner. This is partly because this is a weaker requirement than access to the full set of knowledge. However, there is real benefit in the summary power of the learner’s knowledge level. In future, in light of this framework, we would certainly consider incorporating such a summary in any system with a model of sufficient size and complexity. In Row 2, only those systems centrally concerned with right of access and control, sam-coach and SASY-unix, provide an open model that is similar to the underlying representation. This is no coincidence since that representation was designed explicitly to support scrutability of an open learner model. In Row 3, all but the child’s view of Subtraction Master have complete access to uncertainty. For Mr Collins, the uncertainty in the form of parallel belief or confidence measures, the system’s and the learner’s, ensures learners understand the reliability of the model. Similarly, it is important that teachers in Subtraction Master appreciate the reliability of the model, and so there is a numerical representation of this. Both sam-coach and SASY-unix allow the user to see the full set of evidence that informs the model, as does Mr Collins during negotiation, if the issue arises. The role of time, in Row 4, has the current model available in all the systems. Mr Collins encourages learners to reflect on their learning history, to gain a better understanding of the learning process, and to look forward to anticipate future states to help them plan their learning. In sam-coach and SASY-unix, the access to all available evidence gives some historical information. Access to the sources of input to the model (Row 5) is essential, for the access and control are goals of sam-coach and SASY-unix. Mr Collins, too, gives complete access to its sources of input, the learner’s confidence in their answers and the system inferences: these are essential in the negotiation. Row 6, giving access to the effect of the learner model on the personalisation, is the core goal of SASY-unix. In Mr Collins that was considered important to motivate learners to ensure the model’s accuracy. The teacher using Subtraction Master had a sense of the role of the learner model in the operation of the system. Notably, the sam-coach model was independent of any single application and so the interface directly to the learner model could not present any information about its use. In Row 7, summarising presentation issues, it is to be noted that Mr Collins and the teacher view of Subtraction Master were designed to support the user without additional information. By contrast, sam-coach and SASY-unix provide explanations of all aspects, including the meanings of the elements modelled; the child’s Subtraction Master has short explanations of how to use the learner model to correct it, and to trigger reflection, and the teacher may also prompt children to use their open learner model, giving explanations as appropriate. Row 8 reflects the underlying differences in the goals of the systems. With access and control central to sam-coach and SASY-unix, they allow the learner to contribute to the model. Note, however, that this simply adds more evidence to the model: it does not delete or alter existing evidence. Moreover, the interpretation of the evidence is associated with the consumer program and so, for example, the sam-coach only accepts user claims of knowledge if these match other evidence. The teacher can edit Subtraction Master models to include knowledge outside the system, such as results of paper-based tests or classroom interactions. The child cannot edit their model but can ask for a test to persuade the system of their knowledge. With negotiation central to Mr Collins, differences are resolved by a variety of methods as applicable to the current situation, for example: a test; a statement from the user that they have forgotten what they previously knew; both parties accepting a compromise if their views are not too disparate; the user stating that they were relying on the rules of their native or another foreign language known to them. Purpose Elements 1. Extent of model accessible Properties Description: MrC, Sam, SM-C, SM-T, SU Comparison Notes Complete Partial Knowledge level Knowledge Difficulties Misconceptions Learning issues Preferences OO^OO OOOOO O^OO^ OO^OO OO^OOO^OO---O O---O Other users’ LM ^^OO^ All but SM-C allow full access. SM-C omits negative information, reduces detail. SM-C shows only this; not shown in Sam, SU. All but SM-C have this as core to their purpose. Not shown in SM-C, not modelled in SU. Not shown in SM-C, not modelled in SU. MrC models language transfer, SU models learning goals. MrC models learning strategies, SU preferences for jokes, quizzes, hints. In SM-T teachers access child models, SM-C gives individual model compared with peers. Sam and SU match underlying representation – design goal of the representation is to support control; SM-T strength of evidence is similar, but not model contents. SM-C has no access to uncertainty as the children are considered too young to fully comprehend this. Only SM-CT omits historical view. All have current model. Only MrC has future model. SM-CT does not keep details of data. MrC to all inputs, sam and SU to all evidence. Sam model is independent of coach, SM-C avoids distracting the child with this. (See previous Figures for details.) 2. Match underlying Similar representation ^O^OO 3. Access to uncertainty 4. Role of time OO^OO Complete Previous Current Future 5. Access to sources Complete of input System, Self 6. Access to model Complete/Partial effect on personal’n 7. Presentation Textual Graphical Overview Targeted Details OO^^O OOOOO O---OO^^O OO^^O O^^OO All Details Support to use Inspectable Editable Addition Student persuade Negotiated Complete Partial OO^OO ^ O O ^O OOOOO ^O^OO ^O^O^ ^^O^^ O^^^^ OO^^O OOOOO 8. Access method 9. Flexibility of access 10. Access initiative System comes from User Peer, Instructor, … 11. Control over Complete accessibility System (to others) Peer, Other, Instructor OO^^O ^OOO^ OOOO^ OOO^O O^O^^ OOOOO ^^^O^ ^O^^^ ^O^^^ ^O^^^ ^O^^^ SU presents full model, no overview. MrC targets details during negotiation, sam lets learner scrutinise details, SU has details of current page, SM-T has some flexibility on choice of viewing. All but SM-C allow user to display whole model. Sam and SU explain model, SM-C can guide use of model. All are inspectable. Edit is allowed for those in control, sam, SM-T, SU. Add is allowed in sam and SM-T. In SM-C learner can request a test. In MrC, the learner model is negotiated (agreed). MrC provides overview and on request, further details on chosen issues. Sam interface allows arbitrary expansion of displayed model and SU always provides full model and current focus. SM-T has individual, group and comparative views, SM-C has own and peer-comparison. Needed for MrC negotiation and SM-C problems. Reflection goal means all allow user initiation. In SM-T the teacher can prompt a child. Only sam does this, by keeping model in user file space. Sam achieves this using file system access management MrC, SU require password – so not normally accessible. Fig. 8. The SMILI Open Learner Modelling Framework used to compare a set of systems, where 0 means the aspect is present, ^ means it is not. In addition, for Rows 1, 4 and 5, - indicates it is irrelevant in the system. For Row 9, all systems have some degree of flexibility of access. We note that in the child’s view of Subtraction Master, there was initially some concern about the possible discouraging effect of showing weak learners their performance against peers. However, it turned out that these children were already aware of their position in the class and were untroubled by the comparative model. Only Mr Collins and the child view of Subtraction Master have system-initiated model access (Row 10). In the case of Mr Collins, the negotiation requires symmetry of interaction initiative. Subtraction Master takes initiative when the child is having problems. Only the sam-coach model provides control over access (Row 11). It operated on a shared machine, storing the long-term learner model as files in the user’s own file space, and so was controlled by the user. The user could alter file access to control access by others, if they knew about unix access control! SUMMARY AND CONCLUSIONS The SMILI Open Learner Modelling Framework provides a method for researchers to describe and analyse open learner models. It also supports designers of personalised systems with open learner models. The framework also suggests dimensions of useful evaluations. For example, has the selection of the knowledge presented, actually supported the goals of the openness? Reflecting on the SMILI descriptions of our own systems, we have identified aspects that we had not considered but would now think worthy of consideration: for example, the availability of a knowledge level summary of the learner model. Similarly, we believe that it would be valuable to have provided more opportunity to reflect on the learner model in the past and to provide more predictive power. Moreover, the SMILI analysis highlights some of the trade-offs such as the conflicting goals of giving the learner control over who may access their model, against the benefits for reflection if there is a collated class model for comparison. Equally, the decision to make the samcoach model independent of any application, meant that its effect on personalisation could not be made accessible to the learner: this became evident after some time in our work, but with this framework, the trade-off would have been considered and understood from the start. A key issue arising from our work on SMILI was the need to explicitly define the difference between a learner model that can be edited, and one which allows the learner to add evidence to be used alongside existing evidence. Our earlier work talked about an interface for editing the model (Kay, 1995), but we would now call this addition. Similarly, the need to distinguish between model negotiation (Bull & Pain, 1995) and persuasion (Mabbott & Bull, 2006) became clear. Negotiation provides symmetrical moves available for initiating discussion with the requirement that changes to one party’s (student or system) viewpoint on the learner model must be agreed by that party. Persuasion gives the learner responsibility for initiating interaction to attempt to change the model contents, but the final decision rests with the system. Many systems have now opened their learner models. As e-learning, blended learning and personalised teaching systems become increasingly widely deployed, there is a corresponding need to consider the range of purposes that an open learner model can achieve. We have identified seven major purposes: accuracy; reflection; planning and monitoring; collaboration and competition; navigation; right of access and control as well as issues of improving trust; and assessment. Each places somewhat different demands on the features of openness. Without some framework, these varied goals and possibilities, combined with a diverse range of interfaces, appear like a large set of independent instances. Many of the potentially interesting issues are omitted from open learner model descriptions. This makes it harder to learn from previous work and to really understand what each system achieves. In turn, this makes it harder to create new systems that really build upon the past. It also makes it very difficult to see which aspects have been explored and which have not: we simply cannot tell if there are interesting classes of open models and modelling that have not been explored. We believe that if we had such information available, this would point to areas that deserve exploration. If we could also assess the common aspects of open learner models, especially those that have been built most recently, we might be able to see trends in the current research. We would also like to be able to see patterns in combinations of characteristics. We hope SMILI can provide a common ground for descriptions of the open learner models in systems, serving as a starting point for explanations of the design decisions, and making it easier to capture the full set of relevant details for a thorough description. We also hope that designers of future systems will find that SMILI assists them in working through the design decisions and trade-offs, and helps them to make more systematic design decisions. ACKNOWLEDGEMENTS This is an extended version of a paper presented at the Learner Modelling for Reflection workshop at the International Conference on Artificial Intelligence in Education 2005. We thank the participants of that workshop for trying out the framework, and in particular the following, whose suggestions have been incorporated into this version of the framework: Peter Brusilovsky, Albert Corbett, Ken Koedinger and Steve Tanimoto. We also thank Andrew Lum for reading the draft, trying to apply it to his own system, and making suggestions for clarifications. Subtraction Master was implemented by Mark McKay. REFERENCES Beck, J., Stern, M., & Woolf, B.P. (1997). Cooperative Student Models. In B. du Boulay & R. Mizoguchi (Eds.) Artificial Intelligence in Education: Knowledge and Media in Learning Systems (pp. 127-134). Amsterdam: IOS Press. Benyon, D., Kay, J., & Thomas, R.C. (1992). Building user models of editor usage. In E. Andre, R. Cohen, W.H. Graf, B. Kass, C.L. Paris & W. Wahlster (Eds.) UM 92: Third International Workshop on User Modeling (pp. 113 - 132). Wadern, Germany: Schloss Dagstuhl. Boud, D. (1985). Promoting Reflection in Learning: a Model. In D. Boud, R. Keogh & D. Walker (Eds.) Reflection: turning experience into learning (pp. 18-40). London, UK: Kogan Page. Brna, P., Self, J., Bull, S., & Pain, H. (1999). Negotiated Collaborative Assessment through Collaborative Student Modelling. In Proceedings of the Workshop on Open, Interactive and other Overt Approaches to Learner Modelling (pp. 35-42). 9th International Conference on Artificial Intelligence in Education, Le Mans, France. Brusilovsky, P., Sosnovsky, S., & Scherbinina, O. (2004). QuizGuide: Increasing the Educational Value of Individualized Self-Assessment Quizzes with Adaptive Navigation Support. In J. Nall & R. Robson (Eds.) Proceedings of E-Learn 2004: World Conference on E-Learning (pp. 1806-1813). Charlottesville, VA, USA: Association for the Advancement of Computing in Education. Bull, S., & Mabbott, A. (2006). 20000 Inspections of a Domain-Independent Open Learner Model with Individual and Comparison Views. In M. Ikeda, K. Ashley & T-W. Chan (Eds.) Intelligent Tutoring Systems: 8th International Conference, ITS 2006 (pp. 422-432). Berlin Heidelberg: Springer-Verlag. Bull, S., & McEvoy, A. (2003). An Intelligent Learning Environment with an Open Learner Model for the Desktop PC and Pocket PC. In U. Hoppe, F. Verdejo & J. Kay (Eds.), Artificial Intelligence in Education (pp. 389-391). Amsterdam: IOS Press. Bull, S., & McKay, M. (2004). An Open Learner Model for Children and Teachers: Inspecting Knowledge Level of Individuals and Peers. In J.C. Lester, R.M. Vicari & F. Paraguaçu (Eds.) Intelligent Tutoring Systems: 7th International Conference, ITS 2004 (pp. 646-655). Berlin Heidelberg: Springer-Verlag. Bull, S., & Nghiem, T. (2002). Helping Learners to Understand Themselves with a Learner Model Open to Students, Peers and Instructors. In Proceedings of the Workshop on Individual and Group Modelling Methods that Help Learners Understand Themselves (pp. 5-13). International Conference on Intelligent Tutoring Systems 2002, San Sebastian, Spain. Bull, S., & Pain, H. (1995). Did I say what I think I said, and do you agree with me? Inspecting and Questioning the Student Model. In J. Greer (Ed.) Proceedings of the World Conference on Artificial Intelligence in Education (pp. 501-508). Charlottesville, VA, USA: Association for the Advancement of Computing in Education. Bull, S., & Reid, E. (2004). Individualised Revision Material for Use on a Handheld Computer. In J. Attewell & C. Savill-Smith (Eds.) Learning with Mobile Devices: Research and Development (pp. 35-42). London: Learning and Skills Development Agency. Bull, S., Brna, P., & Pain, H. (1995). Extending the Scope of the Student Model. User Modeling and UserAdapted Interaction, 5, 45-65. Bull, S., Mabbott, A., & Abu-Issa, A.S. (2007). UMPTEEN: Named and Anonymous Learner Model Access for Instructors and Peers. To appear in International Journal of Artificial Intelligence in Education, 17(3). Bull, S., Quigley, S., & Mabbott, A. (2006). Computer-Based Formative Assessment to Promote Reflection and Learner Autonomy. Engineering Education: Journal of the Higher Education Academy Engineering Subject Centre, 1(1), 1-18. Bull, S., Mangat, M., Mabbott, A., Abu Issa, A.S., & Marsh, J. (2005). Reactions to Inspectable Learner Models: Seven Year Olds to University Students. In Proceedings of Workshop on Learner Modelling for Reflection (pp. 1-10). 10th International Conference on Artificial Intelligence in Education, Sydney, Australia. Burton, R.R., & Brown, J.S. (1979). An Investigation of Computer Coaching for Informal Learning Activities. International Journal of Man-Machine Studies, 11, 5-24. Chen, Z-H., Deng, Y.C., Chou, C-Y., & Chan, T-W. (2005). Motivating Learners by Nurturing Animal Companions: My Pet and Our Pet. In C-K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.) Artificial Intelligence in Education (pp. 136-143). Amsterdam: IOS Press. Collins, J.A., Greer, J.E., Kumar, V.S. McCalla, G.I., Meagher, P. & Tkatch, R. (1997). Inspectable user models for just-in-time workplace training. In A. Jameson, C. Paris & C. Tasso (Eds.) UM 97: The Sixth International Conference on User Modeling (pp. 327-337). New York, NY: Springer. Cook, R., & Kay, J. (1994). The justified user model: a viewable, explained user model. In A. Kobsa & D. Litman (Eds.) UM 94: The Fourth International Conference on User Modeling (pp. 145-150). New York: Springer. Corbett, A.T., & Anderson, J. (1995). Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278. Corbett, A.T., & Bhatnagar, A. (1997). Student Modeling in the ACT Programming Tutor: Adjusting a Procedural Learning Model with Declarative Knowledge. In A. Jameson, C. Paris & C. Tasso (Eds.) UM 97: The Sixth International Conference on User Modeling (pp. 243-254). New York: Springer. Crawford, K., & Kay, J. (1992). Shaping learning approaches with intelligent learning systems. In N. Estes & M. Thomas (Eds.) Proceedings of the International Conference for Technology in Education (pp. 1472-1476). Czarkowski, M., & Kay, J. (2000). Bringing scrutability to adaptive hypertext teaching. In G. Gauthier, C. Frasson & K. VanLehn (Eds.) ITS '00: Proceedings of the 5th International Conference on Intelligent Tutoring Systems (pp. 423-433). London: Springer-Verlag. Czarkowski, M., & Kay, J. (2006). Giving learners a real sense of control over adaptivity, even if they are not quite ready for it yet. In S. Chen & G. Magoulas (Eds.) Advances in Web-based Education: Personalized Learning Environments (pp. 93-125). IDEA. Czarkowski. M., Kay, J., & Potts, S. (2005). Web Framework for Scrutable Adaptation. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 11-18). International Conference on Artificial Intelligence in Education, Amsterdam, Netherlands. Dimitrova, V. (2003). StyLE-OLM: Interactive Open Learner Modelling. International Journal of Artificial Intelligence in Education, 13(1), 35-78. Dimitrova, V., Self, J., & Brna, P. (1999). The interactive maintenance of open learner models. In S. Lajoie & M. Vivet (Eds.) Artificial Intelligence in Education – Open Learning Environments: New Computational Technologies to Support Learning, Exploration and Collaboration (pp. 405-412). Amsterdam: IOS Press. Dufresne, A., & Hudon, M. (2002). Modeling the Learner Preferences for Embodied Agents: Experimenting with the Control of Humor. In Proceedings of the Workshop on Individual and Group Modelling Methods that Help Learners Understand Themselves (pp. 43-51). International Conference on Intelligent Tutoring Systems, San Sebastian, Spain. Grigoriadou, M., Tsaganou, G., & Cavoura, T. (2003). Dialogue-Based Reflective System for Historical Text Comprehension. In Proceedings of the Workshop on Learner Modelling for Reflection (Supplemental Proceedings Vol 5, pp. 238-247). International Conference on Artificial Intelligence in Education, Sydney, Australia. Grigoriadou, M., Papanikolaou, K., Kornilakis, H., & Magoulas, G. (2001). INSPIRE: An INtelligent System for Personalized Instruction in a Remote Environment. In P. De Bra, P. Brusilovsky & A. Kobsa (Eds.) PreWorkshop Proceedings: Third Workshop on Adaptive Hypertext and Hypermedia (pp. 31-40). Eighth International Conference on User Modeling. Hansen, C., & McCalla, G. (2003). Active Open Learner Modelling. In Proceedings of the Workshop on Learner Modelling for Reflection (Supplemental Proceedings Vol 5, pp. 248-257). International Conference on Artificial Intelligence in Education 2003, Sydney, Australia. Jean-Daubias, S., & Eyssautier-Bavay, C. (2005). An Environment Helping Teachers to Track Students' Competencies. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 19-22). International Conference on Artificial Intelligence in Education 2005, Amsterdam, Netherlands. Johnson, W.L., Vilhjalmsson, H., & Marsella, S. (2005). Serious Games for Language Learning: How Much Game, How Much AI? In C-K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology (pp. 306-313). Amsterdam: IOS Press. Kay, J. (1995). The um Toolkit for Cooperative User Modeling. User Modeling and User-Adapted Interaction, 4(3), 149-196. Kay, J. (1997). Learner Know Thyself: Student Models to Give Learner Control and Responsibility. In International Conference on Computers in Education (pp. 17-24). Charlottesville, VA: AACE. Kay, J., & Lum, A. (2003). Building User Models from Observations of Users Accessing Multimedia Learning Objects. In A. Nuernberger & M. Detyniecki (Eds.) Adaptive Multimedia Retrieval (pp. 36-57). Berlin Heidelberg: Springer. Kay, J., & Lum, A. (2005). Exploiting Readily Available Web Data for Scrutable Student Models. In C-K. Looi, G. McCalla, B. Bredeweg & J. Breuker Artificial Intelligence in Education (pp. 338-345). Amsterdam: IOS Press. Kay, J., & Thomas, R.C. (1995) Studying long term system use. Communications of the ACM, 38(7), 131-154. Kay, J., Lum, A., & Uther, J. (2003). How Can Users Edit and Control their Models in Ubicomp Environments? In Proceedings of the Workshop on User Modeling in Ubiquitous Computing (pp. 12-16). International Conference on User Modeling 2003. Kerly, A., Hall, P., & Bull, S. (2007). Bringing Chatbots into Education: Towards Natural Language Negotiation of Open Learner Models. Knowledge-Based Systems, 20, 177-185. Kobsa, A. (2002). Personalized hypermedia and international privacy. Communications of the ACM, 45(5), 6467. Li, L., & Kay, J. (2005). Assess: Promoting Learner Reflection in Student Self-Assessment. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 32-41). International Conference on Artificial Intelligence in Education 2005, Sydney, Australia. Linton, F., & Schaefer, H-P. (2000). Recommender Systems for Learning: Building User and Expert Models through Long-Term Observation of Application Use. User Modeling and User-Adapted Interaction, 10, 181-207. Lloyd, T., & Bull, S. (2006). A Haptic Learner Model. International Journal of Continuing Engineering Education and Lifelong Learning, 16(1/2), 137-149. Mabbott, A., & Bull, S. (2006). Student Preferences for Editing, Persuading and Negotiating the Open Learner Model. In M. Ikeda, K. Ashley & T-W. Chan (Eds.) Intelligent Tutoring Systems: 8th International Conference, ITS 2006 (pp. 481-490). Berlin Heidelberg: Springer-Verlag. Mazza, R., & Dimitrova, V. (2003). CourseVis: Externalising Student Information to Facilitate Instructors in Distance Learning. In U. Hoppe, F. Verdejo & J. Kay (Eds.) Artificial Intelligence in Education (pp. 279286). Amsterdam: IOS Press. Mitrovic, A., & Martin, B. (2002). Evaluating the Effects of Open Student Models on Learning. In P. De Bra, P. Brusilovsky & R. Conejo (Eds.) Adaptive Hypermedia and Adaptive Web-Based Systems, Proceedings of Second International Conference (pp. 296-305). Berlin Heidelberg: Springer-Verlag. Mohanarajah, S., Kemp, R., & Kemp, E. (2005). Opening a Fuzzy Learner Model. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 62-71). International Conference on Artificial Intelligence in Education 2005, Amsterdam, Netherlands. Morales, R., Pain, H., & Conlon, T. (2000). Understandable Learner Models for a Sensorimotor Control Task. In G. Gauthier, C. Frasson & K. VanLehn (Eds.) Intelligent Tutoring Systems: Proceedings of 5th International Conference, ITS 2000 (pp. 222-231). Berlin Heidelberg: Springer-Verlag. Mühlenbrock, M., Tewissen, F., & Hoppe, H.U. (1998). A Framework System for Intelligent Support in Open Distributed Learning Environments. International Journal of Artificial Intelligence in Education, 9(3-4), 256-274. Nielsen, J. (1994). Estimating the number of subjects needed for a thinking aloud test. International Journal of Human-Computer Studies, 41, 3, 385-397. Paiva, A., Self, J., & Hartley, R. (1995). Externalising Learner Models. In J. Greer (Ed.) Proceedings of the World Conference on Artificial Intelligence in Education (pp. 509-516). Charlottesville, VA: Association for the Advancement of Computing in Education. Papanikolaou, K.A., Grigoriadou, M., Kornilakis, H., & Magoulas, G.D. (2003). Personalizing the Interaction in a Web-Based Educational Hypermedia System: The Case of INSPIRE. User-Modeling and User-Adapted Interaction, 13(3), 213-267. Rueda, U., Larrañaga, M., Ferrero, B., Arruarte, A., & Elorriaga, J.A. (2003). Study of Graphical Issues in a Tool for Dynamically Visualising Student Models. In Proceedings of the Workshop on Learner Modelling for Reflection (Supplemental Proceedings Vol 5, pp. 268-277). International Conference on Artificial Intelligence in Education 2003, Sydney, Australia. Schon, D. (1983). The Reflective Practitioner. USA: Basic Books. Self, J. (1999). Open Sesame? Fifteen variations on the theme of openness in learning environments. International Journal of Artificial Intelligence in Education, 10, 350-364. Tanimoto, S. (2005). Dimensions of Transparency in Open Learner Models. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 100-106). International Conference on Artificial Intelligence in Education 2005, Amsterdam, Netherlands. Uther, J., & Kay, J. (2003). VlUM, a Web-Based Visualisation of Large User Models. In P. Brusilovsky, A. Corbett, & F. de Rosi (Eds.) User Modeling 2003 (pp. 198-202). Berlin Heidelberg: Springer-Verlag. Uther, M., & Kay, J. (2004). Learner modelling for mobile devices: from knowledge to context. In E. Murelli, G. DaBormida & C. Alborghetti (Eds.) Proceedings of MLEARN 2004: Learning Anytime Everywhere. Rome, Italy. Weber, G., & Brusilovsky, P. (2001). ELM-ART: An Adaptive Versatile System for Web-Based Instruction. International Journal of Artificial Intelligence in Education, 12(4), 351-384. Weber, G., & Specht, M. (1997). User Modeling and Adaptive Navigation Support in WWW-based Tutoring Systems. In A. Jameson, C. Paris & C. Tasso (Eds.) UM 97: The Sixth International Conference on User Modeling (pp. 289-300). New York: Springer. Weber, G., Kuhl, H-C., & Weibelzahl, S. (2001). Developing Adaptive Internet Based Courses with the Authoring System NetCoach. In Third Workshop on Adaptive Hypertext and Hypermedia: Pre-Workshop Proceedings (pp. 31-40). Eindhoven: Technische Universiteit. Yacef, K. (2005). The Logic-ITA in the classroom: a medium scale experiment. International Journal of Artificial Intelligence in Education, 15(1), 41-60. Zapata-Rivera, J-D., & Greer, J.E. (2001). Externalising Learner Modelling Representations. In Proceedings of the Workshop on External Representations (pp. 71-76). International Conference on Artificial Intelligence in Education 2001, San Antonio, Texas. Zapata-Rivera, J-D., Underwood, J.S. & Bauer, M. (2005). Advanced Reporting Systems in Assessment Environments. In Proceedings of the Workshop on Learner Modelling for Reflection (pp. 23-31). International Conference on Artificial Intelligence in Education 2005, Amsterdam, Netherlands.