Towards dialogue dimensions for a robotic tutor in collaborative learning scenarios

Alves-Oliveira, P., Janarthanam, S., Candeias, A., Deshmukh, A. , Ribeiro, T., Hastie, H., Paiva, A. and Aylett, R. (2014) Towards Dialogue Dimensions for a Robotic Tutor in Collaborative Learning Scenarios. In: 23rd IEEE International Symposium on Robot and Human Interactive Communication (2014 RO-MAN), Edinburgh, UK, 25-29 Aug 2017, pp. 862-867. ISBN 9781479967650 (doi:10.1109/ROMAN.2014.6926361) This is the author’s final accepted version. There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it. http://eprints.gla.ac.uk/160838/ Deposited on: 17 April 2018 Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Towards Dialogue Dimensions for a Robotic Tutor in Collaborative Learning Scenarios Patrı́cia Alves-Oliveiraa , Srinivasan Janarthanamb , Ana Candeiasa , Amol Deshmukhb , Tiago Ribeiroa , Helen Hastieb , Ana Paivaa , Ruth Aylettb Abstract— There has been some studies in applying robots to education and recent research on socially intelligent robots show robots as partners that collaborate with people. On the other hand, serious games and interaction technologies have also proved to be important pedagogical tools, enhancing collaboration and interest in the learning process. This paper relates to the collaborative scenario in EMOTE EU FP7 project and its main goal is to develop and present the dialogue dimensions for a robotic tutor in a collaborative learning scenario grounded in human studies. Overall, seven dialogue dimensions between the teacher and students interaction were identified from data collected over 10 sessions of a collaborative serious game. Preliminary results regarding the teachers perspective of the students interaction suggest that student collaboration led to learning during the game. Besides, students seem to have learned a number of concepts as they played the game. We also present the protocol that was followed for the purposes of future data collection in human-human and human-robot interaction in similar scenarios. I. I NTRODUCTION Recently, with the explosion of online learning, individualised and intelligent tutoring systems are gaining significant attention from different stakeholders. There have been some studies investigating the use of robotic tutors as a new interactive technology for learning [1], [2], [3]. In the EMOTE EU FP7 project1 , we aim to build robots to be used in individual and collaborative learning scenarios [4], [5]. The project aims to design, develop and evaluate a new generation of virtual and robotic embodied tutors with perceptive capabilities to engage in empathic interactions with learners in a shared physical space. The work presented in this paper relates to the collaborative scenario in EMOTE: a gamebased learning environment related to Geography curriculum running on a multi-touch table [4], [5]. This learning environment is set apart from traditional Intelligent Tutoring Systems (ITS) in that the robot embodiment provides the system with the ability to demonstrate intention to interact and can make use of gaze, mutual eye contact and ostensive signals (e.g., posture) that have been shown to improve learning [6]. Besides technologies to enhance learning, it is important to pay attention to interaction between teachers and students in classrooms. Such interactions are extremely rich combining many aspects of human communication through a INESC-ID & Instituto Superior Técnico, Universidade de Lisboa, Portugal. {patricia.alves.oliveira,tiago.ribeiro,ana.paiva}@inesc-id.pt, ana.m.candeias@tecnico.ulisboa.pt b School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh. {sc445, a.deshmukh, h.hastie, r.s.aylett}@hw.ac.uk 1 www.emote-project.eu both verbal and non-verbal cues. Such verbal interaction can be inherently seen as social, comprising important educational dialogue content [7]. As such, in order to develop an interactive robotic tutor, it becomes necessary to understand the multiple dialogue dimensions that teachers and students share during a learning process[8]. Towards this goal, we designed an experiment to collect dialogue data from students and teachers playing a collaborative serious game on Sustainable Development, on a large multi-touch table and a tablet. From this data, we identified the different dialogue dimensions underlying the teacherstudent interactions. The experiment was conducted in two schools - one in Edinburgh, UK and another in Lisbon, Portugal. This paper presents: (1) the experimental protocol that can be used in future research in similar educational settings, (2) an analysis of the dialogue dimensions for a robotic tutor in a collaborative learning scenario from sample data, and (3) preliminary findings regarding the students learning gains and teachers comments about collaborative learning in their interactions with students. II. R ELATED W ORK There has been very few studies in applying robots to education [1]. Recent research on socially intelligent robots show robots as partners that collaborate with people [9] and has made the use of robotic platforms in experimental learning more approachable [10]. Also, social supportive behavior in robotic tutors has been shown to have a positive impact on student’s learning performance [11]. On the other hand, serious games - educational games that provide learning content to players in addition to entertainment - have been used successfully for learning purposes [12]. Such games have shown to be very effective in helping students to learn new concepts [13]. Likewise, recent interaction technologies (e.g., multi-touch tables) have also proved to be important pedagogical tools, enhancing collaboration and interest in the learning process [14], [15]. III. C OLLABORATIVE L EARNING S CENARIO Our collaborative learning scenario is based on a serious game called Enercities2 . Enercities teaches sustainable development through discovery learning. The game presents opportunities for students to learn concepts such as pollution, energy shortages, renewable energy, etc. Project Enercities was co-funded by the European Commission programme 2 www.enercities.eu Fig. 1. Screenshot of Enercities game Intelligent Energy Europe and this project reports that this serious game made students aware of energy expenditure, among others [16]. Schools across Europe have already used Enercities in their curriculum and this game was therefore chosen as the basis of our collaborative scenario. Within the EMOTE project, the original online single player game was adapted to a multiplayer version where participants interact by using a multi-touch table or a tablet. This was done to stimulate collaborative learning [17], [4]. Overview of the game The game will be played by three players and the basic objective is to build a sustainable city over time (See figure 1). Players can build and improve structures such as housing, businesses, public amenities, etc in order to grow the population of the city. The team’s performance is measured in terms of points they score for economy, environment and citizen’s happiness. These are combined to form the overall team score. Players can choose one of the three available roles: environmentalist, economist and mayor. The three roles share resources such as oil, power and money. Some of the structures such as housing and power plants can be built by all of the players. But there are some structures that are role specific. For instance, only the environmentalist can build parks, forests and wildlife reserves. The economist can build businesses and the mayor can build public amenities. There are several levels in this game. The players can advance to the next level when they achieve the goal population set for each level. The game finishes when the team completes the final level of the game. The team loses the game when they run out of non-renewable resources midway. During the game, oil decreases over time (yet, its decrease can be slowed down), money can be earned by building or improving certain structures, and energy can be generated by building power plants or improving existing structure. The challenge is to manage the balance between several interlinked factors: growing the population, keeping them happy, keeping the environment green, generating enough power, building businesses to make money, etc. The game provides opportunities for students to learn several new concepts and how they affect each other. Game Moves During each turn, a player can perform one of three possible actions: build a structure, upgrade existing structures, implement a policy or skip the turn. To build a structure, the player has to select one of the structures available for his role. He/she can then try placing the structure at different places in the city to see how it affects the various indicators and then build it at someplace that is deemed beneficial. Another option is to perform up to three upgrades on existing structures to make them more efficient (i.e., produce less CO2 emissions, consume less power, etc.). The third option is to implement one of the five available policies, which improve several aspects and structures of the city in a single move. The player can also skip his/her turn. The game allows the players to collaborate and work as a team to build a healthy and sustainable city. It also allows players to be competitive in order to maximize their own individual scores. This presents them with several opportunities to learn the causal link between the indicators and the scores. Robotic Tutor and Enercities Our long term objective is to build a system where the game will be played by two students and a robotic tutor (see Figure 2 and attached video) [17]. The objectives of the robotic tutor will be to play the role of a team member, play the game, contribute positively to the discussions, and tutor the students on various underlying educational concepts as the game progresses. In order to develop the dialogue for the robot tutor, human studies with a teachers and students were conducted to serve as basis for the future verbal behavior of the robot tutor. Our study presents the first step towards this goal by developing the dialogue dimensions of the collaborative interaction. Fig. 2. Robotic tutor playing Enercities with students IV. O UR APPROACH Fig. 3. Experimental layout A. Participants The study was conducted in two European cities (Lisbon, Portugal and Edinburgh, UK). A total of 20 students and 4 Geography teachers participated in our study. Each session consisted of one teacher and two students. The students were from early secondary schools, aged between 12 and 15 (M age = 12.9). Table 1 shows the distribution of sessions for each teacher and the total time duration of the game interaction. Teacher Teacher Teacher Teacher Teacher A (Edinburgh) B (Edinburgh) C (Edinburgh) D (Lisbon) No. of sessions 2 2 1 5 Total interaction time 40min 40min 20min 100min TABLE I T EACHERS AND SESSIONS B. Experimental setup The game was presented to the participants on an interactive multi-touch device. In Edinburgh, an 18 inch Windows 8 DELL tablet was used and in Lisbon a 55 inch Multitaction table was used. In both cases, players could interact with the game using touch gestures. Three video cameras, one for each player, were used to capture their physical actions (i.e. body and facial gestures). Voice recorders and collar microphones were used to capture their speech. The experimental layout is shown in Figure 3. The teachers, students and their parents signed a consent form to participate in our study. Two students and a teacher were paired up as a team to play the game for each session (see Figure 4). regarding the interaction of students during the game. The questionnaire was based on the key-principles of collaborative learning [18], and contained 3 questions. An example of a question is the following: “In this collaborative game students have interacted with each other influencing their learning process”. The questions rated the interaction of the students in a 7 point-likert scale ranging from Never to Always. There was also a section where teachers could express their qualitatively opinion about the interaction. Students’ Knowledge Questionnaire A two-part knowledge questionnaire was designed with the help of a Geography teacher. The questions were based on the concepts and causeeffect relationships presented in the game. The teacher was consulted to identify the difficulty levels of the questions, based on which the question paper was divided into two parts. Part 1 consisted of 10 easy questions and Part 2 consisted of 10 hard questions. Each question was presented as a statement to which the student has to answer “yes” if he/she agrees with the statement and “no” if he/she doesn’t. In case they are not sure what the answer is, they can answer “I don’t know”. The main goal of this measure was to identify learning outcomes related to sustainable development addressed in the game. This questionnaire was applied only in the study performed in Scotland and each student answered this questionnaire before and after playing the game. D. Procedure The present study comprised two phases: the initial phase where we met the teachers, and the experimental task itself. C. Questionnaires We designed two questionnaires, one for students and another for teachers. Teacher Questionnaire Teachers’ questionnaire was developed with the intention of getting their feedback Phase 1: Meet the teachers In Phase 1, we met the teachers in order to introduce the game to them. In this phase, the following issues were discussed with the teachers: Game basics: Basics of the game were discussed such the goals of the team, roles they play, structures that can be built and upgraded, etc. This was done so that the teachers got familiarized with the game and could understand the underlying educational concepts. Teacher’s role: The teacher’s role in the game was discussed. Teachers were informed that, although they play the game as yet another player, they should also be proactive by presenting ideas, encouraging students to explore the game, and learn the underlying concepts. They were asked to balance the fun and learning aspects of the game as they deemed appropriate. It is to note that teachers always played the Mayor role in the game. Demonstration: The game was demonstrated and the teachers were encouraged to play in order to get familiar with the interface, the game, and its features. Phase 2: Experimental task In the second phase, we conducted the actual experiment with students and teachers playing together (see figure 4 and video). The experimental task consisted in the following 5-step process: Step 1. The students were asked to sign a consent form. Step 2. They were administered a pre-test (in Edinburgh only). Step 3. The students and the teachers were given a brief introduction and a game tutorial for 10 minutes. The game was demonstrated showing them how it worked and what each person could do in their turn. Step 4. For each team, roles were assigned: the teacher always played the role of the mayor and the two students were randomly assigned to the other two roles as economist and environmentalist. Their goal was set to develop a sustainable city and move up as many levels as possible. The team played the game for 20 minutes. Their interactions were video and audio recorded. Experimenters left the room in order to give privacy to their interaction. Step 5: The teacher was then given a questionnaire to fill (in both Lisbon and Edinburgh). Meanwhile, the students did the post test questions (in Edinburgh only). V. D IALOGUE C ODING S CHEME Understanding the dialogue utterances is important to analyse and understand how the interaction between students and teachers happened in the collaborative learning scenario. Dialogue utterances from the participants can be pragmatically grouped together based on different aspects of the interactions. These are called dialogue dimensions. For instance, some utterances are about the game, whereas some other are about the educational concepts underlying the game. In order to develop virtual agents and robotic tutors that are capable of performing such conversations with students in collaborative learning scenarios, it is essential to identify and understand the different dimensions in such interactions[19]. Several dialogue coding schemes for analysing studentteacher dialogue interactions have been presented in the past [20], [21], [22]. These studies considered either individual learning (student-teacher interaction) or collaborative settings (students dyads) [23], [24]. To our knowledge, there is no study in which the dialogues actions and dimensions between three or more participants in a collaborative learning setting have been analysed. In order to identify dimensions in the conversation, two researchers watched some of the recorded sessions and identified different dialogue content of the participants that emerged during the game. The analysis performed by the two researchers was discussed and combined to establish dialogue dimensions that occurred during the game. Seven dimensions were identified. Each of the seven identified dimensions is briefly explained below, followed by transcribed examples taken from the recorded interaction. 1. Social: Social dialogue dimension includes utterances such as welcome and goodbye greetings (e.g., “Hello Mr environmentalist and Ms economist.”). 2. Game Status: This dimension includes dialogues regarding the availability of resources in the game, the population, or the scores. It is mainly a set of the utterances that represents the indicators of the game (e.g., “What is the environmentalist score?”). 3. Game Rules: Game rules dimension includes dialogue utterances related to instructions regarding whose turn is, the interface, and rules of the game (e.g., “The Smile belongs to the mayor, right?”). 4. Curricular: This dialogue dimension represents the utterances that links to the curriculum. It also includes the cost-benefit analysis on proposals and the relation between the different game indicators and real world concepts (e.g., “We shouldn’t build this because we are suppose to create a sustainable city.”). 5. Strategy: Strategy dimension includes utterances related to players’ expectations and goals. The strategy includes players’ utterances about game moves, negotiations (e.g., of scores, regarding the city plan), exploring the game and predicting consequences. The strategy can change along the game according to each player game status (e.g., “It is better if you compare other options before you decide, otherwise my score will be affected.”). 6. Evaluation: Evaluation concerns dialogue related to players’ auto and/or hetero evaluation of the game performance. In this type of dialogue dimension players make judgments about their own performance or about other players’ performance, for example, regarding scores evaluation (e.g., “Economist: “My economy sucks.”). 7. Task Unrelated: Task Unrelated dialogue dimension comprises dialogue that occurs during the game but is not directly influencing the learning process itself. Requesting thinking time (stalling), irony, joke and technical dialogue moves (unresponsive or faulty interface) are also included in this dimension (e.g., Irony/Joke example: “Put it on the Fig. 4. Teacher playing Enercities with 2 students sea. Just kidding!”). These dimensions will have to be further analysed and types of dialogue actions (semantic representation of utterances) need to be identified for each dimension in the near future. VI. P RELIMINARY A NALYSIS Teacher Perspectives: The teachers reported that during the game the students almost always interacted with each other influencing their learning process. Also, the students almost always maintained a balanced dialogue between team members through the game in teachers’ perspective. In addition, students always deliberated upon various game actions, and these deliberations took into account academic topics, game resources and their scores. Student’s Learning gains: We analysed the pre- and post-test knowledge questionnaires from the Edinburgh dataset to examine how playing the game affected the students’ learning. The learning gain was calculated from the pre- and post-test scores based on the following formula [25]: Learning Gain (LG) = (Post test score - Pre test score) / (Max Score - Pre test score) * 100 The above learning gain formula measures the percentage of concepts that were learned out of the concepts that needed to be learned after the pretest. For example, a student scored 5 out of 10 in the pre-test signifying that he has to learn 5 more concepts. He then scored 8 in the post-test meaning he learned 3 out the 5 concepts he had wasn’t aware of at the time of pre-test. The learning gain is calculated as (8-5)/(105) * 100 = 3 / 5 * 100 = 60%. Table II shows the mean scores for the pre- and posttest questionnaire from the 10 students in Edinburgh dataset. Most students found part 1 easy and part 2 a bit more difficult, which aligns with the earlier assessment of the teacher. Taking as an example the pre-test questionnaire, they correctly answered about 8.6 out of 10 in Part 1, and only answered 4.4 questions (out of 10) in Part 2 correctly. The learning gains for the two parts were 42.85% and 28.57%, respectively. Metric Mean pre-test score Mean post-test score Mean learning gain Part 1 8.6 9.2 42.85% Part 2 4.4 6.0 28.57% TABLE II L EARNING GAIN We also analysed the frequency of “I don’t know” answers. There seemed to be a drastic reduction between the pre- and post-tests. The number of “I don’t know” answers decreased by 25 to 40% for both parts of the knowledge questionnaire. The increase in learning gain and the decrease in uncertainty can be attributed to knowledge acquired during game-play. Despite differences in terms of learning gains, results show that these gains were not significative, p > .05. We attribute this to the small sample size. Metric Mean pre “I don’t know” Mean post “I don’t know” Mean relative change Part 1 1 0.6 -40% Part 2 3.8 2.8 -24% TABLE III R EDUCTION IN UNCERTAINTY It should also be noted that Part 2 consisted of difficult questions and to learn all of them the team would have to play several levels of the game. However, in the Edinburgh sessions (where the pre- and post-tests were applied), the students were only able to complete a maximum of 2 levels (out of 4) during the 20-minute time limit. This could have impacted their learning of more difficult concepts and therefore produced less learning gain compared to part 1. We will analyse this further in future. VII. C ONCLUSIONS The present study had three goals: (1) the main goal was to develop the dialogue dimensions for a robotic tutor in a collaborative learning scenario; (2) we also aimed to present preliminary findings regarding the students’ learning gains, and teachers’ perspective about collaborative learning regarding their interactions; and (3) to describe the experimental protocol that can be used in future research in human-human and human-robot interactive educational settings. Seven dialogue dimensions were identified from the teacher-student interaction data. These dimensions present the preliminary work done towards building a empathetic social robotic tutorial environment for collaborative learning scenarios within the EMOTE project. These results serve to inspire future collaborative learning scenarios using dialogue dimensions for a robotic tutor inspired in human studies. The identifyed dialogue dimensions can contribute on the students’ learning performance complementing the impact of social supportive behaviour[11], and using games for education[12]. Also, the preliminary results regarding the teachers’ perspective of the students’ interaction suggest that students collaborate in their learning process during the game. Besides, preliminary results regarding students’ learning outcomes suggest that students learned the underlying educational concepts during the game. Future work: In the future, we will validate the dialogue coding scheme with the data we have collected from these studies and identify dialogue actions within each dimension to represent student and teacher utterances. We also will annotate important aspects of verbal and nonverbal behavior of the interaction in this specific learning context. The annotated dialogue data, along with the outcomes from the teachers and students will be used to build simulations of the teacher-student behaviour to learn and test dialogue strategies for the robotic tutor. ACKNOWLEDGMENT This work was partially supported by the European Commission (EC) and was funded by the EU FP7 ICT-317923 project EMOTE and partially supported by national funds through FCT - Fundação para a Ciência e a Tecnologia, under the project PEst-OE/EEI/LA0021/2013. The authors are solely responsible for the content of this publication. It does not represent the opinion of the EC, and the EC is not responsible for any use that might be made of data appearing therein. R EFERENCES [1] T. Kanda, T. Hirano, D. Eaton, and H. Ishiguro, “Interactive robots as social partners and peer tutors for children: A field trial,” HCI, vol. 19(1-2), pp. 61–84, 2004. [2] E. Hyun, H. Lee, and H. Yeon, “Young children’s perception of irobiq, the teacher assistive robot, with reference to speech register,” in ICCM, 2012 8th International Conference on, vol. 1. IEEE, 2012, pp. 366– 369. [3] E. Hyun, H. Yoon, and S. Son, “Relationships between user experiences and children’s perceptions of the education robot,” in Proc. 5th ACM/IEEE International Conference on HRI. [4] A. Deshmukh, G. Castellano, A. Kappas, W. Barendregt, F. Nabais, A. Paiva, T. Ribeiro, I. Leite, and R. Aylett, “Towards empathic artificial tutors,” in Proc. of the 8th ACM/IEEE international conference HRI, 2013. [5] S. Serholt, W. Barendregt, T. Ribeiro, G. Castellano, A. Paiva, A. Kappas, R. Aylett, and F. Nabais, “Emote: Emboided-perceptive tutors for empathy-based learning in game environment,” in European Conference GBL, Porto, Portugal, 2013. [6] G. Csibra and G. Gergely, “Natural pedagogy,” Trends in Cognitive Sciences, vol. 13, no. 4, pp. 148 – 153, 2009. [7] R. Tiberius, “The why of teacher/student relationships,” Essays on teaching excellence, Professional and Organizational Development, vol. 10(4), 1993. [8] H. Bunt, “Dimensions in dialogue act annotation,” in Proc. International Conference LRE, 2006. [9] C. Braezeal, “Role of expressive behaviour for robots that learn from people,” Philosophical Transactions of the Royal Society B, vol. 364, pp. 3527–3538, 2009. [10] I. Leite, G. Castellano, A. Pereira, C. Martinho, and A. Paiva, “Modelling empathic behavior in a robotic game companion for children: An ethnographic study in real-world setting,” in Proc. ACM/IEEE International Conference on HRI, 2012. [11] M. Saerbeck, T. Schut, C. Bartneck, and M. D. Janse, “Expressive robots in education: Varying the degree of social supportive behavior of a robotic tutor,” in CHI 2010, USA, 2010. [12] L. Annetta, “Video Games in Education: Why They Should Be Used and How They Are Being Used,” Theory Into Practice, vol. 47(3), pp. 229–230, 2008. [13] A. Brisson, G. Pereira, R. Prada, A. Paiva, S. Louchart, N. Suttie, T. Lim, R. Lopes, R. Bidarra, F. Bellotti, M. Kravcik, and M. Oliveira, “Artificial intelligence and personalization opportunities for serious games,” in Eighth AIIDE Conference, 2012, 2012, pp. 51–57. [14] S. Higgins, E. Mercier, E. Burd, and A. Hatch, “Multi-touch tables and the relationship with collaborative classroom pedagogies: A synthetic review,” International Journal of Computer-Supported Collaborative Learning, vol. 6(4), p. 515538, 2011. [15] S. Higgins, E. Mercier, L. Burd, and A. Joyce-Gibbons, “Multitouch tables and collaborative learning,” British Journal of Educational Technology, vol. 43(6), p. 10411054, 2012. [16] Knol, E. and de Vries, P, “EnerCities, a serious game to stimulate sustainability and energy conservation: Preliminary results,” eLearning Papers, vol. 25, pp. 1887–1542, 2011. [17] T. Ribeiro, A. Pereira, A. Deshmukh, R. Aylett, and A. Paiva, “I’m the mayor: A robot tutor in enercities-2,” in 13th International Conference on AAMAS ’14, In press. [18] P. Dillenbourg, “What do you mean by collaborative learning?” in Collaborative-learning: Cognitive and Computational Approaches. Oxford: Elsevier, 1999, pp. 1–19. [19] S. Janarthanam, H. Hastie, A. Deshmukh, and R. Aylett, “Towards a serious game playing empathic robotic tutorial dialogue system,” in HRI2014, 2014. [20] N. K. Person and A. C. Graesser, “Fourteen facts about human tutoring: Food for thoughts for ITS developers,” in Proc. AIED 2003 WK, 2003. [21] A. C. Graesser, P. Chipman, B. C. Haynes, and A. Olney, “Autotutor: An intelligent tutoring system with mixed-initiative dialogue,” IEEE Transactions on Education, vol. 48(4), 2005. [22] S. Alexander and A. Sarrafzdeh, “Foundation of an affective tutoring system: Learning how human tutors adapt to student emotion,” International Journal of ISTA, vol. 4-3/4, pp. 355–367, 2008. [23] C. A. Graesser, K. Wiemer-Hastings, P. Wiemer-Hastings, and R. Kreuz, “AutoTutor: A simulation of a human tutor,” Journal of CSR, vol. 1, pp. 35–51, 1995. [24] C. van Boxtel, J. van der Linden, and G. Kanselaar, “Collaborative learning tasks and the elaboration of conceptual knowledge,” Learning and Instruction, vol. 10, pp. 311–330, 2000. [25] A. C. Graesser, G. Jackson, E. Mathews, H. Mitchell, A. Olney, M. Ventura, P. Chipman, D. Franceschetti, X. Hu, M. Louwerse, N. Person, and the Tutoring Research Group, “Why/AutoTutor: A test of learning gains from a physics tutor with natural language dialog,” in Proc. 25th Annual Conference of the CogSci. Boston, MA, 2003.