April 2006 Volume 9 Number 2 Educational Technology & Society An International Journal Aims and Scope Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology & Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such systems. The articles should discuss the perspectives of both communities and their relation to each other: • Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by proactively harnessing the available technologies and how they might be able to influence further developments through systematic feedback and suggestions. • Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has been forgotten. The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to Educational Technology & Society and three months thereafter. The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal: Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, HumanComputer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications, Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training Editors Kinshuk, Massey University, New Zealand; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany. Associate editors Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture, Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland; Vladimir Uskov, Bradley University, USA. Advisory board Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork, University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University, Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology, India; Paul Kirschner, Open University of the Netherlands, The Netherlands; Rob Koper, Open University of the Netherlands, The Netherlands; Ruddy Lelouche, Universite Laval, Canada; Rory McGreal, Athabasca University, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan; Toshio Okamoto, The University of Electro-Communications, Japan; Gilly Salmon, University of Leicester, United Kingdom; Timothy K. Shih, Tamkang University, Taiwan; Yoshiaki Shindo, Nippon Institute of Technology, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector, Florida State University, USA. Assistant Editors Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini, University of Michigan, USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland. Executive peer-reviewers http://www.ifets.info/ Subscription Prices and Ordering Information Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling. Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling. Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling. Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk, Information Systems Department, Massey University, Private Bag 11-222, Palmerston North, New Zealand. Tel: +64 6 350 5799 ext 2090. Fax: +64 6 350 5725. E-mail: kinshuk@ieee.org. Advertisements Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the journal, those involved in education and educational technology. Contact the editors at kinshuk@ieee.org. Abstracting and Indexing Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences, ISI Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED. ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at kinshuk@massey.ac.nz. from the editors at kinshuk@ieee.org. i Guidelines for authors Submissions are invited in the following categories: • Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words) • Book reviews • Software reviews • Website reviews All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review. • Each peer review submission should have at least following items: ƒ title (up to 10 words), ƒ complete communication details of ALL authors , ƒ an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, ƒ four - five descriptive keywords, ƒ main body of paper (in 10 point font), ƒ conclusion, ƒ references. • Submissions should be single spaced. • Footnotes and endnotes are not accepted, all such information should be included in main text. • The paragraphs should not be indented. There should be one line space between consecutive paragraphs. • There should be single space between full stop of previous sentence and first word of next sentence in a paragraph. • The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example, Internet based system, Distance learning). • Do not use 'underline' to highlight text. Use 'italic' instead. Headings Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are consecutive, there should be one blank like between them). Tables Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be provided for every table (below the table) and must be referenced in the text. Figures Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width. Please also provide all figures separately (besides embedding them in the text). References • All references should be listed in alphabetical order at the end of the article under the heading 'References'. • All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) or • • "(authors1, year1; authors2, year2)" style e.g. (Merrill, 1999; Kommers et al., 1997). Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]." It is important to provide complete information in references. Please follow the patterns below: Journal article Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology, 35 (2), 7-13. Newspaper article Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15. Or Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from http://www.vnunet.co.uk/News/88174. Book (authored or edited) Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University. Chapter in book/proceedings Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95. Internet reference Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004, from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html. Submission procedure Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest Editors will advise the authors regarding submission procedure for the final version. All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible. The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text. Please provide following details with each submission: ƒ Author(s) full name(s) including title(s), ƒ Name of corresponding author, ƒ Job title(s), ƒ Organisation(s), ƒ Full contact details of ALL authors including email address, postal address, telephone and fax numbers. The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via email to (Subject: Submission for Educational Technology & Society journal): kinshuk@ieee.org. In the email, please state clearly that the manuscript is original material that has not been published, and is not being considered for publication elsewhere. ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at kinshuk@massey.ac.nz. from the editors at kinshuk@ieee.org. ii Journal of Educational Technology & Society Volume 9 Number 2 2006 Table of contents Special issue articles Theme: Interoperability of Educational Systems Editorial: Interoperability of Educational Systems - Editorial of Special Issue Daniel Olmedilla, Nobuo Saito and Bernd Simon Interoperability in Personalized Adaptive Learning Lora Aroyo, Peter Dolog, Geert-Jan Houben, Milos Kravcik, Ambjörn Naeve, Mikael Nilsson and Fridolin Wild 1-3 4-18 Providing Author-Defined State Data Storage to Learning Objects Ayalew Kassahun, Adrie Beulens and Rob Hartog 19-32 Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus Approach Elisabetta Di Nitto, Luca Mainetti, Mattia Monga, Licia Sbattella and Roberto Tedesco 33-50 Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface Frans van Assche, Erik Duval, David Massart, Daniel Olmedilla, Bernd Simon, Stefan Sobernig, Stefaan Ternier and Fridolin Wild 51-67 Full length articles The Tablet PC For Faculty: A Pilot Project Rob R. Weitz, Bert Wachsmuth and Danielle Mirliss 68-83 Teachers’ Collaborative Task Authoring to Help Students Learn a Science Unit Yavuz Akpinar and Volkan Bal 84-95 Teaching ICT to Teacher Candidates Using PBL: A Qualitative and Quantitative Evaluation Sevinç Gülseçen and Arif Kubat 96-106 Recency Effect on Problem Solving in Interactive Multimedia Learning Robert Zheng and Bei Zhou 107-118 Can a Hypermedia Cooperative e-Learning Environment Stimulate Constructive Collaboration? Mary Victoria Pragnell, Teresa Roselli and Veronica Rossano 119-132 Web-based Learning in a Geometry Course Hsungrow Chan, Pengheng Tsai and Tien-Yu Huang 133-140 Logistic Regression Modeling for Predicting Task-Related ICT Use in Teaching Petek Askar, Yasemin Kocak Usluel and Filiz Kuskaya Mumcu 141-151 The Relationship between Educational Ideologies and Technology Acceptance in Pre-service Teachers Ercan Kiraz and Devrim Ozdemir 152-165 A Web-Based Synchronous Collaborative Review Tool: A Case Study of an On-line Graduate Course Fatma Cemile Serce and Soner Yildirim 166-177 An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students Petri Nokelainen 178-197 A framework and a methodology for developing authentic constructivist e-Learning environments Imran A. Zualkernan 198-212 Technology Adoption of Medical Faculty in Teaching: Differentiating Factors in Adopter Categories Nese Zayim, Soner Yildirim and Osman Saka 213-222 Inducing Fuzzy Models for Student Classification Ossi Nykänen 223-234 ISSN 1436-4522 1436-4522.(online) © International and 1176-3647 Forum(print). of Educational © International Technology Forum&ofSociety Educational (IFETS). Technology The authors & Society and the (IFETS). forum The jointly authors retainand thethecopyright forum jointly of theretain articles. the Permissionoftothe copyright make articles. digital Permission or hard copies to make of part digital or all orof hard thiscopies work for of part personal or allorofclassroom this work use for is personal grantedorwithout classroom fee provided use is granted that copies without arefee notprovided made or that distributed copies for are profit not made or commercial or distributed advantage for profitand or that commercial copies bear advantage the fulland citation that copies on the bear first page. the full Copyrights citation onfor thecomponents first page. Copyrights of this workfor owned components by others of than this work IFETS owned must by be honoured. others thanAbstracting IFETS mustwith be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to post otherwise, on servers, to republish, or to redistribute to post on to lists, servers, requires or to prior redistribute specifictopermission lists, requires and/or priora specific fee. Request permission permissions and/orfrom a fee. theRequest editors permissions at kinshuk@massey.ac.nz. from the editors at kinshuk@ieee.org. iii Book review(s) Using Technology in Teaching Reviewer: Jarkko Suhonen 235-237 Unlock the genius within; neurobiological trauma, teaching, and transformative learning Reviewer: Richard Malinski 238-240 ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at kinshuk@massey.ac.nz. from the editors at kinshuk@ieee.org. iv Olmedilla, D., Saito, N. & Simon. B. (2006). Interoperability of Educational Systems - Editorial of Special Issue. Educational Technology & Society, 9 (2), 1-3. Interoperability of Educational Systems - Editorial of Special Issue Daniel Olmedilla L3S Research Center and Hannover University, Deutscher Pavillon, Expo plaza 1, D-30539 Hannover, Germany Tel: +49 511 762.9741 olmedilla@l3s.de Nobuo Saito Keio University - Shonan Fujisawa Campus, Keio Research Institute at SFC, 5322 Endo, Fujisawa, Japan Tel: +81 466 47 5049 nobuo.saito@w3.org Bernd Simon Vienna University of Economics and Business Administration, Institute of Information Systems & New Media Augasse 2-6, A-1090 Vienna, Austria Tel: +43 1 31336 4443 bsimon@wu-wien.ac.at Why a Special Issue on Interoperability of Educational Systems? Interoperability - defined as the capability of different systems to share functionalities or data - has become a hot topic for educational technologists. From the educational point of view the increasing attention for interoperability research has been driven, for example, by ¾ the desire to collaborate on the development of content (maybe stored in multiple systems), ¾ the need for making content accessible in or via various systems (re-use), ¾ cross-organisational, collaborative learning and teaching, ¾ sharing of assessment data for the purpose of effective personalization of learning environments. Economical motivations for interoperability include: ¾ securing investments in content development, ¾ making designs of learning environments exchangeable (good practices), ¾ increasing the user value of the systems provided by integrating components of other systems, ¾ allowing specialisation in the field, so that vendors can focus on particular aspects of the educational value chain (e.g., content creation, assessment, skill management). From the information systems (IS) design point of view interoperability is required in order to ¾ break up the technological isolation of learning management systems and alike by fully integrating them in a company’s IT infrastructure, ¾ get access to crucial data stored in legacy systems, ¾ integrate business process-driven solutions with learning management, ¾ reduce time for and costs of system integration by providing reference specifications. National and international standardization consortia, such as ISO, IEEE, ADL, AACE, CEN/ISSS, HR-XML, IMS, or DIN, sometimes accompanied by research projects have produced an overwhelming number of standards and specifications. The sometimes even competing work is still heavily discussed in the field. At this stage of a global discussion process, not all answers have been provided yet. Many of us working in the field of the technology-enhanced learning have been frequently confronted with issues such as: what are the use cases or pedagogical models standards such as SCORM, LOM, or QTI have been designed for? Can the specifications help to fulfil the requirements of learning environments or system integration projects? What about industry adoption - have they reached a critical mass and if so, is an outstanding ability to solve a particular problem driving the adoption or are other forces driving the process? Motivated by the pedagogical, technological, and IS design driven need for interoperability, on the one hand, and the still ongoing discussion about specifications and standards, on the other, this special issue is devoted to give an overview about and advance current state-of-the art of interoperability research. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 1 Some Metrics on the Call for Contributions to this Special Issue This special issue has followed up a workshop held in conjunction with the 14th International World Wide Web Conference (WWW2005), May 10-14, 2005, in Chiba (Japan). Supported by the workshop the call for contributions has attracted 38 papers in total, from which 33 of them went into the review process (5 had to be rejected at an early stage, because of a lack of relevancy to the special issue). Each paper was reviewed by a minimum of two reviewers. In total 85 reviews were collected in a double-blind review process. Finally, four papers were accepted for leading with an acceptance rate of 11%. Each accepted paper was assigned to a responsible editor in order to ensure that reviewer’s comments were interpreted in the most beneficial way. In this Special Issue The issue is composed of five articles covering interoperability-related topics including the authoring of learning material using metadata and improvement of reusability, protocols for the exchange of such metadata, extensions to standard specifications for inclusion of new features in Learning Management Systems in a common way, and integration of cross-domain technologies and standards for learning. Aroyo et al. claim in “Interoperability in Personalized Adaptive Learning” (accepting Editor: Nobuo Saito) that interoperability plays a crucial role in adaptive technologies. They discuss state-of-the art and main challenges of interoperability specifications. Based on an enhanced Adaptive Hypermedia Application Model they model systems that support distributed user profiling in the context of interconnected educational repositories. Kassahun et al. (accepting editor: Daniel Olmedilla) extends the current SCORM standard in order to support author-defined data for learning objects which require the management of state information. “Providing AuthorDefined State Data Storage to Learning Objects” describes such an extension and a plug-in, DBLink, that implements it. Di Nitto et al. discuss in “Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus Approach” (accepting editor: Bernd Simon) strengths and weaknesses of the latest SCORM specification. Based on their experiences in the Virtual Campus project they propose specific extensions to SCORM’s metadata and sequencing specifications. They claim that reusability of learning objects can be increased by intruding attributes such as supervision mode, access modality or learning objective. The relationships they propose for connecting learning objects in a learning flow shall provide enhanced capabilities for instructional designers to model effective learning environments. “Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface”, by Van Assche et al. (accepting Editor: Nobuo Saito), describe the Simple Query Interface, a protocol intended to provide an universal interoperability layer for the search and exchange of resource’s metadata. Authors discuss the technical details of the interface and provide several case studies as a proof of its applicability. Conclusions The papers received to this call for contributions gave a clear sign that interoperability will still remain to be a hot topic of research in the forthcoming years. The standards and specifications currently available seem to solve only a small piece of the big puzzle. Issues such as how to describe learning resources best for re-use are still not fully resolved while completely new topics such as connecting business processes with learning, for example via re-usable competency definitions, arise while technology enhanced learning disseminates in corporate environments. So it remains clear that the seamless development of high quality content and making it accessible in a smart way is still a vision that requires a lot of research to be performed. However, the authors of the above papers have made their contributions to bring this ultimate goal one step closer. Acknowledgements Special thanks to our reviewers who have provided relevant and detailed comments to the authors of all papers submitted and helped us in the selection process: 2 Allyn Radford, Ambjörn Naeve, Andreas Pinterits, Arthur Stutt, Bill Blackmon, Daniel R. Rehak, David Massart, Eric Roberts, Erik Duval, Frans Van Assche, Hiroaki Chiyokura, Hiroshi Komatsuwaga, Hiroshi Makoshi, Jan Brase, John Toews, John Williams, Juan Quemada, Keiko Okawa, Kinshuk, Kinya Tamaki, Lora Aroyo, Lorna M. Campbell, Makiko Miwa, Marek Hatala, Martin Dzbor, Massimo Marchiori, Matthew J. Dovey, Michael Sintek, Mitsuru Ikeda, Nobuyau Makoshi, Norm Friesen, Oleg Liber, Peter Dolog, Peter Spyns, Peter van Rosmalen, Pythagoras Karampiperis, Rob Koper, Ruimin Shen, Seok-Choon Lew, Simos Retalis, Stefano Ceri, Uwe Zdun and Wayne Hodgins. 3 Aroyo, L., Dolog, P., Houben, G-J., Kravcik, M., Naeve, A., Nilsson, M. & Wild, F. (2006). Interoperability in Personalized Adaptive Learning. Educational Technology & Society, 9 (2), 4-18. Interoperability in Personalized Adaptive Learning Lora Aroyo Technische Universiteit Eindhoven, Computer Science – Information Systems PO Box 513, 5600 MB Eindhoven, the Netherlands l.m.aroyo@tue.nl Peter Dolog L3S Research Center, University of Hannover, Expo Plaza 1, 30539 Hannover, Germany dolog@l3s.de Geert-Jan Houben Vrije Universiteit Brussel, Computer Science, Pleinlaan 2, 1050 Brussels, Belgium Geert-Jan.Houben@vub.ac.be Milos Kravcik Fraunhofer FIT, Institute for Applied Information Technology Schloss Birlinghoven, 53754 Sankt Augustin, Germany Milos.Kravcik@fit.fraunhofer.de Ambjörn Naeve and Mikael Nilsson Knowledge Management Research (KMR) Group, Centre for user oriented IT-Design (CID) Royal Institute of Technology (KTH), 100 44 Stockholm, Sweden amb@nada.kth.se mini@nada.kth.se Fridolin Wild Vienna University of Economics, and Business Administration Institute of Information Systems and New Media, Augasse 2-6, 1090 Vienna, Austria Fridolin.Wild@wu-wien.ac.at ABSTRACT Personalized adaptive learning requires semantic-based and context-aware systems to manage the Web knowledge efficiently as well as to achieve semantic interoperability between heterogeneous information resources and services. The technological and conceptual differences can be bridged either by means of standards or via approaches based on the Semantic Web. This article deals with the issue of semantic interoperability of educational contents on the Web by considering the integration of learning standards, Semantic Web, and adaptive technologies to meet the requirements of learners. Discussion is m ade on the state of the art and the main challenges in this field, including metadata access and design issues relating to adaptive learning. Additionally, a way how to integrate several original approaches is proposed. Keywords Semantic Interoperability, Learning Standards, Personalized Adaptive Learning, Meta-Data Introduction Looking into the past, we can see that most ideas how to learn are not new. What is new, are the circumstances and opportunities. The existing education and training system has been engineered in an industrial age to suit the needs of routinely performed manufacturing processes. The knowledge age we live in, however, is demanding higher skilled jobs with capabilities for critical thinking, creativity, self-regulation, collaboration, and interpretation. Accordingly, the number of so called ‘knowledge workers’ is rapidly growing. Moreover, economic success has become increasingly dependant on flexible and anticipatory adjustment of business models, processes, and behaviours (Hamel & Välikangas, 2003). Again accordingly, studies show that 50% of all employee skills become outdated within three to five years (Moe & Blodgett, 2000). As a consequence, we need new pedagogical methods to cover the exigencies in education and training visible today. For technology enhanced learning, this boils down to one of the contemporary grand challenges (CRA, 2003): to develop environments that effectively enable each learner to get individual support in filling everchanging skills and competence gaps – i.e. to create environments for personalised adaptive learning. More ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 4 specifically, this requires “developing semantic-based and context-aware systems to acquire, organise, personalise, share and use the knowledge embedded in web and multimedia content, […] and to achieve semantic interoperability between heterogeneous information resources and services” (IST, 2004). If our educational solutions are to be cost effective they have to share and reuse distributed learning resources to achieve a critical mass of material. Then efficient retrieval mechanisms providing access to relevant and accurate information are needed, like federated searches over distributed learning repositories. To reduce the overload of retrieved resources and to provide individualized educational experiences personalization of learning activities and learning materials has to be considered. As learning repositories have distinct systems, semantic and structural heterogeneity, various interoperability issues arise that must be solved in open environments. These issues concern all the conceptual components in adaptive educational systems, like sharing of learning content, concepts, learner profiles, context models, learning design, adaptation and presentation specifications. There are already solutions; e.g. for the exchange of learning objects and learner profiles, but this is just a part of the whole complex problem. Interoperability can be circumscribed as “a condition that exists when the distinctions between information systems are not a barrier to accomplishing a task that spans multiple systems”(Christian, 1994). To minimize costs and raise quality, we are not as interested in the possibility of interfacing through human effort. Semantics (the study of meaning) is usually defined to investigate the relation of signs to their corresponding objects. In our context, semantic interoperability thus denotes the study of how to bridge differences between information systems on two levels: ¾ on an access level, where system and organisational boundaries have to be crossed by creating standardised interfaces that sharing of system-internal services in a loosely-coupled way ¾ on a meaning level, where agreements about transported data have to be made in order to permit their correct interpretation Semantic interoperability can be achieved when models are available that more or less resemble each other and where not too many (semantic) differences are to be detected and resolved. Here, we define model as a formal and explicit representation that can be used to precisely describe a part of the design, e.g. the content or a user's knowledge state. The rest of this article is structured as follows. In Section 2, we are going to discuss the implications put forth by semantic interoperability when applied to adaptive systems. This encompasses types, architectures, representations, and interoperability problems emerging from distributed adaptive personalisation in technology enhanced learning. In Section 3, we are going to present a formal model for personalised adaptive learning and give an overview on existing interoperability alternatives, related standards, and projects. In Section 4 we investigate issues related to the access of meta-data in interoperable personalised adaptive learning solutions. We will focus on querying learning object repositories at first to subsequently present an illustrative example of querying learner profiles. The article will conclude by a summary of the status quo reached so far in personalised adaptive learning interoperability and the identification of problems which we feel need to be tackled in the future. Semantic Interoperability of Adaptive Systems The increasing demand for personalization in e-learning applications leads to a process of user (learner) profiling which is inherently distributed. For multiple applications to effectively share and exchange user information between them for the sake of adapting the content to the user, they need to know the semantics of this user information and therefore resolve the issues related to semantic interoperability. In this part of the article, before we go into details later on, we consider the general state of the art in semantic interoperability in relation to distributed user profiles by introducing methods, techniques, tools, and issues related to distribution and semantic interoperability of adaptive systems. In the past decade we have witnessed a growing interest in applying adaptation and personalization in numerous application domains. The process of engineering information systems has shown a considerable change and adaptation has been a significant driver of this change. Concept-based systems are information systems that represent content using concept structures, i.e. that a model of the content (often referred to as domain model or content model) is a characteristic element of the design. This approach generally includes, as relevant aspects, the user's knowledge (user model) and the adaptation knowledge (adaptation model). 5 These systems are distinguished from systems in which the adaptation is defined without an explicit model of the content (e.g. because the content and structure are rather straightforward or small). Adaptive concept-based systems are becoming especially accepted in application areas where the main goal is to tailor large amounts of information to the characteristics of predefined groups, or to the individual preferences and knowledge state of the individual users. In the case of educational applications, it has become more or less standard that the systems express a behavior matching the individual user. The construction of concept-based systems is not a straightforward issue, certainly when the challenge is combined with the desire to add adaptation, e.g. for adaptive hypermedia systems, adaptive web information systems, and adaptive task-based systems. When we talk about adaptation and personalization, the user necessarily plays a fundamental role in the system and therefore in its design. The system might want to record a user's preferences (e.g. learning and cognitive styles, language), but also its assumptions about the current user's (knowledge) state. The systems typically maintain a model of the individual user as an overlay of the domain model in order to record the current state of the user with respect to their knowledge of domain concepts. These application dependant user models with the preferences and the state of the user are integrated in the user profile that in general comprises the available information about a user, which is used as a basis for adaptation of the content presentation to the user. In the rest of this section we focus on the user model sharing. Interoperability of adaptive systems generally includes sharing of the content, concepts, context, learning design, and adaptivity. Distributed User Profiles The nature of interoperability between adaptive systems and applications implies that there is a distributed process of sharing and exchanging user profiles. This process includes on the one hand providing profile information and on the other hand consuming user models for personalization or adaptation. The issue of semantic interoperability in the context of user profiling is a direct consequence of the distribution in user profile information. The decentralized process of distributed user profile information management demands a control that is essential for a successful application of user profiles. For a long time it has been very difficult or even impossible to share and exchange user profile data. More recently, suppliers and consumers of user profiles have shown an increased awareness of the need for standards for representing and exchanging user profile data, especially in the e-learning domain significant progress, (as we will see later in this article). At the same time we observe that the amount and diversity of profile-based applications makes it practically impossible to easily create a unified user profile infrastructure. One important aspect that should not be underestimated is that the meta-data for user profile data implies a lot of manual labor before it can effectively be exploited in data exchange between applications. However, the technological advances of the last few years, especially in the context of web and semantic web research, can come to the rescue with tools and methods to combine available data, to annotate profile information (semi-)automatically, and to provide applications with necessary profile metadata. The (semi-)automatic generation of metadata is an essential prerequisite for the semantic interoperability of profile-based applications such as e-learning applications. The creation of such metadata usually requires considerable intellectual input of humans. Contemporary web technology may offer opportunities for semantic interoperability between applications and their meta-data on a large scale basis, which could not be achieved by uncoordinated human input alone. When we investigate the means of automatic creation of metadata, we observe that ontologies provide an option for achieving semantic coherence between profile data items. Tools can help to minimize user efforts required for creating and maintaining annotations – and could therefore help to increase their overall quality. Architecture and Representation To effectively manage distributed user profiles in adaptive concept-based systems, an architecture for distributed profile exchange and management is needed. Different types of systems use different kinds of architectural solutions. There are differences in the way in which user profiles are used, and this has consequences for the personalization. We want to differentiate three basic architectural types: adaptive web-based systems, adaptive hypermedia systems, and adaptive task-based systems. All these architectures share the facility to maintain a representation of assumptions about one or more characteristic(s) in the models of individual users. In other words the system should maintain a model about the 6 user that for instance contains assumptions about their knowledge, misconceptions, goals, plans, preferences, tasks, or abilities. There are numerous issues that need to be considered regarding the user representation in a complex approach, including user environment (e.g. class, school, family, background), roles and stereotypes, historic and sensor information, trust and acceptance, generality and domain independence, expressiveness, inferential capabilities, import and export, privacy, and mobility. Obviously we can also distinguish different ways to communicate user profile data, e.g. via a centralized server, via peer-to-peer communication, using agent-based techniques, or using a constraint-based approach. For distributed user profile architectures, we need data models and languages for profile meta-data, especially to describe the semantics and semantic differences. The languages and technologies designed for the development of the semantic web provide useful instruments for the representation of semantics of profile data. We mention the concept of ontology as an explicit specification of a conceptualization. This basically means that an ontology is a formal way of describing (some aspects of) a possible real world. With this key concept, the semantic web research has given us languages that are useful for the basic interoperability of user profile data. The semantic web provides a framework for expressing and using ontologies through e.g. the use of RDF, RDF Schema and OWL (RDF, 2005). These languages come also with relevant tool support, such as APIs (e.g. Jena and Sesame), browsers and editors (e.g. Protege and KAON), as well as reasoners. Interoperability Issues with Distributed Profiles Representing user profile data is one step in the process, but with the distribution come several interoperability problems and issues related to their semantic differences. As examples, we have incompatibility (both between profiles and between profiles and applications), incompleteness in the sense of information missing from profiles, and contradiction in (unified) profiles. The need to consider these issues arise from the fact that a learner may attempt to use an application that requires more information than the user’s profile can provide, or that responds with information that cannot be accommodated in the user’s profile. The complementary case arises when an application cannot handle parameters such as preferences, specified by the learner or provides a response that contains too little information to enable the user to choose between alternate follow-up actions. Another class of problems arises when the learner’s profile contains sufficient information but the application possesses information of its own that disagrees with the information present in the learner’s profile, because of conflicting values or meanings. When two e-learning applications are directly interacting to provide a learner with a certain service, but without the direct involvement of the user, they may face the situation of having access only to partly overlapping, incomplete profiles. The question now is how to resolve overlaps and fill existing gaps. When applications are allowed to autonomously replenish missing data, it might happen that two applications create contradictory information. Can and should this be prevented from happening? And if not, how can contradictory data be corrected afterwards? How can co-existing, possibly conflicting data be dealt with? Other sources of contradictory information are different versions of the same information (freshness). The issue here is whether to trust the most recent version or to establish a procedure to validate information. When facts are contradictory within one source (document), one speaks of an inconsistency. When information statements from different sources are contradictory, one speaks of disagreement. A disagreement may turn up when two sources are merged (e.g. in a data warehouse project or virtually as described above). These two situations require different handling. In the context of interoperability, one may assume as a starting point that the sources are consistent. The proper treatment of disagreement is the more relevant problem to tackle. These examples illustrate the situations that have to be prepared for and dealt with. All these problems can be discussed from different angles: on the level of schemas or ontologies, at the level of instances, within an information source, or between information sources. The main question is how to identify and deal with missing or incomplete information. Interoperability problems in distributing user profiles include imprecise information, imprecise manipulation, uncertain information, schema and ontology mapping, data clearing, inconsistency, mediation, data dissemination, data replication, conflict detection and reconciliation. Techniques and architectures to be considered for solving these interoperability problems draw from different research areas – such as classical databases, data warehouses, mobile information systems, and semantic web technologies. 7 A Formal Model and Related Standards After giving the general perspective on interoperability and distribution of adaptive components, we now turn to formal models and standards that play a role in the adaptive hypermedia systems that are at the basis of most adaptive learning systems. To start with, we observe that the knowledge driving the adaptation process can be represented in adaptive hypermedia systems as five complementary models (Figure 1) – the domain model specifies what is to be adapted, the user and context models tell according to what parameters it can be adapted, and the activity (instruction) and adaptation models express how the adaptation should be performed. This model enhances the Adaptive Hypermedia Application Model (De Bra et al., 1999; Wu et al., 2001), which is based on the Dexter Hypertext Reference Model (Halasz & Schwartz, 1994). We use an analytical approach to identify the different design aspects in which the separation between various types of knowledge asks for interoperability as well. Figure 1: Enhanced Adaptive Hypermedia Application Model Thus, in accordance with cognitive science results, instead of uniform rules that should control the whole personalized adaptive leaning experience, several specialized parts take care of particular functions and interact with each other. Note that individual models may be distributed in reality. In the following paragraphs we discuss formal models and standards that apply to each of the particular models. As we see, the existing standards do not really support interoperability as a common abstract model is missing. They can be used in isolation, but that is not desirable. Domain Model The domain model specifies the conceptual design of an adaptive hypermedia application, i.e. what will be adapted. The information structure of a domain model in a typical adaptive hypermedia system can be considered as two interconnected networks of objects (Brusilovsky, 2003): ¾ Knowledge Space – a network of concepts ¾ Hyperspace – a network of hyperdocuments Accordingly, the design of an adaptive hypermedia system involves three key sub-steps: structuring the knowledge, structuring the hyperspace, and connecting the knowledge space and the hyperspace. Modern adaptive hypermedia systems model the knowledge space of the domain as a semantic network (Brusilovsky, 2003). They use network models with several kinds of links that represent different kinds of relationships between concepts, e.g. prerequisite links between concepts which represent the fact that one of the related concepts has to be learned before another, or classic semantic links “is-a” and “part-of”. These domain 8 ontologies represent the expert’s knowledge about the domain. The domain model offers also a natural framework for goal modeling. An individual educational goal can be modeled as a structure (e.g. sequence, tree, or stack) of subsets of domain concepts. The hyperspace consists of learning objects. The Learning Object Metadata (LOM) standard defines a learning object as any entity, digital or non-digital, that may be used for learning, education or training (LOM, 2002). Content models identify different kinds of learning objects and their components. A comparative analysis of six known content models (Verbert & Duval, 2004) led to the creation of a general model that includes the existing standards and distinguishes between: ¾ Content fragments – learning content elements in their most basic form (text, audio, video), representing individual resources uncombined with any other; instances ¾ Content objects – sets of content fragments; abstract types ¾ Learning objects – they aggregate instantiated content objects and add a learning objective The standards that can be used at this level include ¾ IMS Content Packaging – description and packaging of learning material ¾ IMS Question and Test Interoperability – XML language for describing questions and tests ¾ IEEE Learning Object Matadata – description of learning resources Learning objects distributed in various repositories with associated metadata can be nowadays retrieved by means of federated search (Simon et al., 2005). Early adopters have already started using these services. These users can be either learners using learning objects in a similar way like textbooks, or teachers that need suitable materials to support their classes and possibly applying blended learning approaches. User Model A user model represents relevant user characteristics, like preferences, knowledge, competencies, tasks, or objectives. The majority of educational adaptive hypermedia systems use an overlay model of user knowledge (Brusilovsky, 2003). The key principle of the overlay model is that for each domain model concept, an individual user knowledge model stores some data that is an estimation of the user knowledge level on this concept. A weighted overlay model of user knowledge can be represented as a set of pairs “concept-value”, one pair for each domain concept. Some systems store multiple evidences about the user level of knowledge separately. Another alternative to model the user knowledge is provided by a historic model that keeps some information about user visits to individual pages. Some systems use this model as a secondary source of adaptation. The learner’s goals can be modeled as a set of concepts (competencies) that can be represented similarly to the overlay model. Additionally to these dynamic dimensions the learner model includes also a more static one – user preferences. The most relevant ones are preferred cognitive and learning styles, as well as the language. The following standards relate to user modeling: ¾ IEEE Public And Private Information – specifies both the syntax and semantics of a 'Learner Model,' which will characterize a learner and his or her knowledge/abilities ¾ IMS Learner Information Package – learner information data exchange between systems that support the Internet learning environment New approaches for open-world user modeling able to elicit extended models of users and to deal with the dynamics of a user’s conceptualization are required to effectively personalize the Semantic Web. The ultimate goal is to pave the road towards utilizing the adaptive learning environments with an enhanced learner model that will integrate different learner perspectives, such as knowledge, personal preferences, and interests, browsing patterns, cognitive and physical state. Complementing to the open-world view is the fact that learner can be also modelled by various peers and learning service providers in a distributed network. The key research issues in this area are: interoperable learner model artefacts, techniques for describing such artefacts, methods for extracting relevant learner model parts for particular learning situations or services, techniques for exchanging and communicating such artefacts. Other major challenges and directions for further research include effective learner modeling, addressing personalization for disabilities, considering the time and evolving user context(s), including the user control, and relating them to the issues of privacy. New challenges are recognized with regard to moving to open educational semantic world. 9 Context Model The term context can be defined as “the circumstances in which an event occurs; a setting”. We are considering it as the environment characteristic rather than the user ones that are represented in the user model. The user (learner) and context models specify to what parameters the application should adapt. One of the primary aims is to generate both objective and subjective metadata automatically, based on the current context and by means of suitable sensors – physical as well as semantic ones. This will enable more precise retrieval of the data when learning objects are processed or elaborated by students and teachers. Context management deals with such issues as automatic acquisition of context metadata, contextualized delivery of content, activities, and services. We observe that the current exchange formats for contextualization of resources need to be extended for capturing and handling additional context data. There are no relevant standards for the context model yet. Modern context-adaptive systems employ generic and mobile user models to provide human centered and ubiquitous services. As the learner models will also include situated learning in the context of work where learning is an integrated part of working (learning on demand) user competence profile has to be taken into account, as well as analyzing group modeling and pattern recognition in the user behavior. Metadata about the learner and her context should enrich queries into learning object repositories to maximize the efficiency of information retrieval. Instruction Model The instruction (pedagogical) and adaptation models specify the navigational design for an adaptive hypermedia application. Together with the presentation specification they tell how the adaptation should be performed, so they describe the dynamics (“flow”) of the system. Learning design is a way of modeling learning activities and scenarios, as different types of learners prefer different learning approaches – learning styles. A key axiom that is common to all major educational approaches says that “learners perform activities in an environment with resources”. The IMS Learning Design (Koper & Tattersall, 2005) uses the metaphor of a theatrical play to describe the workflow involved in learning and teaching scenarios. It separates the design of the pedagogical model from the content. Main challenges include encoding dynamic interactions between users and system, representing scenarios (objectives, tasks/activities), and describing interactions between participating roles and system services. Standards that are related to the design of pedagogical activities are: ¾ IMS Simple Sequencing – representing the intended behavior of an authored learning experience ¾ IMS Learning Design – defining diverse learning approaches (scenarios); it defines 3 levels of implementation and compliance (IMS LDIM, 2003): ƒ Level A – the core vocabulary needed to support pedagogical diversity ƒ Level B – introduces properties and conditions, which enable implementation of adaptive strategies ƒ Level C – notification that can support adaptive self-driven and collaborative learning The primary aim of the learning design standard was to provide an explicit notation that would enable interoperability on the level of systems. Thus the instructional knowledge does not have to be hardwired in the learning environment, but authors can define it specifically for each learning application, representing an appropriate pedagogical pattern. To allow personalization a method can contain conditions, i.e. If-Then-Else rules that further refine the assignment of activities and environment entities for persons and roles. Conditions can be used to personalize learning designs for specific users. The ‘If’ part of the condition uses Boolean expressions on the properties that are defined for persons and roles in the learning design. Thus IMS Learning Design can be used to model and annotate adaptive learning design, but designing more complex adaptivity behavior can cause problems. Currently, it is not possible to annotate learning content or define student roles considering their characteristics. We can say that a primary objective of this standard was interoperability between various systems, rather than reusability of learning design methods in various courses or learning units. 10 Adaptation Model This model specifies the specific adaptation semantics – seen, mastered, recommended objects, etc. Adaptation specifications define the status of individual objects (e.g. content objects or fragments) based on their metadata attributes and the current parameters of the user model and the context model. The adaptation effect is usually achieved by adapting contents and links using suitable adaptation techniques that can be chosen on this level. The taxonomy of adaptive hypermedia technologies (Brusilovsky, 2001) includes: ¾ Adaptive presentation (content level adaptation) to ensure for different classes of users that the (most) relevant information is shown and the user can understand it, e.g. adaptive text presentation, adaptive multimedia presentation, adaptation of modality ¾ Adaptive navigation support (link level adaptation) to guide the user towards the relevant, interesting information, e.g. direct guidance, adaptive link sorting, adaptive link hiding, adaptive link annotation, adaptive link generation, map adaptation There can be also other adaptation dimensions, e.g. adaptive learning activity selection, adaptive recommendation, or adaptive service provision. The Adaptive Hypermedia Application Model or AHAM (mentioned earlier) uses Condition-Action rules and due to their complexity, it is not supposed that authors will write all the rules by hand. Some other models build upon AHAM identifying additional relevant layers, with the objective to enable reusability at various levels, focusing mainly on adaptation strategies and techniques. LAOS (Cristea & de Mooij, 2003) is a generalized model for generic adaptive hypermedia authoring, based on the AHAM model and on concept maps. It aims at clear separation of primitive information (content) and presentation-goal related information (e.g. pedagogical information). Previously they have defined a layered model for adaptive hypermedia authoring design methodology for (WWW) courseware (Cristea & Aroyo, 2002). This model suggested the usage of the following main three layers: ¾ Conceptual layer expressing the domain model (with sub-layers: atomic concepts and composite concepts – with their respective attributes) ¾ Lesson layer (of multiple possible lessons for each concept map or combination of concept maps) ¾ Student adaptation and presentation layer (based on: adaptation model and presentation model) All these layers should have been powered by the adaptation engine. Already they were using the lesson model as an intermediate one between the domain model, the user and adaptation model. LAG (Cristea & Calvi, 2003) is a generalized adaptation model for generic adaptive hypermedia authoring. The idea behind it was to let the author of adaptive educational hypermedia work on a higher semantic level, instead of struggling with the ‘assembly language of adaptation’. Furthermore, these patterns should represent the first level of reusable elements of adaptation. However, reusability can go further than that. Even this adaptation language might still be difficult to handle for some authors (teachers). So reuse should be aimed for even at the level of adaptation strategies (that correspond to cognitive/learning strategies). On a higher level the presentation specification defines how to present the chosen adaptation techniques as well as how the objects with a particular status should be presented to the user (e.g. hiding, sorting, emphasizing, annotation techniques). Accessing Meta-Data of Adaptive Learning Systems Part of solving the semantic interoperability problem is that systems have to mutually understand their access mechanisms to learning content objects and associated meta-data. More specifically, they must know programming interfaces to connect to, retrieve, and manipulate all necessary meta-data. The Application Program Interfaces (API) are either domain specific, i.e. they are based on specific metadata models, or they are generic, usually suitable to query metadata based on multiple schemas by making use of general purpose query languages like SQL. First we are dealing with the issues related to querying learning repositories and then we present an illustrative example for querying learner profiles. Generic APIs to Access Learning and Learner Repositories One possibility to access learning content and learner profiles is to use generic APIs. Those APIs usually provide functions to execute general purpose query languages independent of a domain. Examples of such APIs are 11 Simple Query Interface (Simon et al., 2005) used mostly exclusively for but not limited to querying metadata about learning objects or APIs to access user profiles at User Modeling Servers (Fink & Kobsa, 2002). Interoperability among learning object repositories requires a common communication framework for querying and retrieving references to the stored objects. In the following we present different query APIs in the learning domain. The Learning Object Interoperability (LORI) Framework is an abstract model for interoperability issues in the context of learning technology (Simon et al., 2005). LORI is a layered integration architecture, which defines services in order to achieve interoperability between independent learning object repositories. These services can be differentiated into core services (e.g. authentication services, session management services) and application services (e.g. retrieval services, provision services). Part of the LORI framework is the Simple Query Interface (SQI 2005). SQI is an API that provides method support for asynchronous and synchronous queries. Albeit SQI is in principle not bound to a specific schema and is thus only dependant on a jointly defined canonical schema of the specific community where it shall be applied, several examples like the HCD-Suite (www.hcdonline.com) can be found that use application profiles to mix standardized concepts from IEEE LOM, Dublin Core, and other standards simultaneously in order to satisfy the needs inherent in their community. One of its major design objectives of SQI was to keep the specification simple and easy to implement. The Content Object Repository Discovery and Resolution Architecture (CORDRA) is supposed to develop into an abstract, formal model for repository federations and their interoperability (CORDRA, 2004). Currently, CORDRA encompasses similarly to OpenURL an identifier infrastructure and is currently not bound to any specific search interface implementation (Rehak, et al 2005). OpenURL (OpenURL, 2004) is an initiative to investigate the problem of uniquely identifying resources. Z39.50-International: Next Generation (ZING) covers a number of initiatives by Z39.50 implementers to make Z39.50 (ZING, 2001) more broadly available and to make Z39.50 more attractive to information providers, developers, vendors, and users. SRW is the Search/Retrieve Web Service protocol, which is developed within ZING and aims to integrate access to various networked resources, and to promote interoperability between distributed databases, by providing a common utilization framework. SRW is a web-service-based protocol (SRW, 2004). SRW takes advantage of CQL ("Common Query Language"), a powerful yet human-readable query language. SRU, the Search and Retrieve URL Service, is a companion service to SRW. Its primary difference is its access mechanism: SRU is a simple HTTP GET form of the service (SRU, 2005). SRW encourages the use of Dublin Core, but is in general schema-neutral (like SQI). The purpose of the IMS Digital Repository Interoperability (DRI) Specification (IMS DRI, 2003) is to provide recommendations for the interoperation of the most common repository functions. The DRI specification presents five core commands, i.e. search/expose, gather/expose, alert/expose, submit/store, and request/deliver, on a highly abstract level. The specification leaves many design choices for implementers. For example, while recommending Z39.50 (with its own query language) it also recommends XQuery as a query language. OKI (Open Knowledge Initiative) is a development project for a flexible and open system to support on-line training on Internet (OKI, 2004). OKI has issued specifications for a system architecture adapted to learning management functions. One of the main characteristics of the project is its commitment to the open source approach for software component development. OKI supplies specifications for a model of functional architecture and an API called Open Service Interface Definition (OSID). OSID is directed towards specifications for a flexible and open source model of functional architecture. Service Interface Definitions (SIDs) organize a hierarchy of packages, classes and agents and propose Java versions of these SIDs for use in Java-based systems and also as models for other object-oriented and service-based implementations. Components developed by OKI are compliant with specifications issued by IMS and ADL SCORM. Edutella is an RDF-based Peer-to-Peer infrastructure for querying distributed learning object repositories that comes with its own query language QEL (QEL, 2004). The Resource Description Framework (RDF) is one of the key pillars of the Semantic Web (RDF, 2005). RDF is an extensible way to represent information about (learning) resources. One of RDF’s design assumptions is that resources are identified by a Unique Resource Identifier (URI) allowing various users and agents to make assertions about uniquely identified objects. The API in Edutella was used both to query learning resource metadata and learner profiles in Elena project. 12 Domain Specific APIs to Access Learning and Learner Repositories Another possibility how to access learning and learner repositories is to use domain specific APIs. The learning repositories nowadays conform to Learning Object Metadata standard by applying its profiles. The concepts from LOM can be used to design APIs and libraries to query the repositories by using the concepts from the domain of interest. Similarly, several open specifications have emerged to provide shared information models to represent learner profiles like IEEE PAPI (IEEE PAPI, 2003) or IMS LIP (IMS LIP, 2001). The concepts from those information models can be used to build APIs for accessing learner profiles. An example of learning repositories domain specific API is EduSource. EduSource project (Hatala et al., 2004) aims to implement a holistic approach to building a network for learning repositories. As part of its communication protocol – referred to as the EduSource Communication Language (ECL) – the IMS Digital Repository Specification was bound and implemented. A gateway for connecting between EduSource and the NSDL initiative, as well as a federated search connecting EduSource, EdNA and Smete serve as a first showcase. A different example of domain specific API is the API for accessing learner profiles. The Lerner API (Dolog & Schäfer, 2005) was developed in the context of FP5 EU/IST project Elena – Creating Smart Spaces for Learning. The API is based on a learner ontology. Figure 2 depicts an excerpt of a learner profile ontology configured from fragments based on three specifications (the Elena project web site http://www.elena-project.org and its personalization section provide the complete ontology in RDFS). The abbreviated syntax for namespaces is used in concept and relation labels (e.g. qti stands for Question and Test Interoperability namespace at http://www.elena-project.org/images/other/qtilite.rdfs). The default namespace is http://www.elenaproject.org/images/other/learner.rdfs. The conceptual model describes a situation where a learning performance (IEEE PAPI is used to model performance and portfolio, http://ltsc.ieee.org/archive/harvested-200310/working_groups/wg2.zip) of a student is exchanged in terms of his achieved competencies (IMS RDCEO – Reusable Definition of Competency and Educational Objectives, http://www.imsglobal.org). The competencies have been evaluated by learner assessment (e.g. tests) and were derived from learning objectives of tests (IMS QTI). Furthermore, all other educational activities, further materials, and projects created within the activities are reported within the portfolio of the performance. Additional information which is reported under preferences (IMS LIP) comprises language, device, resource and learning style preferences. The standards and open specifications guarantee wider acceptance between e-learning systems and as such can be seen as good candidates for the learner exchange models. Currently, none of the referenced standards present their metadata in a way that makes it possible to use them in combination as depicted above. Therefore, an RDF translation of these standards had to be developed, which made it possible to use them in combination. This RDF translation is ‘unofficial’, and we therefore view it as an important direction for future standardization work that the standards use a common framework such as RDF and the Semantic Web, to enable the added value of using the standards together. Figure 2: Conceptual model for learner profile – an excerpt Figure 3 depicts several possible scenarios of how to access and exchange learner profile fragments. The fragments can be accessed programmatically by the use of a Java API, the web service which exports the learner model through the API and acts as a learner model server, and through a query infrastructure for RDF repositories like Edutella (Nejdl et al. 2002). 13 Figure 3: Use of API in several scenarios – an example A Java API has been developed. It is structured according to the learner ontology fragments mentioned above. The API is meant to be used to retrieve, insert, and update the learner profiles stored in the structures described above. The API defines a class and properties for each class from the RDFS for the learner model. The interface provides access functions for getting, deleting and updating a model of the fragment. It provides further functions to derive additional information or to process more complex manipulations over referenced information types as well. The API is implemented for the RDF representation (instances of the RDFS described above). The API is easily extensible by providing further specializations if additional extensions and interface implementations for local repositories and data models are needed. The second implementation is provided through web services where several clients can access one model which is persistent on one server. The server holds the main model, i.e. the data of a learner profile gathered from several sources, and handles all requests from the clients. Each client is uniquely identified at the server and can be used by a browsing or assessment system. Furthermore, a client can be used by other learning systems which want to make use of the learner profiles or which want to contribute to them. The model can be accessed directly by invoking functions of a web service or in a synchronized replicated way; i.e. each client has its own repository which is synchronized with the main server every time a change occurs. The web services framework can be used in a distributed way as well (several servers exchanging learner models between each other). The learner profiles are created in RDF. Therefore, a query infrastructure for RDF data is another access option. Edutella provides a Datalog-based language to query RDF data provided in a distributed P2P environment. This option enables to collect various fragments by utilizing for example the algorithm from (Dolog, 2004). Another advantage of the P2P sharing infrastructure used with the learner profiles is that it can facilitate an expert finding based on the provided profile which can be queried by people who need a help in learning. Summary and Conclusion Complex problems require more knowledge than any single person possesses, therefore it is necessary that all involved stakeholders communicate, collaborate, and learn from each other. The process of arranging personalized adaptive learning experiences is a very complex one and usually people with different expertise have to collaborate to achieve a good quality solution. The complexity of this problem comes from the difficulty to formalize all the knowledge necessary in the pedagogical process. The authoring process can be simplified if at various levels of the application reusable components are constructed that can be assigned to the formal models mentioned earlier in this paper – learning objects, domain ontologies, pedagogical methods, and adaptation specifications. Following standards requires an increased initial investment, but has a higher potential for the future. 14 It has been observed that the more context is assigned to learning objects the lower is their reusability (Hodgins, 2005). The validity of this statement can be enhanced also for specifications of learning activities and adaptivity. As stated by R. Koper (Koper & Tattersall, 2005) “the notation must make it possible to identify, isolate, decontextualise and exchange useful parts of a learning design so as to stimulate their reuse in other contexts”. Therefore it would be beneficial to distinguish well-defined learning layers with clear interfaces, so that each object of a given layer can be substituted with other objects of the same layer and combined with other objects from different layers in order to build a comprehensive solution. There is a lack of support for adaptive behavior in existing learning standards that leads to higher costs and lower reusability of personalized learning solutions. B. Towle and M. Halm claim that IMS LD provides a way to implement simple adaptive learning strategies, but not complex forms of adaptive learning, like multiple rules interactions or enforced ordering (Koper & Tattersall, 2005). The aLFanet system (van Rosmalen et al., 2006) was built according to a standard-based model for adaptive e-learning. They have found out that learning standards are not harmonized to work with each other. Additionally, available tools are too complex for nonspecialized authors. It is necessary to improve usability and minimize complexity of the authoring tools. Another approach (Zarraonandia et al., 2006) has focused on reusability at the level of learning design. In this case an architecture is being developed that will automatically adapt units of learning to their actual context of execution via runtime interpretation of small adaptive actions that are specified separately from the Learning Design definition. In the WINDS project (Kravcik et al., 2004; Kravcik & Specht, 2004) authors without programming skills could produce adaptive courses by specifying declarative knowledge for adaptation by means of pedagogical metadata. This together with procedural knowledge encoded in the course player generated adaptive delivery of courses. A generalization of this approach (Kravcik, 2004) aimed at more flexibility, reusability and interoperability of partial learning resources via separation of different kinds of knowledge and their interaction, taking into account a typical learning design process as well as content object preferences for various learner profiles and contexts. A challenge is the creation and use of ontologies to represent various types of knowledge relevant for personalized adaptive learning (Knight et al., 2006). Such ontologies could be used by software agents to assist authors in the design of individualized learning or even to directly generate such experiences themselves. This article is aiming to map the current situation in the area of interoperability for adaptive learning components. We have focused on general aspects of semantic interoperability of adaptive systems, formal models and standards, as well as access to metadata, and have given examples of concrete tools, applications, and suggestions how to integrate different approaches. Interoperability demands can be recognized both at the horizontal level (between various systems) and at the vertical one (between formal models). In none of these two cases we can be satisfied with the existing solutions. There exist standard based solutions supporting interoperability of learning objects and learner models. Standardized learning design enables interoperability between systems, but is not reusable in general. Interoperability of domain ontologies is an open issue, for the context and adaptation models standards are still missing. We can state that in this field we are still far from achieving general interoperability, since the different standards are not enough to realize it and therefore a mediation based or Semantic Web based approach is still to be devised to reach reasonable results. This puts also the impressive looking list of standards and tools in the field in a realistic perspective. Acknowledgement This paper originates in the work done in the PROLEARN project – Network of Excellence in Professional Learning, which is an EU funded project #507310 in the IST programme. We would like to express our gratitude to our colleagues, especially to Stefano Ceri for his valuable internal review comments and to Bernd Simon for his support. References Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User-Adapted Interaction, 11 (1/2), 87-110. Brusilovsky, P. (2003). Developing adaptive educational hypermedia systems: From design models to authoring tools. In Murray. T., Blessing, S. & Ainsworth, S. (Eds.). Authoring tools for advanced technology learning environment. Dordrecht: Kluwer Academic Publishers, 377-409. 15 Christian, E. (1994). The Government Information Locator Service (GILS): Report to the Information Infrastructure Task Force. IITF Committee on Information Policy, retrieved March 16, 2006 from http://www.gils.net/gils.doc. CORDRA (2004). CORDRA: Technical Introduction and Overview, retrieved March, 16, 2006 from http://www.lsal.cmu.edu/lsal/expertise/projects/cordra/intro/intro-v1p00.html. CRA (2003). Grand Research Challenges in Information Systems, Washington, DC, USA: Computing Research Association. Cristea, A. I. & Aroyo, L. (2002). Adaptive Authoring of Adaptive Educational Hypermedia, In Proceedings of AH 2002, Adaptive Hypermedia and Adaptive Web-Based Systems, Springer, 122-132. Cristea, A. & Calvi, L. (2003). The three Layers of Adaptation Granularity. In Proceedings of the International Conference on User Modelling. Springer, 4-14. Cristea, A. & de Mooij, A. (2003). LAOS: Layered WWW AHS Authoring Model and their corresponding Algebraic Operators. In Proceedings of the WWW 2003 Conference, ACM, retrieved March 16, 2006 from http://www2003.org/cdrom/papers/alternate/P301/p301-cristea.pdf. De Bra, P., Houben, G. J. & Wu, H. (1999). AHAM: A Dexter-based Reference Model for Adaptive Hypermedia. In Proceedings of the ACM Conference on Hypertext and Hypermedia. ACM, 147-156. Dolog, P. (2004). Identifying relevant fragments of learner profile on the semantic web. In Proceedings of SWEL’2004 — Intl. Workshop on Semantic Web for eLearning, Intl. Semantic Web Conference 2004, Hiroshima, retrieved March 16, 2006 from http://www.l3s.de/~dolog/pub/swelatiswc2004.pdf. Dolog, P. & Schäfer, M. (2005). A Framework for Browsing, Manipulating and Maintaining Interoperable Learner Profiles. In Proceedings of UM’2005, Springer LNAI, 397-401. Fink, J., & Kobsa, A. (2002). User modeling in personalized city tours. Artificial Intelligence Review, 18 (1), 3374. Halasz, F., & Schwartz, M. (1994). The Dexter hypertext reference model. Communications of the ACM, 2, 3039. Hamel, G., & Välikangas, L. (2003). The Quest for Resilience. Harvard Business Review, 81 (9), 52–63. Hatala, M., Richards, G., Eap, T., & Willms, J. (2004). The Interoperability of Learning Object Repositories and Services: Standards, Implementations and Lessons Learned. In Proceedings of the 13th World Wide Web Conference. New York City, retrieved March 16, 2006 from http://www2004.org/proceedings/docs/2p19.pdf. Hodgins, W. (2005). Grand Challenges for Learning Objects, Learntec, Karlsruhe. IEEE PAPI (2003). IEEE P1484.2 Learner Model Working Group: PAPI Learner, Draft 7 Specification, IEEE Standards Committee on Learning Technology. IMS LIP (2001). IMS Global Learning Consortium: IMS Learner Information Package Specification v. 1.0, retrieved March 16, 2006 from http://www.imsglobal.org/profiles/lipinfo01.html. IMS LDIM (2003). IMS Global Learning Consortium: IMS Learning Design Information Model, retrieved March 16, 2006 from http://www.imsglobal.org/learningdesign/ldv1p0/imsld_infov1p0.html IMS DRI (2003). IMS Digital Repositories Interoperability - Core Functions Information Model, retrieved March 16, 2006 from http://www.imsglobal.org/digitalrepositories/driv1p0/imsdri_infov1p0.html IST (2004). Information Society Technologies - 2005-06 Work Programme, retrieved March 16, 2006 from ftp://ftp.cordis.lu/pub/ist/docs/ist_wp-2005-06_final_en.pdf. 16 Koper, R., & Tattersall, C. (2005). Learning Design. A Handbook on Modelling and Delivering Networked Education and Training, Springer. Knight, C., Gasevic, D., & Richards, G. (2006). An Ontology-Based Framework for Bridging Learning Design and Learning Content. Educational Technology & Society, 9 (1), 23-37. Kravcik, M. (2004). Specification of Adaptation Strategy by FOSP Method. In AH 2004: Workshop Proceedings, TU/e, 429-435. Kravcik, M., & Specht, M. (2004). Authoring Adaptive Courses – ALE Approach. Advanced Technology for Learning, 1 (4), 215-220. Kravcik, M., Specht, M., & Oppermann, R. (2004). Evaluation of WINDS Authoring Environment. In: De Bra, P. & Nejdl, W. (eds.) Proceedings of Adaptive Hypermedia and Adaptive Web-Based Systems, Springer, 166175. LOM (2002). IEEE Standard for Learning Object Metadata, IEEE. Moe, M., & Blodgett, H. (2000). The knowledge web, Merrill Lynch & Co., Global Securities Research & Economics Group, Global Fundamental Equity Research Department. Nejdl, W., Wolf, B., Qu, C., Decker, S., Sintek, M., Naeve, A., Nilsson, M., Palmer, M., & Risch, T. (2002). EDUTELLA: a P2P Networking Infrastructure based on RDF. In Proceedings of the 11th World Wide Web Conference, Hawaii, 604–615. OKI (2004). Open Knowledge Initiative, retrieved March 16, 2006 from http://web.mit.edu/oki/. OpenURL (2004). OpenURL, retrieved March 16, 2006 from http://library.caltech.edu/openurl/. QEL (2004). Edutella Query Exchange http://edutella.jxta.org/spec/qel.html. Language (QEL), retrieved March 16, 2006 from RDF (2005). RDF, retrieved March 16, 2006 from http://www.w3.org/RDF. Rehak, D., Dodds, P., & Lannom, L. (2005). A Model and Infrastructure for Federated Learning Content Repositories. In Proceedings of the WWW 2005 Conference, retrieved March 16, 2006 from http://cordra.net/information/publications/2005/www2005/cordrawww2005.pdf. Simon, B., Massart, D., van Assche, F., Ternier, S., Duval, E., Brantner, S., Olmedilla, D., & Miklos, Z. (2005). A Simple Query Interface for Interoperable Learning Repositories. In Proceedings of the WWW 2005 Conference, retrieved March 16, 2006 from http://nm.wu-wien.ac.at/e-learning/interoperability/www2005workshop-sqi-2005-04-14.pdf. SQI (2005). Simple Query Interface Specification, retrieved March 16, 2006 from http://www.prolearnproject.org/lori. SRU (2005). Search and Retrieve URL http://www.loc.gov/z3950/agency/zing/srw/sru.html. SRW (2004). ZING, Search/Retrieve Web http://lcweb.loc.gov/z3950/agency/zing/srw/. Service, Service (SRW), retrieved retrieved March February 16, 15, 2006 2006 from from van Rosmalen, P., Vogten, H., van Es, R., Passier, H., Poelmans, P., & Koper, R. (2006). Authoring a full life cycle model in standard-based, adaptive e-learning. Educational Technology & Society, 9 (1), 72-83. Verbert, K., & Duval, E. (2004). Towards a global architecture for learning objects: a comparative analysis of learning object content models. In Proceedings of the ED-MEDIA 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications, AACE, 202–209. 17 Wu, H., de Kort E., & De Bra, P. (2001). Design Issues for General-Purpose Adaptive Hypermedia Systems. In Proceedings of the ACM Conference on Hypertext and Hypermedia, 141-150. Zarraonandia, T., Dodero, J. M., & Fernandez C. (2006). Crosscutting runtime adaptations of LD execution. Educational Technology & Society, 9 (1), 123-137. ZING (2001). Z39.50: Part 1 - An Overview, retrieved March 16, 2006 from http://www.bibliotech.com/html/z39_50.html. 18 Kassahun, A., Beulens, A. & Hartog, R. (2006). Providing Author-Defined State Data Storage to Learning Objects, Educational Technology & Society 9 (2), 19-32. Providing Author-Defined State Data Storage to Learning Objects Ayalew Kassahun Wageningen Multimedia Research Center, Wageningen University Dreijenplein 2, 6703 HB Wageningen. The Netherlands Tel. +31 317 484723 Fax: +31 317 483158 Ayalew.Kassahun@wur.nl Adrie Beulens Information Technology Group, Wageningen University Dreijenplein 2, 6703 HB Wageningen. The Netherlands Tel. +31 317 488460 Fax: +31 317 483158 Adrie.Beulens@wur.nl Rob Hartog Wageningen Multimedia Research Center, Wageningen University Dreijenplein 2, 6703 HB Wageningen. The Netherlands. Tel. +31 317 483408 Fax: +31 317 483158 Rob.Hartog@wur.nl ABSTRACT Two major trends in eLearning are the shift from presentational towards activating learning objects and the shift from proprietary towards SCORM conformant delivery systems. In a large program on the design, development and use of digital learning material for food and biotechnology in higher education, a large amount of experience has been gained with regard to the possibilities of learning objects that induce students to be active. These learning objects are highly appreciated by both students and instructors. An important requirement for these learning objects is the need to support the storage, retrieval and sharing of state and history information that is defined by the author of the learning object. However, neither the current learning management systems nor the current SCORM standard provides adequate support for storing author-defined data. In this article, we discuss some of the problems related to the current learning management systems and the SCORM standard in supporting activating learning objects, and we propose a data model and a simple HTTP-based communication protocol as a solution to these problems. This article describes DBLink, a plugin that implements the proposed extensions. Keywords Learning Management System, Author-defined Data, State Information, SCORM, Activating Learning Objects. Introduction Currently, most institutions of higher education operate a learning management system (LMS). Learning management systems are used to control access to digital learning objects, to support communication between students and their peers as well as between students and their teachers and to support assessments. With respect to learning objects, there is a trend in eLearning not only from presentational learning material towards learning material that induces the student into actions and activities but also from proprietary to standards based authoring tools and delivery platforms. The use of learning objects that strongly induce students to perform actions is based on the general belief that learning, understanding and retention benefit from opportunities to become active with and to elaborate on presented information (Merriënboer, 1997; Biggs, 1999; Anderson, 2000). We call interactive learning objects that enable and induce actions and activities activating learning objects. Over the years, the Food and Biotechnology (FBT) program of Wageningen University has produced a number of activating learning objects, which have usually been implemented as Flash movies or Java applets. Moreover, these learning objects are often based on client-server architecture in which the learning objects rely on dedicated servers to store data in any format and structure. Most of these learning objects are also large learning objects that store the state of student’s interaction and even provide internal navigation. Both the required level of user interaction as well as the data requirements make it difficult to break up these large learning objects into ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 19 smaller ones. In this article, we will deal with such learning objects and restrict the term activating learning object to those activating learning objects that are based on client-server architecture. Author-defined data storage (Sessink et al., 2003) is the ability of the learning object (LO) to store and retrieve data that are defined by the author and are stored in some data store that is not a part of nor managed by means of the learning management system. Usually, the author of the learning object has defined the data structure of the data source and the instructor who uses the learning object in his course has to manage the data source. We call this type of data source external data store. Within the FBT program, we have developed activating LOs for activities such as interactive exercises in Food Chemistry (Diederen et al., 2002, Aegerter-Wilmsen et al., 2003, Diederen et al., 2003), experimental design (Aegerter-Wilmsen et al., 2003) and design of downstream processing (Schaaf et al., 2003). These interactive exercises and design teaching aids are large activating LOs of simulations or virtual laboratories in which students take part in a simulation or in an experiment. State information – the state of a student’s interaction with an LO – is stored so that students can do experiments that take longer than a single session. Stored state information also allows the instructor to monitor the progress of students in an instructor-led training. The ability to store and retrieve author-defined data enables content authors and instructors to support activating pedagogical models that need functionalities such as adaptivity, collaborative learning and retrieval of state and shared data (Sessink et al. 2003). Adaptivity refers to a learning model that takes into account the learner’s competence, goals, and preferences. State information refers to the ‘state’ of the learning experience at a given time and is essentially based on tracking of a student’s progress. By shared data, we mean data that can be accessed by more than one learning object or by more than one student. Shared data are especially important in collaborative learning. In the Hygienic design LO (see section 4), for instance, the learning object stores state history and the highest scores of all students. Students can compare their current results with their results from previous attempts. A number of high scores from all students taking the LO are flagged as shared data so that students can compare their results with the highest scores and be encouraged to perform better on the next trial. Due to the lack of standards for supporting author-defined data in the current learning management systems, different ad hoc methods have been used for storing data in external data stores. One of the widely used methods to incorporate a large activating learning object in a learning management system is to host the actual learning object on a separate server and use a proxy learning object that is managed by the LMS. The proxy learning object contains a hyperlink to the actual learning object and when the student launches such a learning object, the proxy learning object forwards him to an external site. To actually use the learning object, the student is then either asked to authenticate himself using password authentication or he is automatically logged in using single sign on. However, the data source on which the actual learning object is based on can only allow or deny access to data based on the validity of the user and not the context in which the user interacts with the LMS. Another method to incorporate an activating learning object is to host the learning object partially in the LMS and store the data on a separate server. However, the same problem that was mentioned above still remains to be solved. These configurations do not enable LO authors to benefit from the facilities of LMSs in managing access to LOs. This shortcoming is unlikely to change in the near future since the recent Sharable Content Object Reference Model (SCORM) (ADL, 2004) does not have sufficient support for author-defined data storage. In this article, we discuss the problems related to the current learning management systems and the SCORM standard in supporting activating learning objects. We propose a number of data model elements as an extension to the SCORM 2004 data model. We describe how LOs can access data from an external data source through LMSs based on DBLink, an abstract plugin. A prototype implementation of DBLink has been built for the Blackboard LMS. For supporting non-SCORM LMSs, we define an HTTP based communication protocol to be used instead of the SCORM run-time API. However, we strongly suggest that the SCORM run-time API should be used over other non-standard methods. Some of the other issues related to the SCORM standard are lack of support for sharing data across learning objects, learners and LMSs; difficulty in migrating large learning objects to SCORM LMSs; scalability issues related to data size restrictions; and lack of support for instructor-led training. Although the scope of this article is restricted to the issue of supporting activating learning objects, some of these other issues are discussed in relation to activating learning objects. The rest of the article is organised into four sections. The next section, section two, discusses the requirements for DBLink for both SCORM as well as non-SCORM learning management systems. Section three covers design considerations for DBLink and explains the DBLink data model as an extension to the SCORM data 20 model; it also defines a communication protocol based on HTTP. Section four presents a use case. In the last section, section 5, concluding remarks are made. Requirements for DBLink The main requirement for DBLink is to support author-defined data storage so that authors of learning objects will be able to support activating learning objects. This requirement has a number of consequences for LMSs, LOs and for the role instructors and LMS administrators play in deploying learning objects. Learning objects that are student activating rely on the ability of storing and retrieving state and history information. Support for data storage for learning objects is very limited or non-existent in most of the current learning management systems. Some of the most widely used LMSs do provide application programming interfaces (API’s) that enable users to provide the necessary functionalities by extending the learning management system. However, without a common data model to support student activation, authors of activating LOs need to develop their own plugins for each LMS on which they plan to deploy their LOs. By providing a standard data model and communication protocol, DBLink should avoid the need for developing plugins by individual authors. In the case of SCORM 2004 conformant LMSs, the SCORM data model provides elements such as cmi.launch_data and cmi.suspend_data, for storing state information, but the use of these elements is very limited for a number of reasons. Among the most important reasons for these limitations are that data cannot be shared among LOs or among students, and the size of data that can be stored is severely limited. The IMS Shareable State Persistence specification (SSP) is proposed to solve these problems; however, it still relies on the LMS for data storage (IMS, 2004). The applicability of SSP is also limited to those LMSs that are SCORM conformant. DBLink should be a lightweight component that does not require conformance to the SCORM standard. However, we suggest that the SCORM run-time API should be used when both the LO and the LMS are already SCORM conformant. Activating LOs can generate large amounts of data; therefore, it is highly desirable to setup a separate server for data storage and to access it through the LMS. In addition, many existing activating LOs are mostly web-based applications and have their own data storage. The new approach should not require a substantial reengineering of already existing LOs. As a result, the way the LOs communicate with the server needs to be supported by DBLink and data storage requirements should not be restricted. DBLink should also support LOs that are SCORM conformant and use the SCORM Runtime API. In addition, because some activating LOs are meant for instructor-led training (ILT), DBLink needs to support ILT by providing data model elements for the most common instructor activities. The requirement that data needs to be stored externally affects the task of the LMS administrator or the instructor in the case of ILT. In addition to deploying learning objects, the instructor or LMS administrator also need to set up data sources. They do this by setting up a data source for each individual activating learning object or by setting up data source at a higher level of aggregation such as a module or course. In many cases, it is sufficient to setup the location of the data source and some access credentials. DBLink should provide data model elements for this purpose. DBLink tries to solve the above problems by providing an extension to the SCORM data model and HTTP based communication protocol. The purpose of this article is to propose a method for supporting accessing and persisting author-defined data that may be considered in future standards. Design and implementation The following sequence of events clarifies the essence of the approach that is widely used at present to host the activating LOs in current LMSs. 1. 2. 3. 4. 5. The student starts an activating learning object. The learning object locates the address of the data source that is included in the learning object. The student authenticates to the data source using either password authentication or single sign-on. The student begins interacting with the learning object. The learning object interacts with data source to retrieve or store data. 21 The data source is an external database or a web server that is usually specifically set up to serve one or more activating learning objects. Due to a lack of support for author-defined data by most of the current LMSs, authors and instructors devise workarounds to host their activating LOs that use external data along the lines of the approach outlined above. Figure 1 shows the schematic representation of the approach. For ease of clarity in this and subsequent figures, terms from the SCORM standard such as SCO and assets are used whether the LMS depicted is SCORM-based or not. There are a few problems associated with this scenario. First, over the years, institutions have made large investments and created several activating learning objects that are generally not designed for modern LMSs. These learning objects are usually large, student activating LOs based on a client-server model. Workarounds for hosting these learning objects in current LMSs can be implemented as outlined above. However, a student who accesses these learning objects via a LMS may still need to authenticate on the external data source. Moreover, in the absence of a standard data model to access data and a standard communication protocol or runtime API, instructional designers should implement custom made solutions. LM S User(Browser) SCO Asset Author-defined data server Author-defined data Figure 1. In the present LMSs, activating LOs access author-defined data directly instead of through the LMS. Students should usually authenticate themselves on the data server. The DBLink plugin The following is the essence of the approach proposed in this article. 1. 2. 3. 4. The student starts an activating learning object. The learning object interacts with the LMS requesting the address of DBLink. The student begins interacting with the learning object. The learning object requests data from DBLink. DBLink seamlessly interacts with the LMS to determine the student access rights and accesses data source on behalf of the student. We propose an extension to the SCORM data model that should be implemented in DBLink. In LOs for SCORM conformant LMSs, the values of the data model elements can be set and get using the SCORM runtime API. In LOs for non-SCORM LMSs, the data model elements will still be used. Since non-SCORM LMSs do not have a (standard) runtime API, we propose a simple HTTP based protocol to access the values of the data model elements. Activating learning objects built for both SCORM-based and non-SCORM LMSs should first try to use SCORM API. Only upon failure to use SCORM runtime API should the learning object revert to the DBLink communication protocol. By adopting HTTP based communication for non-SCORM systems as a means of communication between the learning object and DBLink, we not only use the most common communication protocol of accessing data over the web but also avoid a substantial reengineering of legacy activating learning objects. Most legacy student activating learning objects are web-based and use HTTP protocol. Figure 2 shows a LMS that uses DBLink to access author-defined data. The solid lines (lines 1a, 2a) represent how activating LOs communicate with the LMS server in a non-SCORM LMS. The dotted lines (lines 1b, 2b) represent how an SCO uses the facilities of DBLink. Line 3 shows how DBLink accesses the data source, which is the same in both SCORM and non-SCORM LMSs. 22 LMS Browser 2b DB Link Service 2a API Adapter 1a SCO/LO 1b Asset 3 Author-defined data storage Author-defined data Figure 2. Proposed way of providing access to author-defined data. DBLink seamlessly communicates with the learning management system to determine the student access rights and retrieve data on the student’s behalf. Figure 3 depicts the second approach as a UML sequence diagram (not to be confused with IMS sequencing). The figure shows how the data source for an activating learning object is configured and how the learning object ultimately uses DBLink. First, the instructor or the LMS administrator deploys a course or a specific learning object. In many cases, instructors are responsible for preparing their learning material, but the actual deployment can be done either by the instructor himself or by the LMS administrator. Data source connection parameters are then configured. Data source configuration involves mainly specifying the location of data source and authentication credentials. The authentication credentials are used by DBLink to connect to the data source. Students do not need to authenticate on the data source separately since access to data is controlled by DBLink based on the student’s access rights as specified in the LMS. Once the data source configuration has been set up, the data source is accessible to students who are authorised for the specific learning material for which the data source is configured. The learning material can be the learning object, the whole course or any other aggregation level used by the LMS. When the student interacts with the learning object, the learning object requests data from DBLink (through the LMS), DBLink checks with the LMS if the student is authorised to use the learning object and subsequently serves the requested data. How DBLink and the LMS work together to authenticate the student for accessing data is dependent on the implementation of the specific DBLink and LMS and thus cannot be specified here. To implement the DBLink plugin, the LMS needs to make an API available to access learners’ information, to access LOs information, and to check if the user is authorised to use the specific learning object. The LMS should have an instructor user type so that instructor type users can manage the external data. In cases where this is not possible, the LMS administrator user type can be associated with DBLink instructor user type. The data model The data model elements provided fall into two groups: data elements for the purpose of accessing data and data elements for data source configuration. Data elements to read and modify data are used by both students and LMS Administrators. In the case of ILT, instructors have the same data access rights as LMS administrators. However, instructors can only access those learning materials for which they are responsible, while LMS administrators can access all learning materials. Data source configuration and the associated data elements are required because the data is stored on a separate server and the location of the data source and access credentials are unknown before deployment of the learning object. Therefore, the data source has to be configured by the instructor or the administrator after the learning objects that make use of the data source have been deployed. DBLink defines a very limited set of data types that provide a basic functionality sufficient for most purposes. These data elements are not dependent on other SCORM data elements and there are no requirements on the type of data fields of the data source. In order to demonstrate our approach, we chose data sources to be SQL database applications (in contrast to an arbitrary type of application). Furthermore, we assumed that the parameterised SQL statements (in contrast to more general methods such as web-services) are, for the most part, sufficient as a way of accessing data – thus, the name DBLink (Database Link). 23 Instructor or LMS Administrator (Browser) Student (Browser) LMS DBLink Data Store Add student-activating learning object Set up author-defined data Student activating learning object requests data DBLink interacts with the LMS to determine student authorisation Request for data access or to store data Return data DBLink services author-defined data Figure 3. A UML sequence diagram showing the interactions among the LMS, the activating learning object (student), DBLink and the instructor. Data source configuration can be done either by the LMS administrator or the instructor. The elements of the data model shown in Table 1 and Table 2 resemble, in format, the data model elements of SCORM; therefore, we will explain only those elements that are new and will not discuss items such as _count or id. Table 1 Data model elements for commands Details Number of commands Student and LMS administrators/instructors can get this value Id of the nth command LMS administrators/instructors can get and set this value Student can get this value dblink.author_defined_data. The command string (SQL statement) of the nth command commands.n.command LMS administrators/instructors can get and set this value dblink.author_defined_data. The result of executing the nth command. commands.n.value Student and LMS administrators/instructors can get this value dblink.author_defined_data. Number of parameters for the nth command commands.n.parameters._count Student and LMS administrators/instructors can get this value dblink.author_defined_data. Id of the nth parameter. commands.n.parameters.n.id Student and LMS administrators/instructors can get this value dblink.author_defined_data. The value of the nth parameter. commands.n.parameters.n.value Student and LMS administrators/instructors can get and set this value Dot-Notation Binding dblink.author_defined_data. commands._count dblink.author_defined_data. commands.n.id Table 1 shows the data elements required to execute commands for the purpose of accessing data. The dblink.author_defined_data.commands.n.command data model element refers to the command used to access data from a database. In DBLink access to data is restricted to SQL databases; thus, the commands are expressed as SQL statements. Since some of the values used in SQL statements can only be determined at runtime, unknown values at the time of LO creation are represented as parameters. We call these SQL statements parameterised SQL statements. At runtime, the parameter values are substituted by the values supplied by the student or the LMS. The dblink.author_defined_data.commands.n.value data model element represents the results of executing the command represented by dblink.author_defined_data.commands.n.command. The dblink.author_defined_data.commands.n.command.n.parameters.n.value data model element defines the value of the nth parameter. The values are set by the learning object before executing the corresponding command. 24 In the parameterized SQL statement shown in Figure 4 the parameters (shown in italics) are preceded by a question mark. At runtime, the parameters (in the example the parameters are colParam and keyParam) are substituted by actual values supplied by the student or the LMS. In non-SCORM LMS, The learning object supplies the values to these parameters as HTTP parameter-value pairs using either the GET or POST method. In SCORM conformant LMS, all parameters should be set by the learning object before a getValue method call on dblink.author_defined_data.commands.n.value. UPDATE aTable SET colName=?colParam WHERE primaryKey=?keyParam; Figure 4. A parameterized SQL Command. Data source configuration The method we propose assumes that the data source, which henceforth will be known as database, is not necessarily one central database for the LMS that stores data for all LOs. Therefore, instructors or LMS administrators should be able to set-up the database connection information either for the LMS as a whole, for each course or for each learning object independently. Database configuration consists of specifying the address of the database server and any required credentials for authentication on the database. Table 2 shows the data elements defined for data source configuration. The address (URL and port number) of the database server, the username, the password and optionally a database instance name are usually sufficient to specify database connection information. We, therefore, limit the data elements for configuration to these four elements. Dot-Notation Binding dblink.dblink_address dblink.author_defined_data. server_address dblink.author_defined_data. database dblink.author_defined_data. username dblink.author_defined_data. password Table 2. Data elements for database configuration Details The URL DBLink. Students and LMS administrators/instructors can get this value The URL of the author-defined data store Only LMS administrators/instructors can get and set this value Database used as author-defined data store Only LMS administrators/instructors can get and set this value User name to be used to logon to the database Only LMS administrators/instructors can get and set this value Password to be used to logon to the database Only LMS administrators/instructors can set this value Communication protocol The learning object may interact with DBLink using HTTP protocol. DBLink depends on the LMS to forward commands that are destined for DBLink. For instance, if the learning management system is running on http://www.the-lms.com/, then all calls to http://www.the-lms.com/<dblink> will be forwarded to DBLink. Here <dblink> refers to the relative path to which all calls are forwarded to DBLink by the LMS. Figure 5 shows the protocol used in DB Link. In this figure, variables are shown in italics and need to be substituted with actual values. The URL of the DBLink is the value of dblink_base_url. The command handler in DBLink that processes the HTTP request is represented by command_handler. One of the command handlers is executesql. This handler processes SQL data storage and retrieval from databases. Another command handler is getuserdata, which retrieves user information. The value of content_id is the id of the LO or course for which the database is configured. The values of parameter_id and parameter_value are the optional ID and value of the parameters. Table 3 gives a full description of the protocol elements. 25 dblink_base_url/[command_handler]? [content_id=content_id] [command_id=command_id] [&parameter_id=parameter_value] Figure 5. Communication protocol between learning objects and DBLink. Variable dblink_url content_id command_id parameter_id parameter_value Table 3. Communication protocol parameters Details The URL of DBLink. This is a value returned by getValue(dblink. dblink_address) in a SCORM conformant system. The unique id of the learning object. The id of the command. The learning object should check by calling getValue(dblink.author_defined_data. commands. id) in a SCORM conformant system to make sure that this parameter id is set. The id of the parameter. The learning object should check by calling getValue(dblink.author_defined_data. commands.n.parameters.id) in a SCORM conformant system to make sure that this parameter id is set. The parameter value to be set SCO. A use case Three different implementations of DBLink are satisfactorily in use: one for the Blackboard LMS (Blackboard, 2006); one for the Moodle LMS (Moodle, 2006); and one for the TopShare knowledge management system (TopShare, 2006). In this section, we present a use case based on a LO for a virtual ice-cream factory and DBLink implementation for the Blackboard LMS. The learning object is used in the course Hygienic Design at Wageningen University. A learning object for a virtual ice-cream factory During MSc courses in Food Technology and Food Safety, students learn about hygienic design. For this course, a learning object depicting a virtual factory for the production of ice cream has been built ( Figure 6). In this virtual production facility, the student, who plays the role of a quality manager, enters the factory and has to scrutinise production and storage facilities with respect to hygienic design criteria. Upon entering the virtual factory, the student is personally greeted by and introduced to a virtual employee of the factory. Next the overview of the factory is displayed, which the student can walk through asking critical questions ( Figure 6-1). To proceed through the factory, the student has to click on the individual factory sections and identify items that need to be inspected with respect to hygienic risks (Figure 6-2 and Figure 6-3). When the student interacts with the learning object, such as entering the virtual factory or identifying an item for inspection, the learning object executes a command and provides the necessary variables and variable values. A typical command and the corresponding SQL statement are shown in Table 4. This command gets the location of the student in the learning object. The learning object launches the command, and DBlink translates the command into a SQL statement. Note that in the example shown in Table 4, the values of content_id and user_id have either been previously obtained or have been specified using special place holder variables which DBLink can expand at run time. After the student has answered a number of questions about a specific component of the production facility, he receives feedback on all the questions he answered (Figure 6-4). When the student has finally completed the walk through of the factory, he is presented with his score and the five top scores of all participants in the exercise. In this way, he can compare his own score with those of other students (Figure 6-5). The described functionalities require the learning object to retrieve student information, store selections made by the student, and retrieve the history of student selections to proceed to the next step or to compute the score. To 26 retrieve student user information, which is used throughout the learning object, the learning object uses getuserdata command. In subsequent steps, i.e. Figure 6-1 through Figure 6-5, the learning object launches executesql command to retrieve or store data in a database. Table 5 shows a list of commands used by the ice cream virtual production facility. Table 4. An example of parameterized SQL statement and the corresponding DBLink command. SELECT attempt, allowed_attempts, location FROM SQL (ID=10) command HD_Sate_Table case WHERE UserID=?userId http://<dblink_address>/executesql?content_id=1000& command=10&userId=james Table 5. DBLink URL’s for retrieving student and state information and to store state information. Retrieve student information http://<dblink_address>/getuserdata http://<dblink_address>/executesql?content_id= Retrieve state information Store state information <content_id>&command=<command_id> http://<dblink_address>/executesql?content_id= <content_id>&command=<command_id>?[<parameter_id>= <parameter_value>] Figure 6-1. The student who plays the role of Quality Manager is presented with an overview of the virtual ice cream factory. The student clicks on a section of the factory to perform quality inspection. 27 Figure 6-2. The student has to identify items that need to be inspected. Figure 6-3. If the student identifies an item that needs inspection, the LO presents a predefined question about the item. 28 Figure 6-4. Once the student completes the inspection of a section of the facility, he receives feedback on his performance. Figure 6-5. Upon completion of the exercise, the student’s score and the five top scores in the exercise are presented to the student. Figure 6. A virtual ice cream production facility. 29 DBLink plugin for Blackboard Blackboard is one of the most widely used LMS that manages content and learning processes (Blackboard, 2006). Blackboard is SCORM conformant. It also supports integration of external software tools through its extended “building block” API. We used the building block API to develop a DBLink prototype for the Blackboard LMS (Figure 7). The SCORM component in Blackboard is implemented as a building block and its functionalities are not accessible through the building block API. We therefore use the DBLink HTTP based communication protocol to access the data model elements. DBLink for the Blackboard LMS is built as a course tool. Instructors have access to a configuration interface, provided by DBLink in the course control panel. Through the configuration interface, instructors can specify the values of database connection and command data model elements. To use DBLink, the instructor uploads the learning object and subsequently uploads or configures the database connection parameters and commands. These values can also be set at course or at LMS level in which case there will be no need for the instructor to enter the values of the data model elements for individual LOs. Figure 7. Database configuration of DBLink in Blackboard LMS. Conclusion This article is concerned with providing a way of hosting student activating learning objects in current and future (SCORM-based) learning management systems. An important requirement for activating learning objects is the ability to read and write data to and from an external data source. Using an external data source makes it possible to store arbitrary data types with virtually no size limitation. However, both the current learning management systems and the SCORM standard (ADL, 2004) do not provide functionalities or specifications that enable us to achieve our goals. We proposed an abstract plugin that enables us to access data from an external data source and an extension to the SCORM data model to be used by this abstract plugin. To access data for LMSs that are not based on SCORM, we proposed a communication protocol that the learning object can use to communicate with the 30 plugin based on HTTP. Our method works irrespective of the LMS and implementation of the plugin as long as the DBLink data elements can be set and get with SCORM runtime API. The same applies to non-SCORM LMS; however, in this case, the HTTP protocol we proposed is used instead of SCORM API. We have implemented DBLink for Blackboard, Moodle and TopShare. The data model has two parts: data model elements for accessing external data, which are usually used by students, and data model elements for configuring the data source. Data model elements for data source configuration allow the instructor, or the LMS administrator, to setup the data source and specify the data access commands and parameters. To fully exploit the possibilities of data sharing provided by DBLink, an extended set of metadata about the data stored is required. Developing such a metadata specification is a large task in itself and does not fit in the scope of this article. We described the functionality and the interface based on a plugin called DBLink for accessing database tables that is implemented for the Blackboard learning management system. The purpose of DBLink is, on the one hand, to bring a standardized solution for hosting a number of disparate learning objects developed in the FBT program of Wageningen University (FBT, 2006) and the European Nutrigenomics Organisation (Nugo, 2006). On the other hand, DBLink also shows how the abstract plugin can be implemented. Therefore, DBLink is limited to accessing data from databases using parameterised SQL statements and does not implement all the requirements stated in section 2. Another consideration in the design and implementation of DBLink is to support a step-by-step migration of legacy activating learning objects that are not made to be hosted in any LMS or are made to be hosted only for a specific proprietary LMS. We support that by providing a HTTP based communication protocol, besides the SCORM runtime API, since most learning objects already use HTTP to access data over the web. We believe the communication protocol can easily be extended to support learning objects that rely on web services protocol. We present the method provided in this article as a concept for consideration in future updates of SCORM. Acknowledgements The ice cream factory was developed under the auspices of the European Chair in Food Safety Microbiology by Esther van Asselt, Marc Boncz, Martine Reij, and Leon Gorris, Laboratory of Food Microbiology, Wageningen University. References Aegerter-Wilmsen, T., Hartog, R., & Bisseling, T. (2003). Web Based Learning Support for Experimental Design in Molecular Biology: a Top-Down Approach. Journal of Interactive Learning Research, 14 (3), 301314. ADL (2004), Sharable Content Object Reference Model (SCORM) version 1.3, retrieved March, 21 2006 from http://www.adlnet.org. Anderson, J. R. (2000). Cognitive Psychology and Its Implications, USA: Worth Publishers. Biggs, J. B. (1999). Teaching for quality learning at university, Buckingham: Open University Press. Blackboard. (2006). Blackboard http://www.blackboard.com. learning management system, retrieved March, 21 2006 from Diederen, J.; Gruppen, H.; Hartog, R.; Moerland, G., & Voragen A. G. J. (2003). Design of activating digital learning material for food chemistry education. Chemistry Education: Research and Practice, 4 (3), 353-371. Diederen, J., Gruppen, H., Voragen, A. G. J., Hartog, R., Mulder, M., & Biemans, H. (2002). Design guidelines for digital learning material for food chemistry education. In Barker, P. & Rebelsky, S. (Eds.), Proceedings EdMedia 2002, Denver, Colorado, USA: AACE, 402-407. FBT (2006). Food and Biotechnology Program, retrieved March, 21, 2006 from http://fbt.wur.nl. 31 IMS. (2004). IMS Shareable State Persistence Information Model. Version 1.0 Final Specification, retrieved March, 21, 2006 from http://www.imsglobal.org. Merriënboer, J. J. G. (1997). Training Complex Cognitive Skills: A four-component instructional design model for technical training, Englewood Cliffs, NJ, USA: Educational Technology Publications. Moodle (2006). Moodle course management system (CMS), retrieved March 21, 2006 from http://moodle.org/. NuGO (2006). The European Nutrigenomics Organisation, retrieved March 21, 2006 from http://www.nugo.org. Sessink, O., Beeftink, R., Tramper, J. & Hartog, R. 2003. Author-Defined Storage in the Next Generation Learning Management Systems. In V. Devedzic, J. M. Spector, D. G. Sampson & Kinshuk (Eds.), The 3rd IEEE International Conference on Advanced Learning Technologies Conference Proceedings, Los Alamitos, USA: IEEE Computer Society, 57-61. Schaaf, H. V. D., Vermue, M. Tramper, J., & Hartog, R. (2003). A design environment for downstream processes for bioprocess-engineering students. European Journal of Engineering Education, 28 (4), 507-521. TopShare (2006). TopShare http://www.topshare.com. Knowledge Management Solution, retrieved March 21, 2006 from 32 Di Nitto, E., Mainetti, L., Monga, M., Sbattella, L. & Tedesco, R. (2006). Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus Approach. Educational Technology & Society, 9 (2), 33-50. Supporting Interoperability and Reusability of Learning Objects: The Virtual Campus Approach Elisabetta Di Nitto Politecnico di Milano - DEI, via Ponzio 34/5, I-20133 Milano, Italy dinitto@elet.polimi.it Luca Mainetti Università degli Studi di Lecce - DII, via per Monteroni, I-73100, Lecce, Italy luca.mainetti@unile.it Mattia Monga Università degli Studi di Milano - DICo, via Comelico 39/41, I-20135 Milano, Italy mattia.monga@unimi.it Licia Sbattella and Roberto Tedesco Politecnico di Milano - DEI, via Ponzio 34/5, I-20133 Milano, Italy sbattell@elet.polimi.it tedesco@elet.polimi.it Abstract E-learning has the potential to offer significant advantages over traditional classroom learning. However, it requires a complete redefinition of the dynamics of interaction between the various actors of a classroom. Moreover, in this context, the authoring of instructional material requires much more time than in traditional learning. Therefore, special care has to be posed to the definition of proper authoring approaches where educators can reuse and easily assemble existing materials. In this scenario, a comprehensive learning platform addressing the various interrelated aspects of authoring and fruition of instructional material is needed. Such a platform should enable reusability of materials, so that it is possible to make efficient use of preexisting experiences, and interoperability with existing platforms so that it is possible to take advantage of their strengths. The SCORM standard offers, among other features, a rich data model that can be used to define and share Learning Objects through different e-learning platforms. We argue that, despite the fact that it is the most emerging and promising standard, SCORM does not address some key issues properly, such as specification of metadata and LO composition. In this paper we focus on these issues and propose some extensions to SCORM that aim to address the above issues. Keywords SCORM, Learning Object, Metadata, Aggregation, Sequencing. Introduction Pervasiveness of computing in modern societies enables computer-supported learning tools to be available both in the classroom, and at home. In the classroom, such learning tools facilitate specific – often collaborative – activities, while at home they support students in self-study, in taking part in virtual classes, etc. As pointed out in (Dongsong et al. 2004), e-learning has the potential to deliver significant advantages with respect to traditional classroom learning. As a drawback, by exploiting e-learning, the natural dynamic of interaction between the various actors in a real classroom cannot be recreated completely and new dynamics have to be enabled and driven. Moreover, the preparation of instructional material requires much more time than in traditional learning, therefore, a special care has to be posed to the definition of proper authoring approaches where educators can reuse and easily assemble existing materials. In this scenario, a comprehensive learning platform addressing the various complex and interrelated aspects of authoring and fruition of instructional material is needed. Such a platform should address the needs of three main classes of actors: Authors, Teachers, and Learners. Authors design and build courses, possibly modifying and composing LOs; Teachers enact and manage courses, exploiting available LOs; finally, Learners attend courses by consuming LOs, possibly with the supervision of Teachers. Moreover, the platform should enable reusability of materials, so that it is possible to make efficient ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 33 use of preexisting experiences, and interoperability with existing platforms, so that it is possible to take advantage of the functionality they offer. In general, there are at least two possible strategies that can be adopted and combined for interoperability between heterogeneous systems: one interface-oriented and the other model-oriented. The former refers to the idea of defining well-known interfaces that systems should expose. By exploiting such interfaces, systems can call each other services thus enabling exchange of data, execution of queries on remote data, etc. The Simple Query Interface (SQI) (Simon et al., 2004) is an example of such an interface for the e-learning domain. The model-oriented strategy refers to the idea of having standard, semantically-rich data models shared by various systems. Sharing the same data model enables the possibility of reusing the same data in different systems, dramatically increasing the interoperability among them. Within the context of e-learning, the SCORM standard (ADL, 2004a) supports both strategies. Restricting our analysis to the model-oriented aspect (which is the focus of our paper), SCORM provides a rich data model that can be used to define and share Learning Objects (LOs). LOs are defined not only in terms of their content, but also in terms of the technological platform needed to exploit them and in terms of the way they need to be sequenced for the purpose of a particular course. While experimenting with this standard in our Virtual Campus project (Cesarini et al., 2004), we have realized that, despite the fact that it is the most emerging and promising standard, SCORM does not properly address some key issues, which concern the specification of metadata describing LOs, and the composition of LOs. As we discuss in this paper, such issues directly affect the possibility to effectively perform two main activities related to reuse of instructional materials both within the same e-learning system and, even more, across different systems: reuse for authoring and reuse for teaching. Reuse for authoring, during creation of instructional materials, requires a data model for LOs that is rich enough to support reuse of parts (independently of their level of granularity), creation of LOs that reassemble existing ones, modification of the workflow that defines the way LOs will be executed (sequencing, in the SCORM terminology). Reuse for teaching, at the time a course is given, requires mechanisms and metadata that simplify the publication of the LOs and their enactment. In this paper, based on the experiences we gained within the Virtual Campus project, we propose some extensions to SCORM aiming at supporting reuse for authoring by empowering the mechanisms for LO composition and reuse for teaching through the definition of proper metadata for LOs. The paper is structured as follows. In Section 2 we present the SCORM approach and highlight its advantages and disadvantages. In Section 3 we present the core contribution of this paper, that is, our extensions to SCORM. In section 4 we present the high level architecture of the Virtual Campus runtime platform. In Section 5 we evaluate our approach, comparing it against SCORM through an example. Finally, in Section 6 we present some related work, and in Section 7 we draw some conclusions. The SCORM Approach: Pros and Cons SCORM (Sharable Content Object Reference Model) has been defined by the Advanced Distributed Learning initiative established in 1997 by the US DoD. The aim was to develop an overall standardization strategy to modernize education and training, and to promote cooperation between government, academia and business. To address these goals, the initiative has defined high-level requirements for learning contents, such as content reusability, accessibility, durability, and interoperability, and has defined three main standards aiming at addressing the aforementioned requirements: The Content Aggregation Model, the Run-Time Environment, and the Sequencing and Navigation. The Content Aggregation Model (ADL, 2004b) contains guidance for identifying and aggregating resources into structured learning contents. The model is based on the IMS Content Packaging Information Model (IMS, 2004). It is based on three components Assets, Sharable Content Objects (SCOs), and Content Organizations that are enacted by means of the Run-Time Environment (see next point). Assets are electronic representations of media, text, images, sound, web pages, assessment objects or other pieces of data that can be delivered to a Web client. Assets can be grouped to produce complex Assets. A SCO is a collection of one or more Assets and represents a single executable learning resource. Since SCOs represent the lowest level of granularity of a learning resource that communicates with the Run-Time Environment, they can be launched by the Learner, while Assets cannot. Finally, a Content Organization is a tree composed of so-called Activity items, which can be mapped on SCOs or Assets (see Figure 1, extracted from (ADL, 2004b)). All of these components can be tagged with metadata. As a metadata standard, SCORM defines a variation of the IEEE Learning Object Model (LOM) (IEEE LTSC, 2002). 34 LOM envelops instructional material in metadata that describe it. Examples of metadata are the Language of the instructional material, the Description associated to the instructional material, etc. LOM defines a Learning Object (LO) as “any entity -digital or non-digital- that may be used for learning, education or training”. Figure 1 - SCORM Content Organization The Run-Time Environment (ADL, 2004c) describes a content object launch mechanism, a communication mechanism between content objects and the SCORM engine on the server, as well as a data model for tracking Learners’ experience with content objects. It is derived from the run-time environment functionality defined in (AICC, 2004) CMI001 Guidelines for Interoperability. The Sequencing and Navigation (ADL, 2004d) describes how SCORM-conformant contents may be sequenced through a set of Learner-initiated or system-initiated navigation events. It allows the Author to define a path through Activities. This path is used to guide the Learner in the way she/he takes the instructional material. Sequencing and Navigation is based on the IMS Simple Sequencing standard (IMS, 2005). It introduces new structures to aggregate Activities. An Activity Tree describes the branching and flow of Activities. Notice that the Activity Tree coexists with the Content Organization structure, as they represent two different "views" on the same objects. As in the Content Organization structure, Activities in the Activity Tree need to be mapped on the previously defined SCOs and Assets. Within the tree, a single parent Activity and its first-level children are considered as a new entity, called Cluster (see Figure 2, extracted from (ADL, 2004d)). Each Cluster has associated a Sequencing Definition Model (SDM) to define sequencing behaviors of its first-level children. SDM permits to define sequencing behavior in several ways. In particular, Sequencing Control Modes define the instructional path in terms of workflow-like statements; Sequencing Rules represent a set of conditions that are evaluated in the context of the Activities for which they are defined; Rollup Rules define how to evaluate tracking information collected during the fruition; Objectives define how to evaluate Activities’ progress information, etc. Figure 2 – Simple Sequencing Activity Tree and Clusters SCORM can certainly be considered a step towards the establishment of a comprehensive framework for the definition and execution of LOs. If it will be actually accepted by the various vendors working in the area, interoperability and reuse of existing assets will be dramatically enhanced. However, while experimenting with it we have identified some weaknesses that are summarized in the following of this section. 35 The Content Aggregation Model defines components (i.e., SCOs and Assets) that seem to be slightly different implementations of the same concept (the Learning Object) under different reusability properties and limitations. As SCOs and Assets are both tagged with metadata and can be composed by means of the Content Organization tree, the only difference seems to be how they are handled at runtime. Only SCOs are able to fully exploit the functionalities provided by the Run-Time Environment in order, for example, to keep track of the interaction of Learners with the system. The Content Organization tree permits to aggregate SCOs and Assets. However, it actually aggregates Activity items, which, in turn, need to be mapped onto previously defined SCOs and Assets. This choice maximizes the flexibility of the standard but leads to a very complicated and redundant definition, as Authors need to define Assets and SCOs, build the Content Organization tree based on Activities, and then provide a mapping scheme. The Sequencing and Navigation specification further complicates the situation, as it adds new aggregation structures - Activity Trees and Clusters - which seem to partially replicate the Content Organization definition. The aim of allowing their coexistence is to separate simple aggregation (provided by the Content Aggregation Model) from sequencing (defined via Sequencing and Navigation). The drawback is that the overall specification becomes very complex and odd, since several concepts overlap and do not seem to be clearly defined. Sequencing and Navigation provides several ways to define a learning path. This is interesting as it enables the definition of very complex learning paths. However, the interactions among these sequencing statements can be very hard to understand and lead to unexpected run-time behaviors. In fact, an Author can use the Sequencing Control Mode to define a path through a workflow-like specification. Then, she/he can add Sequencing Rules and Limit Conditions to assert when Learners are allowed to start a given Activity. These rules and conditions are completely independent of the workflow specification and can easily result in contradictory definitions. Authors are likely to commit such errors, as the complexity of the instructional material increases. Thus, they need a verification tool able to detect such inconsistencies. This however makes the authoring cycle more complex, and increases the difficulty in developing and using authoring tools. Another aspect that needs to be analyzed concerns the purpose and potential of LOM. Shortcomings of LOM are well known in literature. In particular (Duval & Hodgins 2003) proposes an LOM research agenda to improve the standard in areas such as LO taxonomy, LO composition, automatic metadata generation, search, etc. In (Quemada & Simon, 2003) it is argued that the fundamental problem with LOM is the broad definition of the term “learning object” and the consequent inadequacy of the metadata set, when trying to apply it to a particular scenario. In our opinion, LOM has been mainly defined to support search and discovery of instructional material, but it could be extended to provide support to many other activities such as: a) Automatic configuration of required software. Following the metadata specification, the platform could automatically configure pieces of software required by the LO. As an example, whenever a given LO exports a video content, a video streaming server could be automatically installed/configured. b) Automatic configuration of supporting software. By supporting software we mean tools that are not mandatory for LO execution, but that could improve the way the LO is exploited. As an example, an LO requiring asynchronous communication among Learners and Tutors could be supported by a forum, one requiring synchronous communication could exploit a chat, while one requiring the cooperative production of some homework could take advantage of a shared, versioned repository. In summary, information about the kind of communication and cooperation required by an LO could trigger the configuration of proper software supporting these activities. c) Tutoring. Metadata expressing instructional requirements could be useful to provide Learners with personalized automatic tutoring. In fact, they could support the selection of the most appropriate LO for a Learner, depending on his personal preferences and attitudes. d) Evaluation. Metadata could be used by Teachers to analyze and evaluate the effectiveness of LOs. These activities cannot take advantage of LOM metadata in their current form, because of the following weaknesses. First of all, the exact meaning of some metadata is not clearly specified (e.g., Semantic Density, Difficulty, etc.). Moreover, some important characteristics of LOs cannot be expressed. As an example, there is no way to say whether a given LO has been designed to support group study or individual study. Finally, the defined metadata are not fully machine-processable: Some of them are defined as free-text (e.g. Installation Remarks that could be relevant to the configuration of required software) while others rely on vocabularies that are not precise enough to enable a fully automatic processing. 36 The LOM extensions we propose in Section 3.1 try to address the points from A to D, while the LO composition model we present in Section 3.2 integrates the composition and the sequencing aspect thus reducing the effort of Authors and avoiding inconsistencies at runtime. The Virtual Campus conceptual model as an extension to SCORM The Virtual Campus project aims at offering a clean and simple conceptual model, called VC-LOM, that supports both LO definition and fruition in an interoperable and reusable way. Moreover, it offers a software platform supporting the entire life cycle of LO and their usage by Learners and Instructors. For the purpose of our approach, we define a Learning Object (LO) as the union of instructional materials and the corresponding metadata. In this section we focus on the conceptual model. In the following paragraphs, we present, in more detail, the extensions we propose to the LOM standard, and then discuss our composition approach that supports both aggregation and sequencing of LOs. Metadata specification Our extensions to the LOM mainly aim at supporting the activities of automatic configuration, tutoring, and evaluation by explicitly expressing some LO properties the LOM does not consider. Of course other extensions aiming at improving other aspects, such as discovery, could be included, but these are not our focus. The extensions we have defined are summarized as follows (see Table 1): ¾ A new set of metadata providing details on how an LO should be used. These concern the level of supervision a given LO requires (e.g., if a tutor should be available during fruition of the LO), whether the LO requires group (cooperative) study, whether some artifacts have to be created at the end of fruition, whether the LO requires communication facilities among Learners, and whether the communication or cooperation have to be synchronous or asynchronous. Such metadata can have various uses. In particular, they can help address issues presented in point B of Section 2. For instance, if the “Cooperation Attribute” and “Artifact Attribute” are set, the runtime platform could automatically configure a version-control server to facilitate the learners in working on shared documents. In addition, if the “Supervision Mode” is set to tutored, an upload facility could be configured in order to allow Learners to release the created documents, and Teachers to manage and evaluate them. These data can also provide information useful for automating the tutoring of Learners and the evaluation of LOs (points C and D in Section 2). For example, the fact that the "Cooperation Attribute" of an LO is set to true triggers the monitoring of the cooperative abilities of Learners taking that LO (for instance, the runtime environment could check the number of communications among Learners, the number of check-in/out operations they perform on shared documents, etc.). The collected data are added to the Learners’ profile and could be used by a tutoring system to push shy Learners toward interaction with the others, or to infer, through automatic evaluation, other data both about the performance of LOs and of Learners. ¾ Pre-conditions and Learning Objectives constraining the start and end of LO fruition. Both can predicate on Time and on data available from the Learner’s profile. ¾ The possibility of considering traditional situated didactical activities and digital e-learning material in a uniform way. Both of them in fact can be represented as LOs. A LO representing a situated activity will have the Access Modality attribute set to “Situated”, and Date and Place will provide information on the schedule and location where the activity will take place. Table 1 - Examples of Virtual Campus additional metadata Field description The level of supervision on Learners' activities: “none” (no supervision); “tutored” (a tutor is available, during fruition Learners can explicitly request his/her supervision); “supervised” (the supervisor is always present during the instructional process); “driven” (Learners act in a passive way, by strictly following the Teacher's instructions.) Cooperation Attribute Whether Learners should take the LO in cooperation. Communication Whether Learners will be provided with communication facilities, while exploiting Attribute the LO. Synchronism Attribute In case of a cooperative or communicative LO, it specifies if the LO must be taken synchronously by all Learners or asynchronously. Group Cardinality The cardinality of the group involved into the fruition of the LO. Meaningful group Field name Supervision Mode 37 Artifact Attribute Access Modality Place Date Precondition on time Precondition on user profiles Learning objective on time Learning objective on user profiles cardinalities are “1” (self-study), “2” (pair study, both Learner-Learner and Learner-Teacher), “m” (group study). Note: 2 is the minimum group cardinality for cooperative LOs and the maximum one for non-cooperative LOs. Whether the LO requires Lerner(s) to produce an artifact. “Situated” (in a specific physical location, e.g. a live lecture held in a classroom), “Digital”. The physical location name. If Modality is “Digital”, this field is ignored. The date (including starting and ending time) the LO is given. It is used for “Situated” LOs, but also for some kinds of “Digital” LOs that can be taken only at certain points in time. Constraints on time that have to hold before the LO is taken. The skills and knowledge a Learner must have in order to exploit the LO. It can also predicate on administrative constraints that have to be fulfilled by the user before exploiting the LO (e.g., he/she must have paid the enrollment fee.) The min/max amount of time required/allowed to complete the LO. Educational objectives of the LO in terms of skills a Learner can obtain by exploiting it. It can also express objectives that are not strictly educational (e.g. the fact that the Learner achieves some kind of degree by completing the LO). In order to support automatic installation of software required by LOs (see point A in Section 2), we are working at a more precise specification of the LOM metadata 4.4 Requirements. In particular, we are currently investigating on the possibility to express requirements and capabilities of software, by using the CC/PP (W3C, 2004) (Composite Capabilities/Preference Profiles) standard. The CC/PP is a standard aiming at describing device capabilities and user preferences. It introduces the structure for representing delivery-context information by means of profiles. Profiles consists of attribute/value pairs, which characterize in detail a certain feature of the delivery context. A vocabulary defines a specific set of attributes and component classes, designed for bringing together and describing the characteristics of a certain category of devices. The CC/PP itself does not depend on specific vocabularies, it just provides a structure for constructing and representing them. Although CC/PP’s initial goal is to describe devices and user-preferences, we argue its flexibility allows for the implementation of an LO ad-hoc profile. This profile will be able to express LO requirements for software in a far more precise way than the one permitted by LOM metadata. We also plan to investigate the IMS Learning Design specification (IMS, 2003), as it explicitly models the concept of “service facilities”. Service facilities are resources that cannot be given a URL at design time. They have to be instantiated by a local runtime service. Current specification allows the following services: send-mail, conference, monitor, and index search. This seems an interesting approach to support automatic configuration of supporting software (see point B in Section 2). Other approaches for the design of LOM extensions can be found in literature. In (Rivera et al., 2004), the LOM Application Profiles is used to adapt semantics of LOM elements, and make LOM-conformant extension to represent usage conditions and learning activities, in the context of the Elena project. In (Brase et al., 2003), constraints on LOM’s fields are defined by means of inference rules. RDF is exploited to define metadata, while an inference language explicitly developed for RDF (TRIPLE) represents axioms. LO composition We overcome the limits imposed on LO reusability by the Content Aggregation Model by defining a unique model for aggregated and simple Learning Objects and by enabling a powerful recursive composition mechanism. Moreover, we coherently integrate it with proper sequencing mechanisms. As shown in Figure 3, we define an Atomic LO (ALO) as the smallest unit of reuse for LOs that may or may not be associated to one or more multimedia contents. A Complex LO (CLO) is defined as an LO whose instructional material is an aggregation of Learning Objects. Being an LO, a Complex LO can be treated exactly as any other LO. Moreover, our composition mechanism defines the way some metadata of a CLO are automatically derived from the metadata of the component LOs (e.g., the Size of a CLO can be computed from the size of its components). 38 Figure 3 - The Virtual Campus LO model Of course, LOs defined within a CLO can be sequenced in a way that depends on their content, on some external conditions such as the availability of Teachers and Tutors, on the purpose of the specific CLO in which they are contextualized, etc. As it can be noticed, the information that constrain such sequencing are not all known at the same time (e.g., A Teacher’s appointment can be known few days before a seminar, while the need for taking Mathematics before Physics could be intrinsic in some specific curricula). Indeed, they do not have the same scope (again, the Teacher’s appointment can enforce certain sequencing only on a specific edition of a course, while the need for having Mathematics before Physics should be always enforced). Based on these considerations, we distinguish between three levels of abstraction that offer different but coherently related facilities to define sequencing constraints LOs: ¾ At the Reusable Level, Authors define each CLO in terms of a graph where nodes univocally represent LOs (either Atomic or Complex) while edges represent relationships between LOs. For instance, in Figure 4, History of Mathematics is referred into Basic Concepts LO and, therefore, can be seen as an optional, in-dept instructional material. ¾ At the Didactical Level, each CLO is seen in terms of a workflow (see Figure 5) that can be automatically generated starting from the Reusable Level definition. Such a workflow shows all possible learning paths for a CLO and it can be customized by the Teacher for specific purposes. For instance, an available path can be disabled for some specific reasons (see Figure 6). ¾ At the Fruition Level, some details needed for fruition of LOs are introduced (see later). More in detail, at the Reusable Level, rounded-corner rectangles inside a CLO represent particular CLOs called Inner CLOs. They provide a mechanism to aggregate LOs, but, differently from other CLOs, they do not have an identity and cannot be reused outside of the context of the CLO in which they are defined. They can however participate in any relationship connecting two generic LOs. LOs inside an Inner CLO can be mutually exclusive or not: The Inner CLO is labeled with two values indicating the minimum and the maximum number of LOs the Learner has to exploit. The syntax is (n,m) where n represents the minimum and m the maximum. Values as (1,1) indicates that LOs are mutually exclusive, while (0,1) states that Learners are allowed to skip the fruition of LOs. Finally, if the label is not indicated, all of the LOs have to be exploited. LOs labeled with T are called Test-LOs. They are used to model assessments Learners have to take. Relationships indicate the presence of instructional constraints between two LOs in the context of a containing CLO (outside that CLO, the relationship is no longer valid). A generic relationship from x to y in the context of z, with x,y being LOs (either Atomic or Complex) and z a CLO (on Inner CLO), is represented by an arrow from x to y inside z, labeled with the relationship name. The relationships can be named IsRequiredBy, References, and RequiresOnFailure. Their meaning is summarized in Table 2, where, for the sake of brevity, we omit the indication of the CLO where the relationship takes place. Relationship IsRequiredBy References Table 2 - Relationships between Reusable-Level LOs Description A IsRequiredBy B indicates that LO A must be completed before starting LO B; i.e., the Learner has to possess A-related knowledge in order to achieve a correct understanding of B. However, the IsRequiredBy relationship does not mean that Learners must complete A immediately before B: Learners are allowed to make use of other LOs after A and before B's fruition. A References B indicates that A cites B as a source of more details on a topic related to A itself. Taking B at fruition time is not compulsory but, in case it is taken, it has to be entered after A and before taking any other LO not connected to A through a References relationship. Many References can depart from the same referencing LO. In this case, 39 RequiresOnFailure Learners can make use of one or more of the referenced LOs. The RequiresOnFailure relationship always connects a Test-LO with some other LO. If the Test-LO is failed, then the LO at the other end of the RequiresOnFailure relationship has to be taken by the Learner. If no RequiresOnFailure is specified, Learners failing a Test-LO have to re-start the fruition of the whole CLO. The example shown in Figure 4 defines two different CLOs. The first one, “Mathematics”, is composed of several LOs. “Basic concepts” and “Algebra” are both required by the Inner CLO enclosing “Calculus”, “Geometry”, and “Limits”, so they should be taken in the first place. Notice that the relative order between “Basic concepts” and “Algebra” is not constrained. All the LOs inside the Inner CLO have to be exploited, since no label is present. The “History of mathematics” is left as an optional activity and, in case it is taken, it must immediately follow “Basic concepts” that references to it. “Exam” is a Test-LO. In this example, since no RequiresOnFailure relationship is defined, if “Exam” is failed the whole CLO have to be repeated. The second CLO, “Engineering first year”, is composed by reusing “Mathematics” as well as some other LOs. Learners have to complete “Mathematics” before entering “Physics”. “Chemistry A” and “Chemistry B” are alternative. This is defined by the label (1,1) that states the Learner must exploit only one of the two. Notice that Learners can take one of the “Chemistry” LOs either before “Mathematics”, or after “Physics”, or in between. It is interesting to note that “Mathematics”, being reused in this context, appears as a black-box. Its internal complexity is hidden thus allowing for an easy composition. Even more interesting is the fact that, in such a representation, less arcs are drawn means that more freedom is left to Learners. As an extreme example, a simple collection of LOs, with no arcs at all, permits the fruition of a course in which all possible paths are allowed. Figure 4 - CLO definitions LOs can be (re)used either to define other LOs or to provide them to the Learners. In the second case, they need to be translated into their didactical level representation. For instance, Figure 5 shows the workflow representations of the two CLOs defined in Figure 4. In this representation, LOs are mapped into activities that represent the fruition of the corresponding LOs. The syntax is derived from the UML activity diagram. However, it has been adapted to our particular case. Simple arrows connecting activities represent a sequence. Vertical bars encode the fork/join semantics (i.e. parallel execution or, more properly in our case, absence of constraints on the relative fruition order), as well as the multiple switch semantics (i.e. execution of one or more of the involved activities). A label (n,m) permits to define the behavior of the vertical bar (the meaning of (n,m) is the same as in the Reusable Level language). The dashed, bidirectional arc denotes the fact that the corresponding path is not mandatory (i.e. there is an optional activity). Finally, the double-arrow arc indicates the path Learners must follow whenever they fail a Test-LO. If such an arc is not indicated, Learners have to take again the whole CLO (in the example, the arc is drawn in order to permit a complete explanation of the workflow syntax and it could be removed). 40 Figure 5 - Workflow description of CLOs By using the editing tools provided by Virtual Campus (see Section 4), such a workflow representation is automatically derived from the Reusable Level model of an LO. If needed, the Teacher can customize this representation by performing the following operations: ¾ elimination of alternative paths by selecting a single path or a subset of the available ones; ¾ elimination/enforcement of optional activities; ¾ enforcement of the order of fruition, in case of parallel activities. All these operations preserve the consistency between the resulting workflow and the corresponding high-level description since they further constrain the way LOs are used by Learners. Figure 6 shows a possible customization of the workflows depicted in Figure 5. It is important to notice that if the Teacher does not need to customize the generated workflow, this step can be left completely hidden. Thus, in the simplest case, the teacher does not have to deal with the workflow representation. Figure 6 – Workflow customization At the Didactical Level an LO is not ready for fruition yet. Some additional details need to be added concerning the course edition, the enrollment method, start and end dates, the course calendar, announcements, the Teacher's name, the list of already enrolled students, etc. Such information is defined at the Fruition Level of abstraction and cause the creation of a Course which is ready for fruition. 41 The Virtual Campus runtime platform The Virtual Campus platform implements the VC-LOM and supports the design, deployment, fruition, and evaluation of learning materials. It is composed of two main subsystems: The Authoring Environment and the Fruition Environment (see Figure 7). Figure 7 - Virtual Campus high-level architecture The Virtual Campus authoring environment provides several editors to define Atomic LOs and Complex LOs at the Reusable, the Didactical, and the Fruition Levels. The editor used at the Reusable Level is shown in Figure 8. It is provided as an extended and personalized version of Microsoft Visio, to take full advantage of its drawing capabilities. The tool permits the definition of a new Atomic LO, specifying its content (if any) and its metadata. Complex LOs can be created, reusing and aggregating other LOs. In order to find LOs within the Virtual Campus repositories, a dialog box permits the specification of searching criteria, using metadata. It is also possible for authors to import SCORM LOs from external systems. Such LOs can be either standard SCORM compliant, or extended SCORM compliant (incorporating the Virtual Campus extensions). Figure 8 - The CLO editor (Reusable Level) 42 The application also features a Didactical Level Complex LO generator that automatically generates a first version of the workflow associated to a Complex LO and then supports Teachers in customizing it. Finally, a Fruition Level LO tailoring tool supports the insertion of all the necessary Fruition Level details (e.g. class start time). A LO can be serialized in the SCORM format at any of the abstraction levels defined in the Virtual Campus conceptual model. In such a way, Authors and Teachers are free to choose the preferred trade-off between abstract, highly reusable LOs (needing the final tailoring phase by the Teacher), and less reusable, ready-to-use LOs (no effort by the Teacher is needed). Reusable Level, Didactical Level, and Fruition Level specifications can all be exported at the same time in the same SCORM packet. The serialization algorithm maps both the Reusable Level and the Didactical Level languages on a subset of the SCORM Simple Sequencing tags. Moreover, the VC-LOM metadata are stored into the package as extensions to the standard LOM. When SCORM packages, generated with our platform, are exported to a Virtual Campus system, the whole specification is extracted. Otherwise, standard SCORM systems would simply ignore our extensions. The Fruition Environment is based on the SCORM Run-Time specifications; it allows Learners to attend courses, and Teachers to guide fruition (when required). The environment is Web-based. Learners and Teachers only need a Web browser (possibly augmented with plug-ins, such as Flash, the Java Virtual Machine, etc.) to exploit the contents of LOs. Our SCORM-compatible Engine incorporates both the standard SCORM Run-Time engine, and a Sequencing Engine. The Sequencing Engine enacts the fruition workflow associated to an LO, guiding Learners and Teachers in the execution of the activities related to the usage of the LO. It is composed of several modules. Besides the Workflow Engine, the other components are: a Precondition Resolution module which takes into account constraints added to LO metadata (e.g. class date and time); a Learning Objectives Evaluation module which evaluates the Learning Objectives specified in LO metadata. Whenever a given Learner finishes an LO, the Learning Objectives Evaluation module is invoked in order to apply the specified rules. Then, the Workflow Engine is asked for the list of LOs the Learner is allowed to enter as a next step. The resulting list is then filtered by the Precondition Resolution module, leading to the final list that is returned to the SCORM Engine. Finally, a Web page showing the list is presented to the involved Learner who can communicate her/his decision by selecting one of the proposed alternatives. Once the confirm button is pressed, an event is generated and notified to the Sequencing Engine. The Engine activates the tools that are required for the execution of LOs. These tools can be of the following types: ¾ Stand-alone applications (e.g., Microsoft PowerPoint or Flash), in this case off-line study is enabled by the platform. ¾ Server-based applications (e.g., tools supporting, collaboration among Learners). As an example, WebTalk (Barbieri et al., 1999) is an application, developed at Politecnico of Milano, providing a 3D metaphor to browse through teaching material. The tool allows the user to maintain awareness of presence of other users who browse through the same material, thus enabling unstructured interaction and sharing of information among Learners. WebTalk exploits the Macromedia Shockwave Multiuser Server to permit information exchange among Learner, while a Web browser (enhanced with the Macromedia Shockwave plugin) is required as a client. In this case fruition requires Learners to be on-line, connected to the Virtual Campus platform. The Fruition Environment (Sbattella et al., 2004) is instrumented with a monitoring tool that collects data on actions performed by users. Profiles are then generated and exploited, on one side, to provide feedbacks to Teachers on the validity of specific LOs and applications, and on the behaviour of Learners. On the other side, they are exploited to instruct automated tutoring agents, enabling them to suggest and guide Learners throughout the fruition process. Virtual Campus Vs. Scorm – Comparative Evaluation We have tested the Virtual Campus platform by experiments. The experimentation aimed at assessing the possibility of arranging in complex structures heterogeneous LOs (lectures, studying activities, cooperative sessions of work, and exams), and at testing the whole platform covering all the phases of course development and fruition. A first experiment focused on evaluating the behavior of students while working on a group project, 43 exploiting collaborative LOs. Then, we developed a complete on-line version of a course on “Web Design”, taught at the Politecnico di Milano. Finally, the course on “Software Architectures”, depicted below, has been given to some undergraduate students of a software engineering course at Politecnico di Milano. The results are still under evaluation. In general, students enjoyed the user-friendly interface and the easy interaction with the system and their colleagues. Moreover, they appreciated the support to collaboration offered by the Virtual Campus platform. All of them took the LOs defined as optional in the course, probably because of the novelty of being able to autonomously make choices. We also found that the added value of contents based on virtual environments was not perceived as determinant for the learning objectives. We expect that virtual environments could be more deeply exploited and appreciated in presence of students with disabilities, for whom traditional media are difficult to use. From the teacher’s perspective the system has been helpful to integrate existing materials and build Complex LOs in a few hours. Moreover, the monitoring facilities enabled him and the tutors to profile students’ behavior and to evaluate the effectiveness of the proposed Complex and Atomic LOs. As a complementary way to test Virtual Campus, we have evaluated the expressiveness of the model against the one of SCORM, referring to the example of the “Software Architectures” course. In the following we discuss this comparison. The Virtual Campus model The LO we have realized is composed of four main modules: “Introduction”, “Software Design”, “Design Patterns”, and “Architectural Styles”. The optional “Seminary” focuses on presenting advanced mechanisms for late-binding of software components. Finally, students are evaluated on both theoretical aspects (through a quiz) and practical abilities (by means of a laboratory session). Figure 9 shows the Reusable Complex LO associated to the course. The module on “Design Patterns” is offered by two alternative Atomic LOs. One of them is based on slides while the other provides a video. Note that the fact that such LOs are alternative is established by the Author of the course (considering the goals of the course), and the scope of such relationship is limited to the Complex LO “Software Architectures” itself. The quiz is implemented by the Test LO. Learners failing the test are required to take again the course starting from the “Software Design” LO. Figure 9 - The Reusable Complex LO of the course on Software Architectures The laboratory session is a Complex LO whose components are “Meeting Point” and “Software Engineering Project”. The former requires that Learners create groups, download the requirements of the simple event based application to be developed and exchange comments on them. The latter requires Learners implement the 44 application. Since they have to work in teams, they are requested to coordinate themselves by using a configuration management application. Notice the usage of a nested Inner CLO asserting that “Design Patterns 1” and “Design Patterns 2” are alternatives. It is enclosed in another Inner CLO that requires all its LOs to be exploited. In this specific case this means that one of the two “Design Patterns” LOs, and the “Architectural Styles” LO must be exploited. The optional LO “Seminary” can be exploited after “Architectural Styles”. All the LOs depicted in Figure 9 are decorated with metadata. For example, "Software Engineering Project" is a collaborative LO in which groups of Learners have to build a simple software application. Since it is tagged with "Supervision Mode = tutored", the Virtual Campus platform enables a Tutor to access all the “Software Engineering Project” instances exploited by the Learner groups. The tag "Artifact Attribute = true" tells the system to configure an up-load page to let the groups deliver their artifacts. The tags "Cooperation attribute = true" and "Group Cardinality = 2" allow for the creation of student pairs and for the installation of a versioning server to support the cooperative creation of the artifact. The tags "Communication Attribute = true" and "Synchronism = asynchronous" instruct the system to provide a forum. Finally, "Access Modality = Digital" specifies that the LO does not need a classroom. Figure 10 shows the Didactical Level Complex LO that is automatically derived from the Reusable LO in Figure 9. It can be customized depending on the needs of the specific course and teacher. For instance, Figure 11 shows a customization where the video version of “Design Patterns” has been eliminated and a fruition order between “Design Patterns” and “Architectural Styles” has been forced. This customized version of the course was then translated into a Fruition level LO and made ready for fruition. Figure 10 - The Didactical Level Complex LO of the course on Software Architectures Figure 11 - The customized version of the course on Software Architectures The SCORM model The aforementioned example can be described using the SCORM Sequencing and Navigation. Figure 12 depicts the SCORM Activity Tree representing our “Software Architectures” LO. Boxes labeled as “Anonim-n“ represent Activities we need to introduce, in order to reproduce the CLO aggregation structure shown in Figure 9. For example, “Anonym-4” represents the fact that “Architectural Styles” and “Seminary” can be seen as a single entity (see the semantics of the References relationship). 45 Figure 13 depicts the sequencing information needed to reproduce the behavior of the LO “Test”. For the sake of brevity, the description of the whole example is not depicted. It is easy to see that such a specification tends to become very complex, as the number of Activities and/or the number of allowed instructional paths increases. By contrast, the Reusable Level specification tends to be less and less complicated as the number of allowed instructional paths increases (the less the number of arcs drawn, the more freedom is left to Learners). Standard LOM metadata do not allow the specification of the aforementioned “configure up-load page” or “install versioning server” requirements, in a structured, machine-processable manner. Authors can add such requirements as textual notes, but machines cannot easily process such kind of unstructured specification. Figure 12 – Activity Tree Figure 13 - Sequencing information for the LO “Test” SCORM graphic representations In (Jun-Ming Su et al., 2005) a workflow-like, graphic representation of sequencing information is proposed, so we think it is interesting for us to compare our approach against this. SCORM Clusters and the associated set of SDM rules define a so-called Sequencing Object (SO). The model defines six kinds of SOs, each one representing a workflow construct: Linear SO (a simple, linear sequence), Conditional Linear SO (adds conditions αi on arcs, permitting to modify the strict linear behavior), Choice SO (selection among two or more objects), Conditional Choice SO (adds conditions αi, permitting to describe more complex selection behaviors), Loop SO (uses two conditions; useful to describe tests), and Exit SO (adds conditions αi that stop the instructional process). Figure 14 depicts how the SO-based language models the “Software Architectures” example. Gray boxes represent SOs, while white boxes model elementary activities (the leafs in the Activity Tree). The special nodes “Nin” and “Nout” do not represent any actual instructional content. They just act as input and output gates for Choice and Conditional SOs. Notice that, in case of condition failure, the standard behavior of the Loop SO lets learners return only to the SO that immediately precedes the current one. Therefore, a Linear SO is needed in order to group all the previous LOs. This specification resembles our Didactical Level language, while does not provide features equivalent to our Reusable Level language. Thus, the main lack we found is the absence of a composition scheme that permits to 46 decompose Complex LOs into simpler objects, describing them as separate entities (“Software Architectures” and “Laboratory”, in this example). Figure 14 – SO-based description of Software Architectures Related work Besides SCORM, and the approaches directly related to it, in the literature there are various platforms that support authoring and/or fruition of LOs. In this section we focus on the approaches supporting authoring, and classify them into three categories: relationship-based, workflow-based, and rule-based. Relationships, workflows, and rules Relationship-based systems allow teachers to define a course structure by means of logic relationships among the course components. MediBook (Knolmayer, 2001) is an example of such systems. In MediBook the important domain concepts are formalized and related to each other by semantic relationships. In turn, LOs are associated with concepts and are connected through so-called rhetorical relationships (e.g. LO-A deepens LO-B, LO-C ispart-of LO-D). MediBook uses the LOM standard to define LO metadata and to store rhetorical relationships. Learners can navigate through both the rhetorical relationships structure and the semantic relationships structure. In the latter case, they discover LOs starting from the associated concepts. As an alternative approach (Steinacker et al., 2002) uses a sort of “direct prerequisite” relationship to order LOs (e.g. LO-A is a direct prerequisite for LO-B). The matrix associated to the resulting graph shows the total number of direct and indirect prerequisites between two LOs. When Learners choose an LO to exploit, it is possible to calculate the list of required LOs. An integer-programming model is then built, taking into account further constraints (e.g. the time effort required by a given LO). By minimizing the model’s target function, some LOs are removed from the list. A sequencing procedure determining the “best” schedule for the remaining LOs is then executed. A similar approach, described in (Carchiolo et al., 2001), uses the same relationship, but adds weights in order to represent how hard is to access a given topic, coming from a previous one. To choose a path, Learners select it from the whole graph provided by the system. Each route is associated with a numeric index weighting the “effort to learn” the target topic. Ontologies are proposed in (Jin-Tan David Yang et al., 2004) as a mean to create relationships among instructional material. The paper makes use of RDF/RDFS to define the ontology of a given instructional domain. Sharing the ontology among authors, it is possible to generate SCORM-compliant Learning Objects, all of which provide an outline using precisely defined terms. 47 Workflow-based systems allow teachers to define a course structure as a workflow. In (Padrón et al., 2004) workflow languages such as BPEL4WS are exploited for the composition of learning web services and their adaptation to the needs of a Learner or group of Learners. Once composed and packaged as Learning Objects, these composite processes can be executed, instantiated and adapted to the Learner's particular needs. These adaptations can be realized, either by predefined rules implemented into the process description and driven by the Learner behavior, or in a supervised manner. Rule-based systems customize the fruition of LOs by mean of rules. Preconditions and postconditions rules are exploited in (Sicilia, M.A. et al., 2004). Preconditions encode prerequisites required for an instructional process to take place, while postconditions states the expected outcomes. Relying on such rules it is possible to calculate the list of LOs a given Learner is allowed to enter. In (Quarati, 2003), weights are associated to LOs. Weights represent the cognitive contribution of objects within a given context. If an LO has subcomponents, its weight is defined as the maximum weight of its subcomponents. The knowledge already acquired by Learners, together with the weights of LOs, is used to identify the part of the content domain available to an individual user at a certain point in time. Discussion Relationship-based, workflow-based, and rule-based systems offer both advantages and disadvantages. Relationship-based systems focus mainly on the definition of complex association structures between LOs, while the other classes of systems provide run-time mechanisms to check that such relationships are properly satisfied and, in some case, enforced during the fruition of courses. Workflow-based systems define a strict temporal order among activities. In some cases this forces teachers to impose unnecessary constraints. On the contrary, rule-based systems provide support to the definition of constraints on fruition paths, but do not offer mechanisms to let teachers define precise paths if this is needed. Based on our practical experience in giving courses and preparing the related material we have noticed that educators working on LOs at different levels of abstraction may need to exploit features offered by all the three kinds of approaches. In particular, educators in charge of defining the structure of courses want to compose LOs just by aggregating them and imposing some instructional constraints. For instance, should they need to build a math course using LOs “Limits”, “Derivatives”, and “Geometry”, they would probably impose the constraint “Limits” is required by “Derivatives”, while they would not constrain any sequencing between “Geometry” and the other two LOs. To achieve this goal they would probably prefer a relationship-based system. Vice versa, teachers willing to adapt LOs to a specific course instance (e.g., “Mathematics” for Mechanical Engineering) wish to impose more restrictive constraints on the LOs in order to define the structure of a course. In particular, teachers wish to have complete control over the fruition paths, for instance, imposing that “Geometry” has to be taken before “Limits” when the organization of that specific course instance requires it, e.g., because of problems related to the calendar structure. Thus, they would prefer to exploit a workflow-based system. Finally, teachers willing to precisely define some preconditions to the LO fruition (e.g., administrative constraints that have to be fulfilled), would prefer a rule-based system. For the above reasons, we have chosen to exploit all these approaches at the various levels of abstraction. At the Reusable Level, the composition of LOs is expressed according to a relationship-based approach, while at the Didactical Level, workflow-based and rule-based supports are provided. SCORM supports all the three approaches as well, however --as it should be clear from what we presented in the previous sections-- the lack of a clear definition of different levels of abstraction in which these approaches are exploited, and the presence of various overlapping concepts, makes it difficult to be used. Conclusions We see SCORM as a good opportunity to support interoperability among e-learning tools since it enables the definition of a data model that can be shared among them. However, we have noticed some weaknesses in such a data model. These weaknesses mainly concern the way learning resources can be structured and made available for reuse. 48 In our vision all the learning resources have to be thought as LOs, that is, entities described by proper metadata that can be recursively composed. Thanks to the recursive composition mechanisms, reuse both within a single platform and among platforms can be greatly enhanced: A LO at any level of composition can be reused and composed in another context. The definition of proper metadata can support not only browsing and reuse of LOs, but also their installation and execution, and also enable tutoring and evaluation. The Virtual Campus project provides an implementation of the aforementioned concepts. Moreover, it tries to enhance the SCORM run-time environment, exploiting a workflow engine to guide Learners through the instructional paths. We rely on model-oriented strategies to improve interoperability. However, we are aware that adopting standard interfaces is of course a crucial aspect. Therefore, as a future work, we plan to include the Simple Query Interface specification into Virtual Campus. Another aspect that merits further investigation is the definition of proper guidelines to support Authors and Teachers in the design of LOs. Clearly, the more their LOs correspond to fine granularity learning materials, the more such materials are reusable and applicable in various contexts. Indeed, the mechanisms to compose fine granularity LOs are essential in this case in order to avoid the difficulties of having a huge, non-organized collection of LOs. Acknowledgements The Virtual Campus project has been funded by Microsoft Research, Cambridge (UK). We thank Sam Guinea who participated to the implementation of the Virtual Campus authoring environment, and helped us reviewing the paper. We also thank the editor and the anonymous reviewers for their helpful feedbacks and suggestions. References ADL (2004a). SCORM 2004 2nd http://www.adlnet.org/downloads/70.cfm. edition - overview, retrieved March 22, 2006 from ADL (2004b). SCORM content aggregation model - version 1.3.1, retrieved March 22, 2006 from http://www.adlnet.org/downloads/70.cfm. ADL (2004c). SCORM run-time environment http://www.adlnet.org/downloads/70.cfm. - version 1.3, retrieved March 22, 2006 from ADL (2004d). SCORM sequencing and navigation - version 1.3.1, retrieved March 22, 2006 from http://www.adlnet.org/downloads/70.cfm. AICC (2004), CMI001 - AICC/CMI Guidelines For Interoperability v4, retrieved March 22, 2006 from http://www.aicc.org/pages/down-docs-index.htm. Barbieri, T., & Paolini, P. (1999). WebTalk: a 3D Collaborative Environment to Access the Web. In Proceedings of EUROGRAPHICS ’99, 111-113. Brase, J., Painter, M. & Nejdl, W. (2003). Completing LOM - How Additional Axioms Increase the Utility of Learning Object Metadata. In Proceedings of the 3rd IEEE International Conference on Advanced Learning Technologies (ICALT'03), Athens, Greece. Carchiolo, V., Longheu, A. & Malgeri, M. (2001). Learning through ad-hoc formative paths. In Proceedings of the International Conference on Advanced Learning Technologies (ICALT), Madison, Wisconsin, USA. Cesarini, M., Guinea, S., Sbattella, L. & Tedesco, R. (2004). Innovative learning and teaching scenarios in Virtual Campus. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA), Lugano, Switzerland. Dongsong, Z. L., Zhao, J., Zhou, L. & Nunamaker, J. F. (2004). Can e-learning replace classroom learning? Communications of the ACM, 47 (5), 75-79. 49 Duval, E., & Hodgins, W. (2003). A LOM Research Agenda. In Proceedings of the 12th World Wide Web Conference. Budapest, Hungary. IEEE LTSC (2002). IEEE Draft Standard for Learning Object Metadata, retrieved March 22, 2006 from http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf. IMS (2004). Content Packaging Information Model, version 1.1.4 Final Specification, retrieved March 22, 2006 from http://www.imsglobal.org/content/packaging/#version1.1.4. IMS (2003). Learning Design Information Model, Version 1.0 Final Specification, retrieved March 22, 2006 from http://www.imsglobal.org/learningdesign. IMS (2005). Simple Sequencing v1, http://www.imsglobal.org/simplesequencing/index.html. retrieved March 22, 2006 from Knolmayer, G. F. (2002). Decision support models for composing and navigating through e-learning objects. In Proceedings of the 36th Hawaii International Conference on System Sciences (HICSS), Big Island, Hawaii. Padrón, C. L., Torres, J., Dodero, J. M., Díaz, P., & Aedo, I. (2004). Learning Web Services Composition and Learner Communities Support for the Deployment of Complex Learning Processes. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT’04), Joensuu, Finland. Quarati, A. (2003). Designing shareable and personalisable e-learning paths. In Proceedings of the International Conference on Information Technology: Computer and Communications (ITCC), Las Vegas, Nevada, USA. Quemada, J. & Simon, B. (2003). A Use-Case Based Model for Learning Resources in Educational Mediators. Educational Technology & Society, 6 (4), 149-163. Rivera, G. M., Simon, B., Quemada, J., & Salvachúa, J. (2004). Improving LOM-Based Interoperability of Learning Repositories. In Proceedings of the Workshop on Ontologies, Semantics, and E-learning (WOSE), Larnaca, Cyprus. Sbattella, L. & Tedesco, R. (2004). Profiling and tutoring users in virtual campus. In Proceedings of the 5th International Conference on Information Technology Based Higher Education and Training (ITHET ’04), Istanbul, Turkey. Sicilia, M.-A., García, E., Pagés, C., Sánchez-Alonso, S., & Rius, A. (2004). Specifying semantic conformance profiles in reusable learning object metadata. In Proceedings of the 5th International Conference on Information Technology Based Higher Education and Training (ITHET 2004). Istanbul, Turkey. Simon, B., Massart, D., & Duval, E. (2004). Simple query interface specification, retrieved February 15, 2006 from http://nm.wu-wien.ac.at/e-learning/interoperability/query.pdf. Steinacker, A., Faatz, A., Seeberg, C., Rimac, I., Hrmann, S., El Saddik, A., & Steinmetz, R. (2001). Medibook: Combining semantic networks with metadata for learning resources to build a web based learning system. In Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunication (EDMEDIA), Tampere, Finland. Su, J.-M., Tseng, S.-S., Weng, J.-F., Chen, K.-T., Liu, Y.-L., & Tsai, Y.-T. (2005). An Object based Authoring Tool for Creating SCORM Compliant Corse. In Proceedings of the 19th International Conference on Advanced Information Networking and Applications (AINA’05), Tamkang University, Taiwan. W3C (2004). Composite Capability/Preference Profiles (CC/PP): Structure and vocabularies 1.0, retrieved March 22, 2006 from http://www.w3.org/TR/2004/REC-CCPP-struct-vocab-20040115. Yang J.-T., Yu, P.-T., & Chen, W. (2004). An Ontology-Based Course Editor (OBCE) for SCORM-compliant Learning Objects. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT’04), Joensuu, Finland. 50 Van Assche, F., Duval, E., Massart, D., Olmedilla, D., Simon, B., Sobernig, S., Ternier, S. & Wild, F. (2006). Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface. Educational Technology & Society, 9 (2), 51-67. Spinning Interoperable Applications for Teaching & Learning using the Simple Query Interface Frans van Assche European Schoolnet, Rue de Trèves 61, B-1040 Brussels, Belgium Tel: +32 2 790 7575 frans.van.assche@eun.org Erik Duval Katholieke Universiteit Leuven, Dept. Computerwetenschappen Celestijnenlaan 200 A, B-3001 Leuven, Belgium Tel: +32 16 327066 erik.duval@cs.kuleuven.ac.be David Massart European Schoolnet, Rue de Trèves 61, B-1040 Brussels, Belgium Tel: +32 2 790 7575 david.massart@eun.org Daniel Olmedilla L3S Research Center and Hannover University, Deutscher Pavillon, Expo plaza 1, D-30539 Hannover, Germany Tel: +49 511 762.9741 olmedilla@l3s.de Bernd Simon and Stefan Sobernig Vienna University of Economics and Business Administration Institute of Information Systems & New Media, Augasse 2-6, A-1090 Vienna, Austria Tel: +43 1 31336 4443 bsimon@wu-wien.ac.at ssobernig@wu-wien.ac.at Stefaan Ternier Katholieke Universiteit Leuven, Dept. Computerwetenschappen Celestijnenlaan 200 A, B-3001 Leuven, Belgium Tel: +32 16 327066 stefaan.ternier@cs.kuleuven.ac.be Fridolin Wild Vienna University of Economics and Business Administration Institute of Information Systems & New Media, Augasse 2-6, A-1090 Vienna, Austria Tel: +43 1 31336 4443 fwild@wu-wien.ac.at Abstract The Web puts a huge number of learning resources within reach of anyone with Internet access. However, many valuable resources are difficult to find due to the lack of interoperability among learning repositories. In order to achieve interoperability, implementers require a common query framework. This paper discusses a set of methods referred to as Simple Query Interface (SQI) as a universal interoperability layer for educational networks. The methods proposed can be used by a source for configuring and submitting queries to a target system and retrieving results from it. The SQI interface can be implemented in a synchronous or an asynchronous manner and is independent of query languages and metadata schemas. In this paper SQI’s universal applicability has been evaluated by more than a dozen implementations demonstrated in three different case studies. SQI has been finalized as a standard in the CEN/ISSS Learning Technologies Workshop. Latest developments of SQI can be followed at http://www.prolearnproject.org/lori/. Keywords Educational mediators, Learning object metadata, Standards, Interoperability, Brokerage, Application Program Interface, Learning Repositories, Querying, Web Services ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 51 Introduction Performing a search constitutes a typical process for everyone involved in teaching and learning. For example, pupils complement things they have learned in school by looking for learning resources when working on a homework assignment. Teachers aim at enriching their courses with external resources in order to reduce their own workload through re-use while increasing the effectiveness of their course at the same time. Employees responsible for personnel development try to find a cost-efficient but effective learning arrangement that fill a particular competence gap. The Web puts a huge number of resources within reach of anyone with Internet access. However, many valuable resources are difficult to find, because they are hidden in the closed and proprietary worlds of brokerage platforms, learning (content) management systems, streaming media servers and online collaboration tools. These systems - commonly referred to as learning object repositories - hold information on learning objects such as courses, online tutorials, lecture notes, electronic textbooks, tutoring sessions, quizzes, etc. Learning object repositories lacking interoperability create three major drawbacks for its users: 1. The number of learning objects accessible is limited to those held in the local repository. Teachers, learners, and learning managers might miss the latest critical developments due to the fact that the existence of an external learning object is hidden from them. 2. The way users can search for learning objects is restricted by proprietary semantics or by document-centred interfaces offered by today’s search engines. While the proprietary semantics can lead to effective search on the local repository, the lack of semantic alignment with the outside world reduces the usability. On the other hand general search engines hardly provide means for effective searches. For example, it is difficult to satisfy a search such as ‘List me all courses that improve my negotiation skills and are offered in Brussels between 15th of October and 30th of November and that are frequently visited by employees of the pharmaceutical industry’ using state-of-the-art technology. 3. Context information on the learning object is limited to the local environment hereby reducing the possibilities for deploying effective recommendation mechanisms. The evolution of web search engines has largely benefited from re-using context information (Page et al., 1998), e.g. by investigating the hyperlink structure of web documents (‘who links to whom?’). This kind of information is not accessible for closed educational systems, which is especially a drawback for making a highly complex decision such as selecting the right learning object. The context of learning objects – especially information on its usage (‘who has successfully used which learning object?’) – constitutes highly valuable information that is hidden in the closed and proprietary worlds of today’s repositories. Beyond the individual level, institutions are increasingly demanding interoperability in the educational domain in order to realize scenarios, such as ¾ exchanging learner performance records between applications, ¾ integrating course descriptions from heterogeneous providers or sharing them among multiple applications (e.g., learning management systems and performance measuring tools), ¾ delivering learning using a multi-application learning environment (content management systems and video conferencing tools), or ¾ comparing course evaluation data between different sites using different evaluation tools. On the institution level, interoperability is a driving force for integrating organisations, improving decision making processes, or expanding electronic distribution channels. In this paper we propose a common query interface as one part of the solution for making educational systems interoperable and exploring the hidden educational web (Raghavan & Garcia-Molina, 2001). The remainder of this paper starts with an introduction to the various aspects of interoperability. The sub-sequent section is devoted to the specification of a query service, but also reviews related work in this area. We continue by presenting three case studies established in different educational sub-domains that demonstrate interoperability. The paper concludes with a discussion of the status quo and outlines future work in the area. Within its focus on interoperability for querying, this paper targets architects of educational networks, managers of learning object repositories, stakeholders in learning object re-use, as well as researches in web services and system interoperability. 52 Aspects of Interoperability Interoperability can be defined as ‘the ability of two or more systems or components to exchange information and to use the information that has been exchanged’ (IEEE, 1990). This definition comprises three primary aspects that constitute interoperability, namely, exchange (i.e., communication) and use (i.e., correct interpretation) of information (i.e., data plus meaning). Information emerges from data through processing and utilisation. To ensure the correct interpretation in communication, as denoted in the definition, a common semantic model is required. This semantic model – also referred to as ontology – should specify the properties of the learning object accessible within the repository (Wiederhold et al., 1992). Each declaration of a learning object property constitutes an ontological commitment to use the defined term in interactions with the repository. The sum of all ontological commitments is reflected in a common schema. In the last couple of years institutions and associations such as ISO, IEEE, IMS, ADL, CEN/ISSS, or the HR-XML consortium have given rise to an inflationary number of standards and specifications that can be used as the basis for a common schema. In the case of learning object descriptions the IEEE LOM standard is the predominant reference for developing schemas. While in some cases LOM (and its standard XML binding) can be used as such, in other cases application specific adaptations are necessary (Quemada & Simon, 2003, Najjar et al., 2004; Rivera et al., 2004). These specific instantiations of the LOM model are referred to as LOM Application Profiles. On a lower layer, data model & format choices need to be specified, as different options apply for data representation format (i.e., the data model) and character set encodings. Typically, XML and relational databases with UTF-8 support are to be found here. Figure 1. Interoperability Framework In order to realize system to system communication a common messaging service, enabling repositories to interact, is required. XML RPC, Java RMI, and WSDL/SOAP are examples for such a messaging infrastructure. The messaging service is based on lower level network protocols such as TCP/IP, or HTTP. Based on the two pillars ‘communication’ and ‘representation’, interoperable applications can be built (see Figure 1). These applications might take advantage of interoperable atomic services that are aggregated in composite services. Atomic services might be needed, for example, to agree on a common procedure for uniquely identifying learning objects. Another example for an atomic service is related with authenticating users and repositories, or with creating and managing access control rights. Higher level services that create interoperable applications are, for example, an indexing service, a harvesting service or a query service. An indexing service is a kind of replication service that allows repository A to push learning object metadata to repository B. It supports distributed maintenance of metadata through insert, delete or update operations. A harvesting service is a service, where repository A pulls metadata from a repository B. The query service allows repository A to search repository B for suitable learning objects, so the metadata transferred matches a specific query. A contracting service assigns access rights to a learning object stored at a remote repository. A delivery 53 service interacts with the repository where the learning object is stored and delivers an electronic learning object to the end user. The design choices made when implementing the applications are driven by the use cases the applications are designed for. Examples of use cases are ‘Course selection for the purpose of providing certain qualifications’ or ‘Provision of a federated search in order to satisfy teachers in their need for new course material’. Use cases take place in a particular context. The context can be abstracted via the domain the application is designed for. Educational sub-domains such as higher education, vocational training or schools influence interoperability along all aspects. Technological choices made in the context also influence the concrete realization of interoperable application services. The Simple Query Interface – An Interoperable Application Service for Querying While for many aspects of the Interoperability Framework wide-spread solutions do exist, the educational domain is still lacking interoperable application services that take advantage of standards such as IEEE LOM, WSDL/SOAP, and XML. In this section we propose an application service for querying that can be used for federating heterogeneous learning object repositories. An application service for querying needs to specify a number of methods a repository can make available in order to receive and answer queries from other applications. To distinguish the requestor from the answering system in our scenarios, the term ‘source’ is introduced in order to label a system which issues a search (the source of the query). The term ‘target’ labels the system which is queried (the target of the query). Alternatively, the ‘source’ can also be referred to as ‘requestor’ and ‘target’ as ‘provider’. Metadata can be stored using different means, such as file-based repositories, (possibly distributed) relational databases, XML databases, or RDF repositories. In order to make learning repositories interoperable, not only a common interface needs to be defined, but also a common query language together with a common results format for learning object descriptions needs to be agreed on. The query service is used to send a query in the common query language to the target. Next, the query results, represented in the common results format, are transported to the source. On the implementation level, wrappers may need to be created to convert a query from a common query language X to a local query language Y and transform the query and the query results from a proprietary format to a common one and vice-versa. Figure 2 illustrates an exchange process, where Learning Repository A (the source) submits a query to Learning Repository B (the target). It is assumed that both systems have agreed upon a common query language beforehand. The concepts used in the query statement are part of a common (query) schema. At Repository B, the interface component might need to transfer the query from the common query language to the local one. Also some mappings from the common to the proprietary schema might be required before submitting the search. This task is performed by a wrapper component. Once the search has yielded results, the results set is forwarded to the source, formatted according to a common results format. The collaborative effort of combining highly heterogeneous repositories has led to the following requirements: ¾ The application service needs to be neutral in terms of results format, query schema and query language. The repositories connecting can be of highly heterogeneous nature: therefore, no assumptions about these components of the interoperability framework can be made. ¾ The application service needs to support synchronous and asynchronous queries in order to allow the application to be deployed in various use cases and heterogeneous network architectures. ¾ The application service needs to support, both, a stateful and a stateless implementation. ¾ The application service shall be based on a session management concept in order to separate authentication issues from query management, but also for providing an anchor point for implementing simple business models for access control. 54 Figure 2. Communication between two Repositories In addition, the design of the application service itself is based on the following design principles: ¾ Command-Query Separation Principle, ¾ Simple Command Set and Extensibility. The following sub-sections describe each of the above mentioned requirements and design principles of the query service in more detail. Since one design objective of the query service is to keep the specification simple and easy to implement, we decided to call the application service ‘Simple Query Interface (SQI)'. Query Language and Results Format In order to make use of SQI to implement full query functionality, the application service needs to be complemented with agreements about: ¾ the set of attributes and vocabularies that can be used in the query, ¾ the query language and its representation, and ¾ the representation of (a list of) learning object metadata that satisfy the query. Driven by the Interoperability Framework of Section 2, SQI is agnostic on these issues: Any agreement between two or more repositories is valid for SQI. Such agreements can, for example, be expressed via XML schemas or RDF schemas. Although SQI does not directly contribute to overcome the differences of the various paradigms in metadata management (Z39.50, XML-based approaches, RDF community), it aims to become an independent specification for all open educational repositories. Synchronous and Asynchronous Queries SQI can be deployed in two different scenarios: In the synchronous scenario, the source, after having sent a query, keeps waiting for an answer from the target or a time-out. Results retrieval is therefore initiated by the source through the submission of the query and through other methods allowing the source to access the query results. In the asynchronous scenario, results transmission is target-initiated. Whenever a significant amount of matching results is found, these results are forwarded to the source by the target. To support this communication the source must implement a results listener. The source must be able to uniquely identify a query sent to a particular target (even if the same query is sent to multiple targets). Otherwise the source is not able to distinguish the search results retrieved from various targets and/or queries previously submitted to a target. Please note that the asynchronous query mode does not require an asynchronous handling on the messaging layer. It can also be implemented by two synchronous functions at the source and the target, respectively. A query interface operated in synchronous mode can perform multiple queries per session (even simultaneously). In case of an asynchronously operated query interface, the source provides a query ID that allows it to link 55 incoming results to a submitted query (the source might query many targets and each target might answer to a query by returning more than one result to the source). Multiple queries can also be active within a session in asynchronous query mode. Session Management The application interfaces make abstraction from authentication and access control issues. However, there is a need to authenticate the source in order to allow a target, for example, to link query policies to a source repository. For instance: ¾ Repository A is allowed to query Repository B without any limitations, ¾ Repository C is only allowed to retrieve 1000 query results per day from Repository D at a maximum. Ideally, authentication is performed only once for a series of interactions. To accomplish this, a session token needs to be returned after successful authentication that can be used to identify the system in the subsequent communication. Session management needs to be understood as a higher-layer management of configuration settings and authentication. The session ID serves as a mandatory element in the application interfaces in order to identify the requestor/source in all query commands. Therefore, the SQI is based on a simple session management concept. A session has to be established before any further communication can take place. This specification separates query management and processing from authentication and authorization. The source establishes a session at the target and uses the Session ID, which is obtained from the target, to identify itself during communication. Authentication does not need to be based on username/password declarations or credentials, since also anonymous sessions can be created. The specification introduces an incomplete list of possible means for establishing a session for the communication between two systems. Once a session has been established, the source has the right to communicate with the target. In order to establish a session, a user name and password or any other credential may be required. The identification of a source repository can prevent candidate target repositories from opening up their systems to unknown partners, and enables query policies. A session is valid until it is destroyed. Hence, it continues to be active after a query has been executed. Alternatively, a session times out when no communication takes place during e.g. 30 minutes. However, a session might be valid much longer than 30 minutes and sometimes might even require manual destruction. The specification assumes the use of secure authentication, and encryption mechanisms such as those provided by state-of-the-art technology (e.g., SSL). Stateful and Stateless Communication Stateful and stateless are attributes that describe whether repositories are designed to keep track of one or more preceding events in a given sequence of interactions. Stateful means that the target repository keeps track of the state of interaction, for example, by storing the results of a previously submitted query in a cache. Stateless means that there is no record of previous interactions and that each interaction request can only be handled on the basis of the information that comes with it. The SQI specification allows implementers to opt for a stateful or a stateless approach. Command-Query Separation Principle SQI design adheres to the principle of Command/Query Separation, a design convention originally devised for the Eiffel programming language. It states that every method should either be a command that performs an action (also referred to as ‘procedure’), or a query that returns data to the caller (a ‘function’), but not both. More formally, methods should return a value only if they are referentially transparent, i.e. they can be expected to leave an object’s state unchanged and hence cause no side-effects. This convention is valuable for assertion- 56 based programming, a building block of the design-by-contract approach, the latter being essential with respect to service-oriented architectures in general (Henney, 2000). Simple Command Set and Extensibility In order to make the interface easily extensible an approach, minimizing the number of parameters of the various methods rather than the number of methods is adopted. Variations of the interface (e.g., a separation between common query schema and common results format), can easily be introduced by adding a new function (e.g., setSupportedQuerySchema) while no change in the already implemented methods is needed. Hereby, backwards compatibility can be more easily maintained. As a result, additional methods for setting query parameters like maximum duration and maximum number of returned search results were introduced. This design choice leads to simpler methods, but the number of interdependent methods is higher. However, default values can be used for many of these query parameter configuration methods. Overview of SQI Methods Table 1 provides an overview of the various methods that are described below from a workflow perspective. A detailed description of the methods is provided in the specification (CEN/ISSS, 2005). First, the source needs to create a connection with the target, for example by using createAnonymousSession. Once a session has been established, the query interface at the target awaits the submission of a search request. In addition, a number of methods allow for the configuration of the interface at the target. Query parameters such as: ¾ the query language (setQueryLanguage), ¾ the number of results returned within one results set (setResultsSetSize), ¾ the maximum number of query results (setMaxQueryResults), ¾ the maximum duration of query execution (setMaxDuration), ¾ and the results format (setResultsFormat) can be set with the respective methods. The parameters set via these methods remain valid throughout the whole session or until they are set otherwise. If none of the methods is used before the first query is submitted, default values are assumed. The specification provides default values for MaxQueryResults, MaxDuration, and ResultsSetSize. Table 1. SQI Methods Session Management createSession createAnonymousSession destroySession Query Parameter Configuration setResultsFormat setMaxQueryResults setMaxDuration Synchronous Query Interface setResultsSetSize synchronousQuery getTotalResultsCount Asynchronous Query Interface asynchronousQuery setSourceLocation queryResultsListener Next, the source submits a query, using either the asynchronousQuery or the synchronousQuery method. The query is then processed by the target and produces a set of records, referred to as results set. The query is expressed in a query language identified through a query parameter. In the query, reference to a common schema 57 might be made. In synchronous mode the query results are directly returned by the synchronousQuery method. The getTotalResultsCount method returns the total number for matching metadata records found by the target operating. In case of an asynchronously operated query interface the queryResultsListener method is called by the target to forward the query results to the source. In order to report abnormal situations (e.g., erroneous parameters or inability to carry out an operation), an SQIFault is provided, which can be thrown by all the SQI methods. A system of fault codes permits to document those abnormal situations. Related Work In recent years, researchers from various fields have become increasingly interested in interoperability standards for information systems in the (wider) context of Technology Enhanced Learning. In order to dissociate the efforts leading to the shaping and the adoption of SQI from comparable work, we provide a comparison on several dimensions: motivation and constitutive uses cases of the projects under consideration, application services offered, the respective extent of schema and representation handling, and messaging services and networking protocols used. The following account is inspired by the Interoperability Framework presented in Section 2. Interoperability research is driven by initiatives of different levels of granularity. We consider most of these orthogonal to our efforts, therefore differences manifests primarily at the levels of motivation, use case and scope. OpenURL (NISO, 2002; http://library.caltech.edu/openurl/), for example, and the underlying framework emerged from the field of digital libraries, or scholarly journal repositories to be more specific. The OpenURL standard aims at realizing decentralized resource resolution architectures that are adaptive to the human seekers’ organisational context and open to be extended in terms of repositories published and resolution services offered across organizational boundaries. OpenURL adopts a strict separation of referencing resources, i.e. creating annotated and globally valid resource identifiers, and resolving these references by organization-specific resolver facilities into, for instance, seeker-tailored library services (Van de Sompel & Beit-Arie, 2001). This highlights the major difference between OpenURL and SQI: While OpenURL targets identifier creation and user-centric identifier resolution across heterogeneous repositories, SQI realizes lookup infrastructures. An initiative of similar granularity, the Content Object Repository Discovery and Resolution Architecture (Rehak, 2005) is interested in providing a reference model for federating community- or organization-specific repositories and for pyramids of repository federations (‘Federated CORDRA’). The model describes centralized services for discovery (e.g., registries for content, affiliated repositories, system specifics) and resolving resource references. The latter refers to a unique identifier infrastructure based on namespace hierarchies in and across federations (Rehak, 2005). Efforts such as OpenURL and CORDRA are complementary to SQI as the latter – for instance – might be re-used as discovery standard in a (federated) CORDRA network or for querying repositories that refer to their learning objects by OpenURL identifiers (a more in-depth investigation of the applicability of SQI to CORDRA including a comparison of SQI-related approaches has recently been conducted by researchers involved in the CORDRA project (CORDRA, 2004). As for lookup standards or other initiatives at a comparable granularity level, Google Web API (http://www.google.com/apis/) and Amazon E-Commerce Service (www.amzaon.com/gp/aws/landing.html) have recently gained momentum. In principle, while designed for the specific needs of exclusively publishing the respective set of services in a (web) service-oriented manner their interaction standard models could be re-used partly for more generic scenarios. Reacting to their emergence and in continuation of the Z39.50 family of standards, the development of a Search/Retrieve Web Service (SRW) and a Search/Retrieve URL (SRU) for learning object repositories was initiated by the Z39.50-International Next Generation (ZING) community (Board, 2004). The domain of application of SRW/U-compliant services is still centred around digital libraries and research repositories, mainly by providing backward or gateway compatibility to existing Z39.50 nodes, although the protocols and available tool kits are adaptable to other contexts as well. SQI differs in this respect as it has already been adopted in contexts involving metadata about learning activities, as contrasted to learning material. SQI also meets the requirements for cooperating with providers other than digital libraries and scholarly repositories such as corporate providers of professional training (see Section 4.3). Another relevant initiative is the IMS Digital Repository Interoperability (DRI) Specification (IMS, 2003). It provides recommendations for realizing interoperability with respect to most common repository functions identified at a high level of abstraction, delegating many design choices to implementers. In contrast to CORDRA, it emphasizes types of interaction or lookup functions while CORDRA provides an architectural 58 perspective. The IMS DRI specification therefore served as a skeleton, for example, for the EduSource project (Hatala et al., 2004) and its underlying communication protocol – referred to as EduSource Communication Layer (ECL) protocol. The Open Knowledge Initiative (OKI) develops an open system to support on-line training. It provides a collection of Open Service Interface Descriptions (OSIDs) that are conceptualized as contracts in service-oriented environments and range from an OSID for repository interoperability to workflow and various learning service OSIDs. The concept of OSID is – like SQI – at a lower level of abstraction than the IMS DRI specification and is closely tied to ready-made and publicly available software components. The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) has been adopted beyond the digital library community and integrates with resolution standards like OpenURL (Dolog et al., 2003). OAI-PMH, in contrast to SQI or SRW/U, serves as facilitator for actual lookup services, also denoted as “Service Providers” in OAI terminology. A lookup standard that did not originate from the digital library or learning object community is OpenSearch (http://opensearch.a9.com/). Its main objective is the aggregation of lookup results of web-based search engines, therefore the focus lies – in contrast to SQI – on standardizing repository or rather search engine descriptions and result schemas for further processing. More recently, the issue of lookup has been discussed in connection with versatile web-based query languages independent from traditional data base management systems and their query exchanging and processing facilities: The Simple Protocol And RDF Query Language (SPARQL) specification hosts both a query language definition for RDF and a query exchange protocol (“SPARQL Protocol” in short) that is closely interwoven with the latter (DAWG, 2005b). This allows for specific query language constructs to be directly converted into required exchange patterns between a query source and a query target. The SPARQL protocol builds on concepts of (web) service-oriented architectures and targets a large set of use cases including the discovery of RDF-based descriptions of learning resources (DAWG, 2005a). The tight coupling of query language and query exchange / lookup protocol is a clear distinction between SPARQL protocol and SQI. The latter delivers arbitrary languages that are entirely independent from SQI as shuttle protocol. SQI comprises two central atomic services as expanded in more detail in Section 3: a lookup or query shuttle service and a combined session, authentication and authorization management service. As for lookup, this allows for two basic modes of operation, stateless and stateful, and therefore synchronous or asynchronous exchanges of queries and corresponding result sets. In this respect, SRW/U shares similarities with SQI, but also shows major differences. SRW/U is purely synchronous (source-initiated), i.e. query results are returned within a single transaction. Query results, however, can be cached optionally and referred back to be based on a result set identifier up to a configurable time-to-live. While limited in its basic lookup service, SRW/U offers differently flavoured lookup refinements in addition, i.e. an index browsing (‘scan’) and advertising (‘explain’) service that exposes repository capabilities on demand (Board, 2004). In the same way, OAI-PMH takes in the query primitive ‘Identify’ that requests basic repository blueprints (Dolog et al., 2003). The OpenSearch standard, on the other hand, requires depositing repository descriptions with an aggregator’s registry. Nevertheless, all mentioned specifications including the EduSource ECL protocol are limited to synchronicity of their basic lookup services, apart from SQI. The IMS DRI specification outlines five core commands in terms of pre-defined conversational styles, i.e. Search/Expose, Gather/Expose, Alert/Expose, Submit/Store, and Request/Deliver, on a highly abstract level. SQI therefore conforms to the Search/Expose conversation pattern while EduSource, that explicitly implements IMS DRI specification to a great extent in a service-oriented manner, provides a wider spectrum of lookup options, i.e. Search/Expose, Submit/Store, Request/Deliver and Gather/Expose. The SPARQL Protocol itself does not distinguish different types of lookup as they are imposed by the hosted RDF Query Language. The latter comprises four distinctive ‘query forms’ for requesting only variable bound responses (‘select’), for generating custom return RDF graphs (‘construct’), for requesting context information about resources (‘describe’) and for limiting results to binary truth statements (‘ask’) (DAWG, 2004). Most related interoperability standards such as SRW/U, OpenSearch, OAI-PMH or the SPARQL Protocol do not explicitly include a formally conceptualized session or access management service, though they can be complemented by opting for an application profile or by adopting an appropriate specification (CORDRA, 2004). Apart from SQI, the OKI project offers closely coupled Authentication and Authorization OSIDs that form a generic access framework (OKI, 2005.). Moreover, there are some OSIDs defining basic services such as lookup routing. As far as composite services are concerned, EduSource provides the Splash Federated Search facility over a registry of ECL-compliant repositories (EduSource, 2003; Hatala et al., 2004). SQI, however, has proven its applicability for derived or enhanced services such as network for federated search, end-user software and LMS 59 plug-in facilities, and higher level location and storage resolution (see Section 4). OpenSearch defines an aggregation service for result sets conforming to RSS as existing syndication standard and therefore raises the issue of post-processing of lookup results as a composite service. More tailored composite services are devised by OKI by providing OSIDs for a variety of learning administration scenarios, such as assessment, course management, grading and the like (OKI, 2005). The various approaches subject to this comparison also differ with respect to their specificity or neutrality towards underlying representation models and metadata schemas, i.e. prescribed specifications and/or application profiles. The degree of dependency varies widely for the levels of application services, messaging and transport protocols. SRW/U is entirely bound to XML as representation model at all of these levels: The SRW/U protocol is represented in XML, its SOAP flavor (SRW) in particular. SRW/U does not prescribe a specific metadata specification and therefore operates over various XML bindings of metadata standards such as Dublin Core, MARC and LOM. Furthermore, SRW/U was designed in mutual dependence for a specific query language, i.e. the Common Query Language (CQL) that provides both expressive power and readability to human seekers. As for shaping query results, SRW/U offers support for user selection of the schema used. In OAI-PMH, XML is also used at the messaging layer yet exclusively as representation form of responses to OAIPMH requests. Similarly, OAI-PMH allows to use arbitrary XML bindings of various metadata standard with Dublin Core, MARC and RFC1807 being provided off-the-shelf. In contrast, SQI is only partly dependent on XML as representation model by design, provided that SQI is realized as SOAP-based web services. SQI equally supports the selection of result schema although the generic character of SQI would not require it per se: Some query languages offer constructs to achieve schema specificity of result sets, e.g. QEL or SPARQL. The IMS DRI specification contains explicit recommendations for the usage of XML (and therefore XQuery) at the mediating representation model and at the messaging layer, i.e. SOAP. As a consequence, the ECL protocol relies exclusively on XQuery and enforces compatibility of affiliated repositories. The latter supports various metadata schemas, including XML bindings of LOM, Cancore and Dublin Core. The SPARQL language and protocol are closely tight to RDF as representation model with XML solely being used for some query forms (apart from RDF/XML as notational form). At the messaging layer, XML is integral part of some proposed protocol bindings such as SOAP. OpenSearch adopts XML in the shape of RSS as response format. The interoperability standards can be realized differently at the messaging and transport layer. We distinguish between different architectural styles that underlie interaction scenarios, i.e. Remote Procedure Call (RPC), Representational State Transfer (REST) and Document / Message Exchange (D/ME), and embodiments of these styles in terms of protocols (XML-RPC, SOAP, HTTP). SQI and OKI OSIDs reflect the RPC style as a large proportion of the comparable work does, e.g. OAI-PMH, SRW/U, and SOAP is predominantly used as binding protocol. In contrast to SQI and OKI, OAI-PMH, SRW/U (i.e. its SRU flavour) offer alternatives conforming to the REST that are therefore realized as HTTP transactions. OpenSearch, however, builds on REST exclusively. Implementation Case Studies Since the first stable version of the specification was made available in March 2004 many learning repositories have taken advantage of SQI to connect them to the outside world. Under the auspices of the CEN/ISSS Learning Technologies Workshop the following projects took advantage of the SQI specification. Ariadne The basic mission of Ariadne is to enable better quality learning through the development of learning objects, tools and methodologies that enable a "share and reuse" approach for education and training. The Ariadne Learning Object Repository is called the “Knowledge Pool System” (KPS): the basic idea is that the KPS holds learning objects and metadata that describe them, so that they can be more easily managed and made available in appropriate contexts. Typical Ariadne user communities include higher education as well as corporate training. The core of the Ariadne Knowledge Pool System (KPS), (Duval et al., 2001) is a distributed network of Learning Object Repositories that replicate both (the publicly available subset of) content and metadata. On top of this core infrastructure, Ariadne provides its members with a set of tools that are loosely coupled with the KPS (Ternier & Duval, 2003). Through these tools, the user community can transparently manage learning objects. 60 Examples of such applications that were coupled with the Ariadne KPS are Learning Management Systems (like Blackboard, Moodle and Ines) or authoring tools (like Phoenix or Powerpoint). Integrating SQI into these applications provides learners and teachers with transparent access to Learning Objects. Using e.g. the PowerPoint plug-in, a teacher can easily reuse pictures, slides that are already stored in the Ariadne knowledge pool, while the LMS plug-in enables reuse of complete learning objects like courses or presentations. More generally, a distinction is made between two search scenarios that pose different requirements to a search service: searching within the Ariadne KPS only and searching beyond the borders of Ariadne. While implementing an SQI target on the Ariadne KPS, we experienced that a synchronous target lowers the threshold for integrating search functionality into an application such as Microsoft PowerPoint. In this scenario, an application sends queries to a synchronous target and only fetches additional results when they are requested by the user. As a single Ariadne repository acts in this context as if it were a centralized system and as searching these repositories is optimized with search indices, there is no need to cache results for a query. This motivated the use of a stateless scenario. In a first phase, an SQI target was constructed that implements all SQI methods. Next, persistent sessions were introduced. As some applications only need a subset of the capabilities of a repository, we created persistent sessions that store a profile on the server. As an example, the PowerPoint plugin uses a session identifier that is pre-configured to only search within graphical objects using Lucene queries. The Ariadne SQI target comes with three query languages. A simple query language enables a source to send a list of search terms that are then used by the target to conduct a full metadata search. Next, a proprietary query format provides Ariadne tools with high level advanced queries. Finally, the Ariadne target also supports the Lucene query language. In order to provide Ariadne members access to other repositories, a federated search engine (Ternier et al., 2005) has been developed. This federated search engine acts as an intermediary service and makes it possible to transparently search into different repositories. The service exposes itself as a complete SQI target and acts as virtual repository that can be queried by a source. This facilitated the integration of the federated search engine into Ariadne clients as the interface of the federated search engine is the same as the Ariadne SQI Target. In the back-end, the federated search engine asynchronously distributes the query to different SQI enabled repositories. In this scenario, all repositories register their results using the results listener that is implemented on the federated search engine, which makes it easy for this engine to manage results that come from a large set of repositories. When a source invokes a query on this target, a state is being maintained for 15 minutes on the server. During this time, repositories can keep storing results for a query and the client can poll for additional results. In this stateful implementation, invoking the getTotalResultsCount method will result in different results as over time the amount of results that get registered will increase. Currently, searches are distributed into the following repositories: Ariadne, EdNA Online, EducaNext, Merlot, Pond, RDN, Smete, Voced, Citeseer and LionShare. As all these repositories support different query languages, which do not always easily map into one another, we started with an approach where a least common denominator of all query languages was implemented. In this approach, the query only consists of search terms which are translated by each repository into a query it can process. Currently, Ariadne requires each partner to return a minimum of metadata fields, encoded as LOM XML: a URL, an identifier, a title and an identifier of the originating repository. The URL should resolve to the learning object. If access to the learning object is prohibited, the URL resolves to contact information. A repository identifier is necessary to give credits in a proper way to repository that yielded the results. Apart from the data elements mentioned above, all other LOM metadata fields are optional in the results. Celebrate and iClass The iClass project aims at exploiting the potential of ICT to support a personalized, flexible and learner-centric approach. This pedagogical approach strives to facilitate empowerment of both pupils and school teachers, while producing personalized learning experiences. Based on this, the iClass project seeks to establish a framework to deliver a personalized, adaptable, and adaptive learning experiences in a collaborative environment for learners. The framework is based on an advanced learning system, named ‘Intelligent distributed Cognitive-based Learning System for Schools’ (iClass). The latter takes advantage of an ontology-based architecture to sequence knowledge and adapt it to the learner's level of understanding and learning style by dynamically creating individualized learning objects. 61 The cost necessary to acquire, deploy, and integrate an e-learning infrastructure such as iClass makes it an operation that cannot be renewed, especially by schools, each time that new tools are available. Moreover the cost of acquiring and/or producing learning contents adapted to such an infrastructure is often more important than the cost of the infrastructure itself. In this context, interoperability between learning systems, by allowing legacy systems to get access to content developed in new systems (and conversely), makes it easier for schools to accommodate to the technological evolution. The iClass adapter is a component of the iClass system. It enables the end-users of ‘non-iClass’ systems, such as the learning management systems and learning content management systems that are members of the Celebrate federation (van Assche & Massart, 2004), to search and get access to iClass contents (i.e., metadata and learning objects). Figure 3. iClass and its adapter. The diagram of Figure 3 depicts the main components of iClass that interacts with the adapter. The adapter is the entry point to the iClass system for external systems (e.g., system members of Celebrate). The adapter is also connected to: ¾ A content access service (CAS), which manages the access to metadata, ¾ A conductor service, which is in charge of coordinating the different tasks necessary to compose learning objects that are adapted to the user that requested them, and ¾ A rights management service, which in charge of authenticating and authorizing users and requests. Metadata are stored in a peer-to-peer network of metadata repositories accessible thanks to the CAS. Usually, obtaining a learning object is a two-step process: 1. Searching and evaluating metadata: Selecting a learning object that satisfies user needs on the basis of the description provided in the metadata; 2. Consuming the learning object: Getting the selected learning object at the location (usually a URL) provided in the metadata. The current version of the adapter only supports queries expressed in an ad hoc query language referred as ‘very simple query language’. This language consists of a set of keywords that are searched against the field title, description, and keywords of the learning object descriptions --For the second release of the adapter, it is planned to add support to XQUERY. The queries return results in a format based on the Celebrate application profile, an application profile of the IEEE Learning Object Metadata standard, which was especially developed to take into account the requirements of schools in Europe (Nirhamo & Van Assche, 2003). This metadata format provides an identifier of the described learning object rather than its location. Actually, the adaptive and multimedia nature of the iClass learning objects makes it difficult to get access to learning objects directly. This is why an extra step is required to resolve the location of a learning object identified in the metadata. This ‘resolvelocation’ step is used to send a request to the conductor service that takes care of generating the requested object. The newly assembled learning object is then moved to a streaming server (called iClass presenter) and a URL from which, the object can be consumed is returned by the conductor. Since this process has potentially a certain duration, the conductor answers to these requests asynchronously in order to ensure adequate performance of the caller, in this case the iClass adapter. It is only when a learning object is available at the presenter that its location is known and can be returned. 62 Since there does not exist an open interface for performing this step asynchronously and rather than creating an ad hoc interface, it was decided to use SQI for this task as well by taking advantage of SQI independence in terms of query languages and results formats. This is achieved by adding a new ‘query language’ (for requesting a location) and a new ‘results format’ (for returning a location) to the list of languages and formats supported by the iClass adapter (Massart, 2005). The new query language is named ‘ICLASS-LO-ID’. A query in this ad hoc language consists of the requested learning object's identifier as found in the metadata. The results format is named ‘URL’. A result in this format consists of a URL pointing to the requested learning object. This solution permits the minimization of the cost of implementing a ‘resolve-learning-object-location’ step for those learning object repositories that already use SQI for searching metadata. It is currently implemented as an extension of the SQI gateway of Celebrate. Elena’s Smart Space for Learning The third case study is instantiated in the context of personnel development. In order to achieve interoperability among heterogeneous systems for corporate personnel development, the Elena project (Simon et al., 2004) realised a novel search and retrieval infrastructure for learning resources called Smart Space for Learning. This infrastructure allows for the integration of heterogeneous educational nodes in a semantic network and provides ‘smart’ access technology for it (Simon et al., 2003). Integrated into a process-support for learning goal definition, personalized search, and feedback tools, the educational semantic network (the ‘space’) plays a crucial role for supporting corporate personnel development. A driving force for the design of the Elena Smart Space for Learning has been an extensive requirements analysis emphasising the need for integrating divergent resource providers into one common access tool (Gunnarsdottir et al., 2004). The broad variety of learning resources made available and accessible extends the scope of learning resources available. Hereby, potential learners are not dependent on offerings by a particular provider or are restricted to a specific learning format, e.g. a costly classroom-based course. On the contrary, they can expand their search to several types of learning formats and providers, e.g. books from Amazon media store. Another driving force for setting up a smart space for learning is the provider’s interest in disseminating their educational services and products via electronic distribution channels. The project team has evidenced a trend that these providers are increasingly offering electronic distribution channels in machine-processable formats such as XML while technology leaders even offer web services that allow for querying. The infrastructure of Elena’s Smart Space for Learning is built on three corner stones: 1. The Simple Query Interface (SQI) as interaction standard for both providing and consuming learning resources’ metadata within SQI-compliant networks. Elena opted for a service-oriented architecture and therefore makes use of web service bindings (SOAP) for SQI. 2. A global metadata schema (‘Elena schema’) against which queries are issued and in which search results are expressed. This schema is realised as LOM application profile that enriches the LOM element battery by mixing in other metadata standards such as OpenQ, a metadata standard specifically designed for exchanging course information (Rivera et al., 2004). The rational behind developing a LOM application profile is to design a schema according to Elena’s use case requirements while preserving LOM compatibility. As for representation model, Elena employs RDF bindings for the global schema. Users’ queries against this global schema are expressed in the Query Exchange Language (QEL, see Nilsson & Siberski, 2003), initially devised as a building block of Edutella, and semantic mappings are provided to convert from global to local schemas (Olmedilla & Palmèr, 2005). 3. Re-usable software components for integrating existing systems with minimum effort. They are applied for aligning the global and various local representation models (representation management and matching) and for mediating between query languages involved (query transformation). So far, we have connected a number of providers to our network that can all be accessed from the EducaNext Portal for Exchanging Learning Resources (http://www.educanext.org/networksearch/) and from the personnel development portal Human Capital Development (HCD) Online (http://www.hcd-online.org/networksearch/). The recurrent pattern of linking a provider’s system to a SQI-compliant network comprises five steps: In the first two steps, a SQI Target and a SQI Session Management object are to be created and registered with the underlying web service framework. Taking a white box perspective, the SQI target object is then to be populated with a mapping that is established between the respective local metadata representation and schema and the 63 global ones. Next, we devise a query translation facility that makes use of this mapping. Finally, the binding of the local results format to the global schema is realised, again in accordance with the previously defined mapping. The diversity that we experienced when applying this pattern in various concrete integration scenarios is considerable. The quality of partnerships, ranging from highly protective and restrictive to more open ones – open refers to service hosting, disclosure of local metadata schemas and provided interfaces – required differing decisions at the architectural level: In restraining scenarios (e.g., Amazon media store and Seminarshop.com) gateway configurations appear as appropriate, in less restrictive cases (e.g,. Moodle, ULI, etc.) intermediaries are not required, therefore reducing the complexity of implementation, maintenance and guaranteeing service quality. Characteristics of the local architectures affect architectural decisions similarly. A case in point is the integration of the Edutella peer-to-peer network that operates inherently in an asynchronous manner while SQI consumers might expect synchronous interaction (Olmedilla & Palmèr, 2005). As for heterogeneity in terms of metadata representations and schemas, academic and commercial learning (content) management systems (e.g., EducaNext, CLIX) or course databases (course catalogue of the Vienna Executive Academy) predominantly rely on relational data base systems. Data base systems of different vendors, in our case Oracle, Postgresql, MySQL, and Firebird, cause, for instance, discrepancies in the encoding of result sets. Bridging and translating between representation models entail the risk of losses in terms of expressive power with respect to semantics for instance. Proprietary and/or non-standards compliant metadata schemas increase the complexity of mapping to the global schema (e.g., Seminarshop.com). We resolved these issues with respect to heterogeneity both by means of materialised or persistent integration and virtual or non-persistent integration. The former involves the periodic replication and transformation of a targeted metadata repository into a persistent and global-schema-compliant replica. Examples are the Executive Academy of the Vienna University of Economics whose relational repository is replicated into a single triple relation. Other examples include Knowledgebay and LASON whose replication is based on a native XML database. Virtual integration does not involve persistent storage of replicated, intermediate or transformed metadata but a recurring on-the-fly mapping process. This is the case for most integration cases in the current network setting (e.g., Amazon media store, BFI). In rare cases metadata repositories are already aligned to a certain extent both with respect to the representation model and the schema used (ULI). In either case, whether considering materialised or virtual integration, query translation is an issue. The case of Amazon media stores demonstrates how to translate between a formal and highly powerful query language such as QEL and a parameterized canned query format as the Amazon PowerSearch syntax that cannot be considered a fully featured query language. Re-formulating queries expressed in RDF query languages such as QEL and RDQL into SQL is facilitated by re-usable software components provided by Edutella or similar efforts, such as D2RQ (Bizer & Seaborne, 2004). We developed and matured a comparable component for translating between QEL and XQuery that turned out to be highly re-usable in various, virtual and materialised integration contexts (Miklós & Sobernig, 2005). Discussion and Concluding Remarks In order to achieve interoperability among learning repositories, implementers require a common communication framework for querying. This paper discussed a set of methods referred to as SQI as a universal interoperability layer for educational networks. Although the effectiveness of the specification has been proven by several implementations (see previous section), lessons drawn from these and the discussion of comparable research efforts (see Section 3.8) suggest that some issues need to be further investigated. They equally relate to usage scenarios and application services provided. Scope for application: We consider the adoption of SQI in different contexts, such as exchanging (language versions of) vocabularies, evaluation data about learning / training service providers and as building block for an interaction standard for collaboration services in various learning scenarios. Advertise or Explain primitive: Specifications such as SRW/U or OAI-PMH provide such primitives while OpenSearch adopted some sort of capability schema for describing affiliated repositories (Board, 2004; Dolog et al., 2003; OpenSearch, 2005). SQI currently lacks such functionality and should therefore be extended accordingly, for instance, by adding an interface method that returns a SQI profile record that holds information on the query languages and results formats supported. This might alternatively be distributed among a set of 64 interrelated interface methods getSupportedResultsFormats. such as getSupportedQueryMode, getSupportedQueryLanguages, State Inference / Modifier primitives: We evaluate the possibility of supporting search status management, for example, for cancellation of search, or query status reporting. This would allow a user at a query source to infer from and interfere with an active search process at the target. Coupling result processing: OpenSearch primarily focuses on providing syndication of search results retrieved (OpenSearch, 2005). In the context of SQI it is important to find means for leveraging ranking mechanisms to the configuration of the actual lookup. This allows for reducing network traffic, since metadata that is anticipated not to be of high interest to the query issuing user can be omitted. At the same time, the quality of results subject to a process of pre-ranking is improved and the need for post-retrieval processing is considerably reduced. Consolidated interaction model: In its current shape, the SQI specification involves some transaction overhead with respect to passing necessary arguments to the lookup procedure calls. In order to be able to set fundamental sets of arguments at once, a unifying method (setQueryParameters) could be introduced. Moreover, the conceptualization of a session management service at the level of SQI might be questioned: Currently, the HTTP specification and web service standards built on top of it do not explicitly provide session handling. Nonstandardized practices based on cookie transactions, session-specific HTTP parameters and extensions to the SOAP Header element are proposed and realized. From this angle, a basic session management facility as described by SQI can be considered justified. Acknowledgements The SQI specification has been developed and financially supported under the auspices of the CEN/ISSS Workshop on Learning Technologies. This work is supported by European Commission via the IST projects CELEBRATE (http://celebrate.eun.org/), Elena (http://www.elena-project.org/), ICLASS (http://www.iclass.info/) and PROLEARN (http://www.prolearn-project.org/). We acknowledge contributions and comments to the SQI specification from Christian Werner (L3S Research Center), Dan Rehak (Carnegie Mellon University), Griff Richards (Simon Fraser University), Gerhard Müller (IMC), Julien Tane (Universität Karlsruhe), Marek Hatala (Simon Fraser University), Matthew J. Dovey (Oxford University), Michel Arnaud (Université de Paris X Nanterre), Nikos Papazis (NCSR), Peter Dolog (L3S Research Center), Sascha Markus (IMC), Stefan Brantner (BearingPoint Infonova), Stefano Ceri (Politecnico Milano), Simos Retalis (University of Piraeus), and Teo van Veen (Koninklijke Bibliotheek), Zoltán Miklós (Wirtschaftsuniversität Wien, Technische Universität Wien). References Bizer, C., & Seaborne, A. (2004). D2RQ - Treating Non-RDF Databases as Virtual RDF Graphs. In Short Paper/Poster Proceedings of ISWC2004, retrieved March 23, 2006 from http://www.wiwiss.fuberlin.de/suhl/bizer/d2rq/. Board, S. E. (2004). SRW: Search/Retrieve Webservice Version 1.1 (Consolidated Specifications): retrieved March 23, 2006 from http://srw.cheshire3.org/SRW-1.1.pdf. CEN/ISSS (2005). A Simple Query Interface Specification for Learning Repositories (CEN Workshop Agreement #15454). Brussels, Belgium, retrieved 23March 23, 2006 from ftp://ftp.cenorm.be/PUBLIC/CWAs/eEurope/WS-LT/cwa15454-00-2005-Nov.pdf. CORDRA (2004). CORDRA: Technical Introduction and Overview: retrieved March 23, 2006 from http://www.lsal.cmu.edu/lsal/expertise/projects/cordra/intro/intro-v1p00.html. DAWG (2004). SPARQL Query Language for RDF, http://www.w3.org/TR/2004/WD-rdf-sparql-query-20041012/ retrieved March 23, 2006 from DAWG (2005a). RDF Data Access Use Cases and Requirements: W3C. DAWG (2005b). SPARQL Protocol for RDF: W3C. 65 Dolog, P., Gavriloaie, R., Nejdl, W., & Brase, J. (2003). Integrating Adaptive Hypermedia Techniques and Open RDF-based Environments. In Proceedings of 12th International World Wide Web Conference. Budapest, Hungary: ACM. Duval, E., Forte, E., Cardinaels, K., Verhoeven, B., Van Durm, R., Hendrikx, K., Wentland Forte, M., Ebel, N., Macowicz, M., Warkentyne, K., & Haenni, F. (2001). The Ariadne Knowledge Pool System. Communications of the ACM, 44 (5), 72-78. EduSource (2003). ECL Protocol (Version 0.3). http://www.edusplash.net/technical/ecl/ECL_Protocol_0.3.03.pdf retrieved March 23, 2006 from Gunnarsdottir, S., Kieslinger, B., Küchler, T., & Simon, B. (2004). From e-Learning to Learning Management: Results of an International Survey. In Proceedings of 4th International Conference on Knowledge Management. Graz, Austria. Hatala, M., Richards, G., Eap, T., & Willms, J. (2004). The Interoperability of Learning Object Repositories and Services: Standards, Implementations and Lessons Learned. In Proceedings of the 13th World Wide Web Conference. New York City, USA. Henney, K. (2000, December). A tale of two patterns. Java Report, 84-88. IEEE (1990). IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York, NY: IEEE. IMS (2003). IMS Digital Repositories Interoperability - Core Functions Information Model: retrieved March 23, 2006 from http://www.imsglobal.org/digitalrepositories/driv1p0/imsdri_infov1p0.html. Massart, D. (2005). Accessing Learning Contents Using a Simple Query Interface Adapter. In Proceedings of the ProLearn-iClass Thematic Workshop on Learning Objects in Context. Leuven, Belgium. Miklós, Z., & Sobernig, S. (2005). Query Translation between RDF and XML - A Case Study in the Educational Domain. In Olmedilla, D., Saito, N. & Simon, B. (Eds.), WWW Workshop on Interoperability of Web-Based Educational Systems, Chiba, Japan: CEUR, 27-35. Najjar, J., Ternier, S., & Duval, E. (2004). Interoperability of Learning Object Repositories: Complications and Guidelines. IADIS International Journal of WWW/Internet, 2, 1-16. Nilsson, M., & Siberski, W. (2003). RDF Query Exchange Language (QEL) - Concepts, Semantics and RDF Syntax: retrieved March 23, 2006 from http://edutella.jxta.org/spec/qel.html. Nirhamo, L., & Van Assche, F. (2003). The CELEBRATE Metadata Application Profile. Celebrate Deliverable D 3.2 (IST-2001-35188): Available March 23, 2006 at http://celebrate.eun.org/docs/D3_2.pdf. NISO (2002). Information Retrieval (Z39.50): Application Service Definition and Protocol Specification: NISO Press, retrieved March 23, 2006 from http://www.niso.org/standards/resources/Z39-50-200x.pdf. OKI (2005). Open Knowledge Initiative: http://www.okiproject.org/. Olmedilla, D., & Palmèr, M. (2005). Interoperability for Peer-to-Peer Networks: Opening P2P to the rest of the World. In Olmdeilla, D., Saito, N. & Simon, B. (Eds.), Workshop on Interoperability of Web-Based Educational Systems. Chiba, Japan. Page, L., Brin, S., Motwani, R., & Winograd, T. (1998). The PageRank Citation Ranking: Bringing Order to the Web. Stanford Digital Library Technologies Project. Quemada, J., & Simon, B. (2003). A Use-Case Based Model for Learning Resources in Educational Mediators. Educational Technology & Society, 6 (4), 149-163. Raghavan, S., & Garcia-Molina, H. (2001). Crawling the Hidden Web. In Proceedings of the 27th International Conference on Very Large Databases. 66 Rehak, D. R., Dodds, P., & Lannom, L. (2005). A model and Infrastructure for Federated Learning Content Repositories. Paper presented at the WWW Workshop on Interoperability of Web-based Educational Systems, Chiba, Japan, retrieved May 1, 2006 from http://cordra.net/information/publications/2005/www2005/cordrawww2005.pdf. Rivera, G., Simon, B., Quemada, J., & Salvachua, J. (2004). Improving LOM-Based Interoperability of Learning Repositories. In Duval, E. & Spyns, P. (Eds.), Proceedings of the OTM 2004 Workshop on Ontologies, Semantics and E-learning, October, 2004: Springer LNCS (3292). Simon, B., Dolog, P., Miklós, Z., Olmedilla, D., & Sintek, M. (2004). Conceptualizing Smart Spaces for Learning. Journal of Interactive Media in Education, 2004 (9). Simon, B., Retalis, S., & Brantner, S. (2003). Building Interoperability among Learning Content Management Systems. In Proceedings of the 12th World Wide Web Conference. Budapest. Ternier, S., & Duval, E. (2003). Web services for the ARIADNE Knowledge Pool System. In Proceedings of the 3rd Annual Ariadne Conference. Leuven, Belgium. Ternier, S., Olmedilla, D., & Duval, E. (2005). Peer-to-Peer versus Federated Search: Towards more Interoperable Learning Object Repositories. In Proceedings of ED-MEDIA 2005. van Assche, F., & Massart, D. (2004). Federation and brokerage of learning objects and their metadata. In Kinshuk,. Looi, C. K., Sultinen, E., Sampson, D., Aedo, I., Uden, L. & Kähkönen, E. (Eds.), Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies, Los Alamitos, CA: IEEE Computer Society, 316-320. Van de Sompel, H., & Beit-Arie, O. (2001). Open Linking in the Scholarly Information Environment Using the OpenURL Framework. D-Lib Magazine, 7 (3), retrieved May 1, 2006 from http://www.dlib.org/dlib/march01/vandesompel/03vandesompel.html. Wiederhold, G., Wegner, P. & Ceri, S. (1992). Toward Megaprogramming. Communications of the ACM, 35 (11), 89-99. 67 Weitz, R. R., Wachsmuth, B. & Mirliss, D. (2006). The Tablet PC For Faculty: A Pilot Project. Educational Technology & Society, 9 (2), 68-83. The Tablet PC For Faculty: A Pilot Project Rob R. Weitz Associate Professor, Department of Computing and Decision Sciences, Stillman School of Business Seton Hall University, South Orange, NJ 07079, USA weitzrob@shu.edu Tel: 1-973-761-9540 Fax: 1-973-761-9217 Bert Wachsmuth Associate Professor, Department of Mathematics and Computer Science, Seton Hall University wachsmut@shu.edu Tel: 1-973-761-9467 Fax: 1-973-275-2366 Danielle Mirliss Instructional Designer, Teaching, Learning and Technology Center, Seton Hall University mirlisda@shu.edu Tel: 1-973-761-6021 ABSTRACT This paper describes a pilot project with the purpose of evaluating the usefulness of tablet PCs for university professors. The focus is on the value of tablets primarily with respect to teaching and learning (and not for research or administrative work). Sixty-four professors, distributed across the various schools of a university, were provided with tablet PCs and were trained in their use. A survey was distributed to the participants at the end of the semester. There were 59 respondents, and of these 45 used the tablet in at least one of their classes. This paper describes the pilot project and the survey results. We observed that a) only a fraction of faculty are motivated to use tablet technology: roughly a third of faculty expressed an interest in replacing their notebook computer with a tablet computer and b) generally, participating faculty did indeed use tablet functionality in their classes and were convinced that this use resulted in a meaningful impact on teaching and learning. Keywords Tablet PC, Instructional technology, Higher education, Electronic whiteboard Introduction A tablet PC may be defined as “a type of notebook computer that has an LCD screen on which the user can write using a special-purpose pen, or stylus. The handwriting is digitized and can be converted to standard text through handwriting recognition, or it can remain as handwritten text.” (Webopedia 2004). There are two basic versions of tablet PCs, one that includes a keyboard and one that doesn’t. The keyboard variety doubles as a standard notebook computer, with the screen swiveling and being laid flat over the keyboard when utilized in tablet mode. The model without a keyboard, also termed a “slate”, sacrifices functionality for lighter weight and smaller size. Generally speaking, tablet PCs command prices some several hundred dollars higher than comparably equipped (in terms of processing power, RAM, hard drive capacity, etc.) standard notebooks. Targeted commercial markets for tablet PCs include healthcare, insurance, sales force automation, finance and manufacturing/design (Himmelsbach 2004, Niccolai 2003). Several recent projects have focused on student usage of tablet PCs. On the elementary school level, Kravcik et al. (2004) describe the use of a tablet PC in the design of a system enabling “virtual field trips” by elementary school students. Crossland et al. (2000) provided tablet PCs to students in their distance tutoring system for mathematics. Cooper and Brna (2001) discuss the use of tablet PCs in their project that “allows children of age 5-6 years to co-construct a multi-frame cartoon to help story-writing with the help of an empathic agent.” At the higher education level, Scheele et al. (2004) discuss a lecture environment in which students are equipped with a variety of mobile, wireless devices, including tablet PCs. McFall and Dahm (2004) evaluate a prototype electronic textbook application running on the tablet PC platform. Yamasaki and Inami (2004) describe a tablet PC-based system for helping students acquire the skills and knowledge for proper handwriting of Japanese characters. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 68 The use of tablet PCs, principally by students, in higher education was tested during the summer and fall of 2002 by Microsoft and its partners through their “Rapid Adoption Program.” Three universities- MIT, Bentley College (in Waltham, Massachusetts), and the University of Texas at Austin - were chosen to participate in this program (Foster 2003, Microsoft 2002, ITRC 2002, Bentley College 2003, Condon 2003). Though details have not been released, Bentley College’s initial pilot, which ended in May 2003, indicated positive responses to the use of the tablets and Bentley reported plans to extend their pilot through July 2004. MIT utilized tablet PCs during its participation in the annual International Design Competition. Tablets (connected via wireless networking) allowed students to write math equations as well as to sketch, modify and share their designs easily. This approach was a significant improvement over paper-based or non-tablet technology. The participants of the pilot at the University of Texas at Austin were students in The School of Architecture, who used the tablet in four courses. In one they collected field data using the tablets, and in the others they reported positive results when note-taking in class and using the tablets with design and drawing software. Users reported that the tablet’s pen allowed a more natural interaction with the software than that available via standard computers. (However, though the pen allowed a more natural interaction with the electronic design environment, students complained that it lacked the pressure sensitivity required of a drawing tool.) Participants also reported that the tablet allowed greater mobility and creativity. They could use the tablet in almost any setting, enabling users to spontaneously create designs that can then be refined at a later time. It should be noted that users have also commented on some perceived shortcomings in tablets. In addition to the lack of pressure sensitivity mentioned above, these deficiencies include short battery life and excessive weight (Asay, 2002), the premium cost (Bentley College 2003), and difficulty in seeing the screens when outdoors in bright light (ITRC 2002). The aforementioned projects, reasonably enough, have been student-centered, with either explicit or implicit constructivist philosophies (Duffy and Jonassen 1992, Wilson 1996). However, the functionality of tablet PCs provide for some potentially valuable opportunities for faculty in their role as educators in a lecture style format. For example, tablets may be used in the classroom (along with a projector) as an electronic blackboard. Supply and demand curves, drawings of the human body, musical scores, electronic circuit diagrams, mathematical expressions (e.g., summation and integral signs), along with text (in any alphabet) may be captured and displayed. The tablet allows for multiple colors and widths of “ink”, and a virtually unlimited number of pages over which to write without erasing. Further, class notes created in this way may be saved and posted to the Web, where students can retrieve them and view them using freely available software (tablets not required). There are instructional opportunities outside of the classroom as well. For example, tablets have the potential to be used for electronic grading; assignments may be emailed to faculty or dropped off in digital drop boxes whereupon faculty may write their comments on the work and return it to the students the same way. There have been some initial reports of faculty use of tablets in the classroom. Wassgren and Krousgrill (2003) used a tablet-based system to record their in-class, pen annotated, PowerPoint based lectures for (later) storage and dissemination via the Web. Condon (2003) summarized their findings as follows: “(1) taking advantage of the Tablet PC's Journal software, using partially prepared slides and annotating them in class, was the most effective teaching style; (2) posting the recorded lectures on the web had no effect on attendance; (3) the students liked the clearer diagrams with colors and highlighting; (4) class preparation took a little longer; (5) setup and breakdown time for lecture delivery was only 2 to 5 minutes; and (6) few students viewed the online lectures if they had come to class”. Anderson (2004) describes the use of a similar system. In this application, one faculty member, responding to the increased functionality said, “Being able to diagram and spontaneously work examples instead of having to use a pre-scripted PowerPoint slide deck—felt like teaching a real class.” Logic, and these initial studies indicate that tablet PCs have something to offer faculty in academia. The purpose of the pilot project described here was to test and evaluate faculty applications of tablet PCs apropos their contribution to teaching and learning. Put another way, how would real faculty teaching actual classes use tablets, and how would they evaluate the utility of doing so? The pilot would help us answer such questions as: ¾ How do faculty perceive the value of tablet PCs with regard to teaching and learning? ¾ Are tablet PCs actually useful for teaching and learning? ¾ If so, how valuable are they and in what ways? ¾ What challenges/pitfalls are there in using this technology for teaching and learning? 69 Tablets cost from $200 - $400 more than comparable notebooks (Einhorn 2004). As faculty at Seton Hall University, where this study was conducted, were already provided with notebook computers, perhaps the overarching question to be resolved by the study was, do the added benefits of tablet technology indicate that all or some faculty should move from the notebook platform to the tablet? Background Seton Hall University (SHU) is a research and teaching-oriented university with some 10,000 graduate and undergraduate students. It has had a major focus on instructional technology for almost a decade. In the fall of 1995 SHU instituted a pilot project for evaluating a notebook/mobile computing environment (Weitz 1997); this pilot project resulted in notebook computers being distributed to all full-time students and faculty, and the implementation of an infrastructure – technical, administrative and instructional – to support them. In 2003, for example, more than 2,000 notebooks were distributed to freshmen, transfer students, faculty, and administrators, and almost 1,000 notebooks of sophomore students were “refreshed”. (Notebooks are replaced with current models every two years.) The campus has extensive wireless coverage, Internet2 connectivity, active use of standard course management software, and a center supporting numerous faculty instructional technology initiatives. SHU has ranked among the top 15 most wired universities in the United States in Yahoo! Internet Life's survey of technology on college campuses. The university formed a Teaching, Learning and Technology Roundtable (TLTR) after participating in the American Association for Higher Education (AAHE) Summer Institute in 1995. A TLTR has been defined as “a diverse group that meets regularly, considers issues, and (usually) provides advice to the chief academic office, other leaders, and many constituencies” (TLT Group, n.d.). The TLTR at SHU consists of the following action teams: ¾ Administrative Computing ¾ Best Practices ¾ Emerging Technologies ¾ Financial Resources Development ¾ Steering Committee ¾ Technology Assessment The Emerging Technologies team, composed of faculty and administrators, is charged with 1) staying informed of national trends in technology integration in higher education and 2) communicating its findings to the university community. The team submits a yearly report with general recommendations and in early 2003 it was recommended that the university explore the use of tablet PC technology by faculty and, to a lesser extent, by administration and staff. To this end, a small “pre-pilot” study was approved. The “Pre-Pilot” Study – Fall 2003 While open to exploiting new technologies for teaching and learning, SHU is, at the same time, naturally cautious in this type of endeavor. With this in mind, prior to authorizing a full pilot study, in the spring of 2003 a small (16 participant) “pre-pilot” was begun. Faculty, staff, and administrators were invited to participate in this small experiment with tablet computing by filling out a simple form and returning it via email. They were asked, in particular, to complete the following two statements. ¾ I think tablet PCs are more useful than regular laptops because: ¾ I would like to test my tablet PC for the following purpose(s): Fifty-eight valid applications were received from all areas of the university. A subcommittee of the Emerging Technology group met to evaluate the ideas presented and to award machines to the most promising applications. Generally speaking the subcommittee was looking for applicants who were best able to describe the advantages of a tablet PC over a regular laptop, and who had the most interesting suggestions for its use. The kind of things looked upon favorably by the committee included: ¾ Demonstration of real pedagogical payoff; for example, the need to compose formulas, write in foreign alphabets, draw diagrams, etc. ¾ Indication that the faculty member had thought about electronic storage and retrieval of tablet-created documents. 70 ¾ Ideas more significant than mundane usage; e.g., use of the tablet for note-taking at meetings or a preference for a lighter-weight platform than the current notebook. The faculty responses are available at http://www.cs.shu.edu/tabletpc/2003-ideas.html. They have been summarized by Condon (2003) as follows: ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ Several faculty proposed using the tablets in place of writing on the chalkboard or on overheads. One professor who alternates between using a laptop and the chalkboard in lectures has had a problem getting chalk dust all over the computer keyboard. He hopes switching to a Tablet PC might allow him to eliminate the use of the chalkboard altogether. An English professor envisioned projecting a literary passage on the screen and being able to make notes on the printed material. Several professors who teach finance, statistics, and physics look forward to being able to write equations quickly during lectures without resorting to the chalkboard and easily answer e-mailed questions with appropriate equations. A professor of clinical medicine was enthusiastic about teaching anatomy and physiology by annotating anatomical slides. A professor who will be teaching freshmen to use Excel and Access as management information systems would like to be able to project the operating software while answering student questions by drawing on the screen. The biology department plans to issue the Tablet PC to a student who will compare the use of the Tablet PC to her standard laboratory notebook to see which is more efficient as a way to record research. One physics professor sees the Tablet PC as a way to seamlessly incorporate Excel, Java, web animations, and Interactive Physics simulations in his lectures. A professor who teaches computer graphics normally draws sketches on the chalkboard and then teaches how to implement the design in the software. If he could draw the sketch on the Tablet PC, he could make the sketch electronically available to his students outside of class. Ideally the committee hoped for a distribution across schools and departments, with some administrative use. Fortunately the applications broke down naturally along these lines. Sixteen applicants were approved that represented a distribution across schools and departments, with some administrative use. The sixteen participants self-rated their experience-comfort level with computers as follows: Novice/Beginner: 2, Intermediate: 6 and Advanced/Expert: 8. Participants responded to detailed surveys prior to the start of the semester and at its end. The surveys indicated that the tablets were used productively, both inside and outside the classroom, resulting in a positive impact on teaching and learning. A working paper detailing the pre-pilot and its evaluation are available from the authors. With the encouraging results of the pre-pilot in mind a full pilot program was initiated for the fall of 2004. Given financial and technical-support constraints, the pilot would be limited to about 60 participants. The Pilot Study – Fall 2004 As stated previously, SHU provides all faculty and undergraduate students with notebook computers and these computers are replaced with new models every two years. In the spring of 2004 faculty eligible for notebook replacement were asked to express one of the following preferences: they could 1) exchange their current notebook computer for the new model notebook, 2) keep their current notebook for another year, or 3) apply for a tablet computer. If the number of faculty requesting tablets exceeded the maximum, about 60, an application procedure would be established as it had been for the pre-pilot. The specifications of the two machines were made available to the faculty, both via TLTR events and via the Web. Aside from tablet functionality, the essential differences were that the tablet had a smaller screen than the notebook, and an external CD/DVD-RW drive as opposed to an internal one for the notebook. Faculty responses are provided in table 1. 71 Table 1: Faculty Preferences Regarding Notebook Computer Replacement Option Number of Faculty Percentage of Faculty Replace with the new model notebook 113 51.36% Keep current notebook for an extra year 43 19.55% Apply for a tablet PC 64 29.09% Total 220 This data in itself is of interest. First, approximately 20% preferred keeping their two year-old computer as opposed to upgrading. We surmise that the effort of transferring files and settings (even with the aid of technical staff) is not worth the bother to these individuals. Further, it seems that (only) approximately 30% of faculty eligible for hardware replacement felt that a tablet computer would be a better choice than a standard notebook, or a sufficiently better choice to warrant the effort of learning how to use the technology. This figure provides a measure of the perception faculty have of the benefits of tablet technology. We shall return to this data later in the results section. In any case, as all faculty interested in a tablet could be accommodated, no further application process for the pilot was required. Pilot Participants The pilot, with 64 faculty members participating, took place during the fall semester of 2004. A survey was distributed at the end of the semester to participating faculty, and 59 responses were obtained. Of the 59 respondents, 45 utilized the special functionality of their tablet for teaching and learning in at least one of their classes. Table 2 provides the distribution across disciplines of respondents to the survey. Table 2: Distribution of Tablet PCs for Pilot School Arts and Sciences Business Education Nursing Graduate Medical Education Theology Library Total All Survey Respondents 35 7 5 6 3 1 2 59 Respondents Who Used Tablet Functionality for Teaching/ Learning 25 5 5 5 3 1 1 45 Applicants were asked to self-rate themselves in terms of their experience-comfort level with computers. Table 3 summarizes the responses. Table 3: Participant’s Self-Rated Experience/Comfort Level with Computers Respondents Who Used Tablet Experience/Comfort All Respondents Percent Functionality for Teaching/ Level With Learning in At Least One Class Computers Novice/Beginner 8 13.6% 5 Intermediate 30 50.8% 20 Advanced/Expert 21 35.6% 20 Total 59 100.0% 45 Percent 11.1% 44.4% 44.4% 100.0% We see no particular pattern among faculty who did not use tablet functionality in each least one course during the semester. The reasons faculty gave for non-use are compiled below, in table 4. 72 Table 4: Reasons Tablet Functionality Not Used For Teaching and Learning Reason Number of Faculty Sabbatical/Not Teaching 4 Received Tablet Too Late 2 Technical Problems 1 Just Didn’t Use 6 Was a Mistake to Get Tablet 1 Total 14 We see that four faculty simply were not teaching during the semester, and two received the tablet “too late” to incorporate it into their teaching. The individual with technical problems indicated that the tablet was “out for repairs” for most of the semester. This leaves seven who didn’t use tablet functionality other than for some exogenous factor. The six faculty who “just didn’t use” the tablet functionality indicated that due to competing work and time pressures they never got around to learning how to use the tablet or how to integrate it into their course(s). The last individual simply indicated that, in retrospect he didn’t need or want tablet functionality. For the remainder of this paper, except where noted otherwise, we utilize data only from the 45 respondents who utilized tablet functionality in the service of teaching and learning in at least one of their classes. In-Class Use of Tablets This section discusses survey results related to faculty’s in-class use of the tablets. Survey Data Table 5 summarizes faculty responses with respect to the frequency of use of tablet applications. Table 6 focuses on the respondent’s assessment of the impact on teaching and learning of each application used. We recognize that these are different measures (and, in particular, that a low frequency of usage of a particular application does not necessarily mean a low impact of that application). The first column of table 5 provides a range of in-class tablet applications. The next five columns indicate faculty responses regarding the frequency with which they used each application. So, for example, nine faculty (20% of respondents) indicated that they used their tablet for writing mathematics in every or almost every class. Three (6.67%) frequently did so. The “total” column indicates the number of respondents for each application. Table 5: Frequency of use of tablet functionality: in-class applications For this course, please indicate below how frequently you used the unique functionality of the tablet PC (e.g. handwriting/drawing) in class as part of the lecture (e.g. in conjunction with a projector) for each of the applications listed. Always/Almost Never/Not In-class Applications Always Frequently Occasionally Applicable Total 9 3 8 25 45 Writing Mathematics 20.00% 6.67% 17.78% 55.56% Writing non-English text; e.g., 2 0 2 41 45 Hebrew, Russian, Greek (in a non4.44% 0.00% 4.44% 91.11% mathematical context) 8 4 19 14 45 Drawing diagrams, charts, and/or graphs 17.78% 8.89% 42.22% 31.11% Correcting assignments/ projects 6 6 10 23 45 (e.g. writing on documents during 13.33% 13.33% 22.22% 51.11% in-class demonstration) Using the handwriting functionality and a new (blank) Windows Journal or Word file to create lecture notes. (That is, using the tablet in-class as 11 7 10 15 25.58% 16.28% 23.26% 34.88% 43 73 an electronic whiteboard.) Using the handwriting functionality to draw or write over prepared (already has content) Windows Journal, PowerPoint, Word or Excel files. Electronically distributing in-class, tablet-created handouts/lecture notes to students. 8 7 16 14 45 17.78% 15.56% 35.56% 31.11% 9 5 7 23 20.45% 11.36% 15.91% 52.27% 44 (Note that the “always” and “almost always categories”, and the “never” and “not applicable” categories were separate responses in the questionnaires. They have been collapsed into single categories here for clarity.) Respondents’ impressions of the impact of their use of each application are summarized in table 6. (Note that the categories in the first column have been abbreviated in the interest of saving space.) Ten faculty felt that their use of the tablet for writing mathematics had a “very positive” impact on teaching and learning. Percentages in this table are based on faculty who used the application at least occasionally. So, for example, these ten faculty make up 50.00% of all (20) respondents who used their tablet for writing mathematics in class at least occasionally. If faculty did not use an application at all, they were (of course) not able to estimate the impact of their use of that application; these responses are captured in the second-to-last column. Table 6: Perceived impact of tablet functionality: in-class applications (Percentages based on respondents who used the application at least occasionally.) Now please consider the in-class applications listed above *that you have used at least occasionally*. Please rate below the impact you feel YOUR use of that application has had on teaching and learning in this class. neutral/no don't did not use In-class applications very added very know/no at least value positive positive negative negative opinion occasionally total 10 8 1 0 0 1 25 45 Writing Mathematics 50.00% 40.00% 5.00% 0.00% 0.00% 5.00% Writing non-English 3 0 0 0 0 1 41 45 text 75.00% 0.00% 0.00% 0.00% 0.00% 25.00% Drawing diagrams, charts, and/or graphs Correcting assign’ts/projects Using the handwriting functionality to create lecture notes. Using the handwriting functionality to draw or write over prepared content. Electronically distributing in-class, tablet-created handouts/lecture notes to students. 11 17 1 1 0 0 14 44 36.67% 7 31.82% 56.67% 6 27.27% 3.33% 5 22.73% 3.33% 0 0.00% 0.00% 0 0.00% 0.00% 4 18.18% 23 45 10 13 2 0 0 3 15 43 35.71% 46.43% 7.14% 0.00% 0.00% 10.71% 12 11 6 0 0 2 14 45 38.71% 35.48% 19.35% 0.00% 0.00% 6.45% 9 5 4 0 0 3 23 44 42.86% 23.81% 19.05% 0.00% 0.00% 14.29% Discussion Figure 1 provides a summary of the usage of in-class applications and the perceived impact of that usage. We see, for example, that 45 faculty responded to the question regarding the usage of the tablet for writing 74 mathematics in the classroom. Twenty of these respondents indicated that they used this application at least occasionally. Eighteen of these 20 rated the impact of their use of this application as positive or very positive. 50 40 30 20 10 0 Total Respondents Distributing Lecture Notes Whiteboard Create Lecture Notes Whiteboard Writing Over Documents Correcting Assignments Drawing Diagrams/Charts Writing NonEnglish Frequency: Used at Least Occasionally Writing Math Number of Responses Summary of Tablet In-Class Usage and Perceived Impact By Application Impact: Positive or Very Positive Application Figure 1: Summary of in-class results The number of faculty who used an application on at least an occasional basis provides a snapshot of the extent of diffusion of that application. We see that approximately two-thirds of the responding faculty used their tablets as a whiteboard – creating notes from scratch (65.12%) and writing over materials prepared before class (68.89%). Not surprisingly, faculty used the tablets in domains where the drawing functionality is most useful: writing mathematics (44.44%) and drawing diagrams, charts and graphs (68.89%). Almost half of respondents (48.89%) used the tablets to correct assignments/projects as an in-class demonstration and almost half (47.73%) electronically distributed in-class, tablet-created materials to their students. Only four faculty used the tablet for writing non-English characters, though this small number reflects the few participants in the study likely to need this functionality. Faculty felt positively about the impact on teaching and learning of the applications they used. Approximately 90% felt that their use of a tablet for writing mathematics and drawing diagrams, charts and/or graphs had a positive or very positive impact. Some 80% responded that the impact of using the tablet as a whiteboard without prepared materials, was positive or very positive. Approximately 75% thought using the tablet to write over prepared material had a positive or very positive impact. Over 60% felt that using the tablet to correct assignments/projects as part of in-class demonstrations had a positive or very positive impact. Likewise over 60% of faculty who distributing the notes they created in-class felt that doing so had a positive or very positive impact. Three of the four faculty who used their tablet to write non-English text felt that this application had a positive or very positive impact. Out-of-Class Use of Tablets This section discusses survey results related to faculty’s use of the tablets outside of the classroom. Survey Data Again, we separately analyzed frequency of use and perceived impact data. Table 7 summarizes faculty responses with respect to the frequency of use of tablet applications outside of the classroom. Table 8 focuses on the respondent’s assessment of the impact on teaching and learning of each of these applications. 75 The first column of table 7 specifies the outside-of-classroom applications. The next five columns indicate faculty responses regarding the frequency with which they used each application. So, for example, eight faculty (17.78% of respondents) indicated that they used their tablet for grading assignments and/or projects for every or almost every class. Five (11.11%) frequently did so. The “total” column indicates the number of respondents for each application. Table 7: Frequency of use of tablet functionality: outside-of-class applications always/almost never/not Outside-of-class applications always frequently occasionally applicable Using the handwriting functionality 8 5 13 19 of your tablet to grade 17.78% 11.11% 28.89% 42.22% assignments/projects. Using the handwriting functionality of your tablet to prepare handouts/lecture notes before class. Electronically distributing handouts/lecture notes (created outside of class) to students. 5 7 13 20 11.11% 15.56% 28.89% 44.44% 8 4 12 21 17.78% 8.89% 26.67% 46.67% 45 45 45 total Table 8: Perceived impact of tablet functionality: outside-of-class applications (Percentages based on respondents who used the application at least occasionally.) did not use Outside-ofvery positive neutral/no negative very don't at least class positive added value negative know/ occasionally applications no opinion Using 8 12 2 2 0 2 19 handwriting 30.77% 46.15% 7.69% 7.69% 0.00% 7.69% functionality to grade Using 9 10 2 0 0 4 20 handwriting 36.00% 40.00% 8.00% 0.00% 0.00% 16.00% functionality to prepare handouts/ lecture notes Electronically 9 10 2 0 0 3 21 distributing 37.50% 41.67% 8.33% 0.00% 0.00% 12.50% handouts/ lecture notes total 45 45 45 Discussion Figure 2 provides a summary of the usage of applications outside the classroom and the perceived impact of that usage. Forty-five faculty responded to each question. We see, for example, that 26 indicated that they used the handwriting functionality of their tablet for grading assignments/projects outside of class; of these 26, 20 rated the impact of their use of this application as positive or very positive. This data is fairly consistent across applications: a little more than half of the respondents used each outside-ofclass application, and about three-quarters of those who used an application felt that that usage resulted in a positive or very positive impact. 76 50 Total Respondents 40 30 Frequency: Used at Least Occasionally 20 10 Impact: Positive or Very Positive Electronically Distributing Handouts/Lecture Notes Preparing Handouts/Lecture Notes 0 Grading Assignments/ Projects Number of Responses Summary of Tablet Outside-of-Class Usage and Perceived Impact Application Figure 2: Summary of outside-of-class results Qualitative Responses We also polled participating faculty with open-ended questions. When asked to, “Indicate what seems to be the most valuable attributes of the tablet for teaching and learning in this course,” faculty responded as follows. ¾ Ability to write and save lecture notes in class, especially involving graphs and equations ¾ Ability to write on my prepared documents, especially PowerPoint slides! (Gave students a much better sense of what I was describing.) ¾ Because I teach courses about students with special needs, the entire instrument serves as a model for what can be done to enhance opportunities for learning. ¾ Being able to create lecture notes inside of class and being able to distribute them electronically ¾ Being able to write over prepared lecture tools like PowerPoint. ¾ Distributing solutions to problem sets and lecture notes. ¾ Drawing diagrams. ¾ Easier to grade papers than track changes [using standard MS Word] which sometimes does not leave space for me to type in the corrections or comments I want to make ¾ Flexibility; replaces chalk and allows an ongoing discussion without the usual chalkboard distractions. ¾ For enhancing lecture notes during class. ¾ Grading papers -- the light weight of the tablet and my ability to download papers into Word and grade using ink annotation lightens my load, literally and figuratively. Hand writing my comments (including drawing circles, arrows, etc.) makes my grading go faster. ¾ Hand grading papers. ¾ Hand writing for non-English. ¾ Hand writing for grading. ¾ I can truly write equations in little time. Easy to send to Blackboard and student can pay more attention in class. ¾ I used the handwriting/annotating feature frequently in lectures using PowerPoint. However, this often crashed, which diminished its usefulness. Still, I found the ability to annotate very helpful and the students also seemed to feel it was helpful. ¾ If it worked well all the time - I would like the writing ability on my PowerPoint presentations and the ability to compose new slides - technical problems have made this difficult. ¾ It is great not to have to turn away from my students to write, and it’s nice to have the many color and pen options on digital ink, which I’ve used to create helpful diagrams (without carrying a bag of colored markers to class). ¾ It is light and facilitates instruction. 77 ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ [I] liked the handwriting component. Posting in-class lecture on line. Teaching statistics -- writing on the tablet instead of on the whiteboard enables me to face the students while teaching. Also, it’s quicker as there’s no need to erase. I just pull up a new page. Another benefit is that I’m not blocking what the students are looking at. The ability to draw graphs freehand and to add content to prepared materials will be a useful attribute. I also plan to make digital recordings of lecture material for in-class and online deliver for next semester’s classes. The ability to write and edit papers while on line. The portability, added features of handwriting to emphasize points in the moment. The use of the tablet is invaluable for grading papers and providing timely feedback to students. This is a great tool, to be able to write mathematics on the computer, save it and send it to my students. Using Microsoft Equation [Editor] or any of the other software used to type mathematics, is very time consuming and not something I would use on a daily basis. Being able to write solutions to problems using the Tablet PC has enhanced the way I can use the computer in the classroom. Using it as a white-board. Using note pad Visual learners benefit by seeing instructor create real-time (diagrams, charts, etc.) Writing and displaying Greek and Hebrew and distributing transcripts of class work. Writing and displaying that writing directly while I FACE the class Writing on my already prepped PowerPoints so that I don’t have to futz with the projector-in-front-ofthe-whiteboard. Writing on PowerPoints. Writing on the screen - I can get back to my chalkboard!! It appears that faculty have used tablet functionality as expected, and generally have a positive response. We asked, “What difficulties, if any, did you experience in using the tablet?” Responses are summarized below in table 9. Table 9: Difficulties Experienced by Participants with the Tablet Difficulty Experienced No. Respondents System hangs/need to reboot Oversensitive mouse -- cursor jumping associated with mouse/touchpad Problems with the external DVD drive Problems with wireless connectivity Screen orientation Don’t like external DVD drive System instability/crashing when annotating in PowerPoint Battery life Problems with network connectivity Problem with USB port Insufficient hard drive capacity 11 7 7 3 2 2 2 1 1 1 1 The difficulties with the external DVD drive appear to be specific to this particular model. The screen orientation difficulty refers to the fact that the image projected on the LCD projectors in the classroom was, by default, inverted; this meant that the instructor had to rotate the image on the tablet 180 degrees – not a difficult process, but one that requires several steps – and turn the tablet upside down. We don’t believe the connectivity issues are distinct from those found using the standard notebooks at SHU. On the other hand, the problems associated with the cursor skipping, and with the system hanging/crashing are in our view more troubling. Certainly encountering these problems during a lecture can be harmful to continuity and focus. We were interested in overall issues relating to the performance of the tablet. We asked the participants to rate the hardware and technical aspects of the tablet along the dimensions outlined in table 10 below. 78 Table 10: Faculty Satisfaction with Hardware and Technical Aspects of the Tablet very satisfied unsatisfied or not neutral or satisfied very unsatisfied applicable 27 11 7 0 memory/hard drive/ processing speed 60.00% 24.44% 15.56% 0.00% 35 3 7 0 size of keyboard 77.78% 6.67% 15.56% 0.00% 34 2 9 0 size of monitor 75.56% 4.44% 20.00% 0.00% 7 6 28 4 external DVD drive 15.56% 13.33% 62.22% 8.89% 41 4 0 0 weight/portability 91.11% 8.89% 0.00% 0.00% 33 7 4 1 wireless access 73.33% 15.56% 8.89% 2.22% 31 6 7 1 battery life 68.89% 13.33% 15.56% 2.22% 5 7 0 33 speech recognition 11.11% 15.56% 0.00% 73.33% 34 7 3 0 digital ink functionality (handwriting) 77.27% 15.91% 6.82% 0.00% 23 10 6 6 converting handwriting to text 51.11% 22.22% 13.33% 13.33% total 45 45 45 45 45 45 45 45 44 45 Dissatisfaction with the external drive shows up here too, and in fact seems to be the only attribute of the tablet about which there was significant unhappiness. Some 90% of respondents seemed pleased with the weight/portability characteristics of the tablet. We suspect this is partly a function of the fact that the standard notebook provided faculty is at the heavy end of the portable computer spectrum. Roughly 75% of respondents were satisfied or very satisfied with the size of the keyboard, size of the monitor, wireless access and digital ink functionality. Converting handwriting to text seems to get a more mixed picture with 23 respondents (roughly 50%) satisfied or very satisfied, and six unsatisfied or very unsatisfied, and six who didn’t use this functionality. Only a few faculty used the speech recognition capability. Overall, are you satisfied with your Tablet PC or would you prefer to switch to the "regular" notebook if that were possible? No. Responses 60 50 All Respondents 40 30 Respondents Who Used Tablet for Teaching and Learning 20 10 0 satisfied does not switch missing with tablet matter to back to PC me notebook Figure 3: Tablet vs. Notebook Preferences Our view is that these results are very positive. This is reinforced by the results outlined in figure 3; it appears that participants overwhelming preferred their tablet to a standard note book. 79 We see, for example, that some 85% of those who used tablet computing in at least one of their classes would prefer to keep their tablet as opposed to returning to a standard notebook. It is worth noting that three respondents of those who actually used tablet technology preferred to switch back to a notebook. These individuals cited the technical problems noted earlier: failure of the external drive, repeated occurrences of the system hanging, and network difficulties. With regard to all respondents, some 80% would stay with a tablet and seven would prefer to go back to the standard notebook. Of these additional four who preferred a standard notebook, one was the respondent whose tablet was “out for repair”, another indicated that he never used the machine and that “it was a mistake” for him to request a tablet, the third simply didn’t use it and the fourth was a novice user on sabbatical who got frustrated by problems with the external drive and a faulty printer cable. We note that tablet PCs are more expensive than conventional notebook computers. Figure 4 below reproduces the question we posed to the participants that brings this fact into decision-making, and provides a summary of the responses. A tablet pc costs about $300 more than a comparable laptop. Do you think the University should continue to provide a tablet pc option for faculty? definitely yes probably yes no opinion probably not definitely not not applicable no response Figure 4: Preferences Regarding the Tablet Option Given the Additional Cost Approximately 85% (38 out of 45) responded that SHU should definitely or probably continue to provide a tablet PC option to faculty. Five said the university probably or definitely should not, and two expressed no opinion. The results in figure 5 highlight perhaps the most important issue here: the value, as perceived by faculty who used their tablet in at least one course, of this technology vis-à-vis teaching and learning. Overall, how valuable was the tablet in terms of teaching and learning? very positive somewhat positive no measurable value somewhat detrimental very detrimental don't know/no opinion not applicable Figure 5: Value of tablet in terms of teaching and learning 80 Forty of the 45 faculty (approximately 90%) who used the tablet in at least one course responded that their use had a very or somewhat positive value with regard to teaching and learning. Two indicated that they observed no measurable impact, one indicated a “somewhat detrimental” effect. This individual cited system crashes and difficulties with cursor jumping associated with the mouse/touchpad. Results We feel that the results of this study may be summarized as follows. There are tablet applications that will be used by some faculty with good results Some faculty did use tablet functionality and their perception of the utility of that use vis-à-vis teaching and learning was positive. Aside from writing non-English characters, the applications considered here were used at least occasionally by 50% - 75% of respondents; further, on average 75% of these faculty rated the impact of this use as positive or very positive. The percentage of faculty who used the applications considered here at least frequently (as opposed to at least occasionally), ranged from 25% - 40%. The tablet applications actually used by faculty are those predicted. We see a good correspondence between the expectations of faculty and their actual usage of tablet applications. Put another way, generally faculty were not disappointed; they were able to do what they planned on doing with the tablets. Negative feelings seem to be most strongly associated with technical problems. The strongly negative feelings regarding the tablet felt by a few of the participating faculty seemed to result from technical problems they experienced. In particular difficulties with the external DVD/CD drive, system instability and cursor jumping were noted. Our feeling is that these are surmountable difficulties, with better support and training, and are not particular associated with tablet technology. Faculty who use tablets are generally pleased with the results and strongly prefer to continue using tablets. We measured faculty satisfaction with tablet computing via several questions, including one that explicitly took into account (as much as was possible) the added cost of tablet technology. Faculty who used tablets were favorably disposed to their usage, felt that the impact of that usage was positive, wanted to continue using tablet technology, and felt all faculty at the university should have a tablet option. The majority of faculty are not “demanding” tablets We see that (only) 30% of the 220 faculty members at SHU eligible for notebook replacement in the fall of 2004 opted to try a tablet. Data for the fall of 2005 reflect an almost identical breakdown, with again about 30% of faculty selecting the tablet option. Of the 64 who received a tablet in the fall of 2004, 59 replied to our survey, and (only) 45 actually used their tablet in at least one class. Using this last figure, it seems that approximately one-quarter to one-third of faculty perceive tablets to be something beneficial for their usage. Conclusions Despite well-intentioned efforts to limit class sizes at many colleges, the lecture remains in place as the primary teaching format. The economics of higher education are such that few institutions can afford to discard lecture courses altogether, however much a small seminar is preferred by students and faculty members alike. With only one paid teacher and a large roomful of tuitionpaying students, how can you beat the numbers? In my view, this is not necessarily a bad thing. Jay Parini “The Well-Tempered Lecturer” The Chronicle of Higher Education Volume 50, Issue 19, Page B5 January 16, 2004 We believe the above results indicate that tablet technology for a faculty should be pursued for those faculties with an interest in an application for which the tablet is well-suited. In particular, it seems that those fields that 81 require freestyle drawing of diagrams, pictures and charts, as well as fields requiring the use of mathematical symbols are good areas for the use of tablets. These fields include many areas in the sciences and the social sciences, including, for example, physics, biology, chemistry, psychology and economics. We believe that the results are consistent and provide reliable pointers towards the usefulness of tablet computing for faculty at the university level. In academia as elsewhere, there are always technology leaders and followers. The ultimate goal here is not to force all faculty members to use a technology, but rather to provide individual faculty with the tools they believe will assist them in their educational mission. With the results of this study in mind, Seton Hall University is pursuing tablet technology, both continuing the option for faculty (as mentioned previously) and equipping students in our honor programs with tablets, in the fall semester of 2005. References Anderson, R. (2004). Beyond PowerPoint: Building a New Classroom Presenter. Syllabus, Retrieved November 22, 2005 from http://www.syllabus.com/article.asp?id=9537. Asay, P. (2002). The Killer App for Higher Education, Syllabus, Retrieved November 22, 2005 from http://www.syllabus.com/article.asp?id=6246. Bentley College (2003). Will Tablet PCs replace laptops on college campuses? Bentley College news release, February 10, 2003. Retrieved November 22, 2005 from http://www.bentley.edu/newsevents/pr_view.cfm?id=910 Condon, R. (2003). Table PCs in Education, National Institute for Technology and Liberal Education, Retrieved November 22, 2005 from http://www.nitle.org/rsrc_issues_tabletpc.php Cooper, B., & Brna, P. (2001). Fostering cartoon-style creativity with sensitive agent support in tomorrow's classroom. Educational Technology & Society, 4 (2), 32-40. Crossland, C. L, Monteith, L. K., Miller, T. K., Brock, L. B., Cuejilo, R., Wheeler, M. A., Viersen, R., & Zielinski, L. (2000). The FORESEE Project: Connecting Communities to Create Competence. Educational Technology & Society, 3 (3), 267-282. Duffy, T. M., & Jonassen, D. H. (Eds.) (1992). Constructivism and The Technology of Instruction, Lawrence Erlbaum Associates, Hillsdale, NJ, USA. Einhorn, B., & Green, J. (2004) Headaches For Tablet PCs. BusinessWeek, March 1, 14. Foster, A. L. (2003). Tablets Sneak Up on Laptops: The Chronicle of Higher Education, 49 (30), A33. Himmelsbach, V. (2004). Nurturing a fledgling market. Computer Dealer News, 20 (5), Retrieved November 22, 2005 from http://www.looksmartcompanies.com/p/articles/mi_m3563/is_5_20/ai_n6129772. ITRC (2002). Microsoft Tablet PC Rapid Adoption Project at UT Austin, University of Texas at Austin, Information Technology Resource Central (ITRC), November 2002. Retrieved November 22, 2005 from http://www.utexas.edu/computer/tabletpc-rap2002/. Kravcik, M., Kaibel, A., Specht, M., & Terrenghi, L. (2004). Mobile Collector for Field Trips. Educational Technology & Society, 7 (2), 25-33. McFall, R., & Dahm, E. (2004). Evaluation of a Prototype of an Electronic Textbook Application. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004, 1530-1535. Microsoft (2002.) Massachusetts Institute of Technology Changes the Face of Education Using Tablet PCs, Retrieved November 22, 2005 from http://www.microsoft.com/resources/casestudies/ CaseStudy.asp?CaseStudyID=13513. 82 Niccolai, J. (2003). ViewSonic, Toshiba Refresh Tablet PC Lines. ComputerWorld, November 19, 2003, Retrieved November 22, 2005 from http://www.computerworld.com/mobiletopics/mobile/story/ 0,10801,87329,00.html. Scheele, N., Seitz, C., Effelsberg, W., & Wessels, A. (2004). Mobile Devices in Interactive Lectures. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004, 154161. TLT Group (n.d.). Definitions and Characteristics of a TLT Roundtable, Retrieved November 22, 2005 from http://www.tltgroup.org/programs/TLTR/Characteristics.htm. Wassgren, C., & Krousgrill, C. M. (2003). Pen-based PCs for the classroom, Electronic file, April 3, 2003, Retrieved November 22, 2005 from http://widget.ecn.purdue.edu/~wassgren/2003_04_03_TLTWorkshop/. Webopedia Computer Dictionary (2004). http://webopedia.com/TERM/t/tablet_PC.html. tablet PC, Retrieved November 22, 2005 from Weitz, R. (1997). Lessons from Notebook/Mobile Computing. Information and Management, 32, 103-111. Wilson, B. G. (1996). Constructivist Learning Environments, Englewood Cliffs, NJ, USA: Educational Technology Publications. Yamasaki, T., & Inami, N. (2004). Training System for Fair Handwriting of Japanese Characters Based on OnLine Character Recognition. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004, 1692-1698. 83 Akpinar, Y. & Bal, V. (2006). Teachers’ Collaborative Task Authoring to Help Students Learn a Science Unit. Educational Technology & Society, 9 (2), 84-95. Teachers’ Collaborative Task Authoring to Help Students Learn a Science Unit Yavuz Akpinar and Volkan Bal Boğaziçi University, Faculty of Education, Bebek, Istanbul, Turkey Tel: (90) 212 3594497 Fax: (90) 212 2575036 akpinar@boun.edu.tr balvolka@boun.edu.tr ABSTRACT Currently available courseware packages for teaching the work unit may not meet different students’ needs. Also, a single teacher, even with tools, is likely to have difficulties and may need cooperation of other teachers in dealing with students’ problems in the learning unit. This research aims to devise a set of computer based tools to meet the diverse needs of learners to comprehend a science learning unit. A model of computer based tools on the learning unit for developing procedural knowledge to solve work problems was developed together with a set of teacher customization and collaboration tools. The main components, developed and implemented in an integrated manner for both students and teachers, are Student Activity Environment, Curriculum Authoring Center, Global Activity Center and Teacher Collaboration Tools. The framework of supporting students through teachers’ collaborative course authoring, considering the different backgrounds of the students and preferred teaching/learning style of teachers/students, was evaluated with students using two different task regimes. The evaluation studies presented encouraging and promising results. Keywords Teacher collaboration, Collaborative task authoring, Learning environment design, Task regime, Work Introduction Science education should teach students to critically evaluate new information in order to interpret and test their hypotheses. Students have to make correspondences between data related to the context and a formal representation of scientific phenomena. They have to organize data and contexts through relationships e.g. if an object travels faster or slower what kind of change occurs in time? What about the force applied to the object? Reasoning and deduction about situations are needed. That requires students to grasp the operational definition of prior concepts and to correctly superimpose existing knowledge about a physical phenomenon on its formal representation. Students arrive in the science classroom with preconceptions that are often contradictory to accepted science thinking. These naive theories may lead to misconceptions and thus may interfere with accepted concept development (Welmar, 1996). Misconceptions are troubling issues for students and teachers in science. This is especially true in school science and physics due to its often abstract nature. Students may have two distinct perspectives of science: One is reserved for the formal learning setting in the classroom while the other is used outside that setting in everyday life (Cadmus, 1990; Tsai & Chou, 2002). For example, students’ misconceptions in a work learning unit, which this study focuses, are identified as follows (Amasci, 2004): (1) Failing to identify the direction in which a force is acting; (2) Believing that any force times any distance is work; (3) Believing that machines put out more work than we put in it: Not realizing that machines simply transform the form of the work we do; (4) Believing that the mass effects the work done under every condition. Effective science instruction helps children distinguish between errors and misconceptions, and attempts to confront learners with situations such as classroom discussion or computer-based visualization in which everyday life explanations aren’t successful and offers them better explanations from science. Visualization of scientific phenomena and laboratory experiences have been important components of the reinforcing and understanding science concepts: Visualizing phenomena through demonstrations, simulations, models, real-time graphs, and video can contribute to students' understanding by attaching mental images to these concepts (Escalade et al, 1996). These visualization techniques not only allow students to observe how objects behave and interact, but also provide students with visual associations that they may capture, and preserve the essence of physical phenomena more effectively than do verbal descriptions (Cadmus, 1990). Computer simulations can help students to understand invisible conceptual worlds of science through animation, which can lead to more abstract understanding of scientific concepts (Hwang & Esquembre, 2003). Quantitative ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 84 data can be manipulated and visualized to help students form a qualitative mental picture. Such complex experiences can help students to identify patterns within simulations, and formulate explanations for phenomena in terms of models and theories. Simulations must not only allow learners to construct and manipulate screen “objects” to explore underlying concepts, but they must also provide learners with the observation and manipulation tools necessary to explore and to test hypotheses in the simulated world (Jonassen, 1996). These support measures typically guide science learners to (1) focus on particular variables of the underlying model, (2) generate hypotheses about relationships between these variables, (3) conduct simulated experiments to test the hypotheses, and (4) evaluate the hypotheses in the light of the observed results. These tasks are demanding but Chandler (2004) points out that interactive activities in science will be useful if they are specifically related to the learning unit and if the knowledge base of the learner is also taken into account. Teachers may customize and relate interactive computer based activities for the learner by considering the knowledge base of the learner. Computer Based Activities and Science Teachers A study focusing on students’ learning problems, should also consider the teachers’ role in overcoming them. When teachers are able to design and alter applications, they will then be better able to address learning problems. According to Dunlop et al (2000; p.15) teachers need (1) to derive information, monitor the use, and utilize the special resources that computer network technology makes available; (2) to have quick and easy access to the information salient to their planning; (3) notification of salient information and feedback about student progress and problems as well as teaching changes, and (4) ways of managing computer group work that allow for a better understanding of the actions of computer-mediated groups and their instructors and a better ability for teachers to make adjustments. Moreover, planning for remote collaborative work needs to be more flexible and relaxed to allow for instructional differences. Computerized learning environments should not be built only by expert programmers. It is necessary to continue developing new types of cognitive environment tools, so that all teachers can participate in the construction of technology rich learning environments. For teachers, a number of sophisticated tools, including commercial products, have emerged to create interactive multimedia software. However, those tools base the interaction around general models of instruction, which unfortunately are too general to serve as a specification for an educational software (Bell, 1999). To better support the needs of the student and author/teacher, Murray (2003: 500) recommends authoring tools tailored for specific tasks or instructional situations. Further, to adapt to the learners' demands, abilities and knowledge many researchers (Hasebrook & Gremm, 1999; Welch & Brownell, 2000) aim at making their tutoring systems more effective using "intelligent" software technologies. Recently, there have been new efforts to transfer the technology of intelligent tutoring systems and authoring tools over the Internet which needs much effort from domain and computer experts. We argue that efforts are best placed in the middle ground between intelligent systems and the conventional, and should be used to develop a procedural learning environment. That extends the conceptions of simulations to the environment descriptions and typically gives a wider range of instructional functions to the user-system communication language. Purpose of the Study Courseware packages should continuously incorporate teachers’ observations and assessment of student understanding in overcoming students’ difficulties and provide teachers with collaboration tools to make courseware facilities fit individual students’ needs. This study presents a new courseware design framework to overcome students’ learning difficulties in a science learning unit. The courseware framework will then be tried out with students and teachers: (1) to evaluate effects of the system facilities for students’ learning, with particular attention to the new interface and the effect of different task regimes, (2) to study classroom teachers’ reactions about a web based knowledge and experience sharing platform as well as collaborative activity development. Design of Procedural Learning Environment: Workers’ World The review of literature both in science teaching and learning and use of technology suggest several requirements to be considered in the design of learning environment for the learning unit, work, which is not a commonly studied subject area in the software research. A variety of teaching and learning approaches should be 85 supported by learning environments (i) to overcome learners’ difficulties in this area, (ii) to avoid developing misconceptions and (iii) to aid the learner in proceduralising his/her interaction with the system and eventually be less prone to errors. A learning environment for studying abstractions should have a concrete object space (Thomson, 1992) which can accommodate activities. The activities reveal real world facts in order to represent the study domain in the environment. The environment should also provide two sets of mechanisms: One is for checking the validity of students’ methods on the activities and the other is for providing feedback on the appropriateness of students’ actions in relation to the presented task. Students can operate the object world in different ways by a student-environment interaction language. Their operations show the effects of their actions and connect to representations of the object space and its relations at higher levels of abstraction. This requires the environment to be able to move its presentation modes from concrete to symbolic as the learners gain competence. Hence the environment may enable a seamless transition (Draper et al., 1991; Kong & Kwok, 2005) from facts to formalism. The learning environment should allow science students to experiment with concepts and procedures in ways that relate to the students’ experience to support guided discovery (Tait, 1994; Smeets, 2005) and directed methods of instruction. Learning in such an environment should be contextualized and procedural through different type of tasks. The teachers may manage task specifications in a curriculum management office (Akpinar & Hartley, 1996). Hence an extension and integration of the students’ learning environment to the teachers’ customization environment is necessary. Because students may need different tasks, teachers may share students’ experiences and task regimes and may cooperate in developing new task regimes. Such collaboration may be realized through web based data manipulation tools which can send data to a students’ environment. Teacher Global Activity Center Collaboration Global Task Learner Record Teacher Experience Pool Repository Repository Teacher Activity Environment Activity Center Curriculum Authoring Center Activity Specification Tools Activity Space Activity Set Activity Handling Tools Screen Design Records Course Authoring Student Figure 1: The architecture of Workers’ World and Teacher Collaboration Environment The learning environment this study focuses upon should regulate the control between the student and the system, accommodate real-life tasks and their solution methods which are rich in feedback and provide interactive illustrations supporting conceptual understanding, learner controlled inspections and problem solving. Collins and Brown (1988) and Thompson (1992) suggest that through the structured procedure capturing systems which offer simple devices, perceptually reflective learning on concrete items can be achieved. Through the students’ manipulation and inspection, aspects of the device structure can then be explicitly represented, annotated and become the subject of didactic discussion. On the basis of a firm conceptual understanding, the student’s actions, as part of a procedure, are to be evaluated and responded to by the computer program in order 86 to provide feedback about the effects the action would have in real world. The student then takes successive action and each time explores more information. Further, because developing a firm conceptual understanding of science domain depends heavily upon the constructive work with real world objects in a science community and because the students need building materials, tools, patterns and sound work habits to learn operational relationships, the functional components of such “procedural environments” should have the architecture given in Figure 1. The learning/teaching environment, Workers’ World, proposed here, consists of mainly four components to provide teachers and students with pedagogical tools in studying the learning unit. These components are (1) Student Activity Environment, (2) Curriculum Authoring Center for Teachers, (3) Teacher Collaboration Tools and (4) Global Activity Center. Student Activity Environment In a procedural learning environment, it is necessary to provide learners with an object-world whose properties represent the study domain. The object world should be associated with operator(s) carrying out procedure(s) in the study domain. The procedures must be completed by a learner on the operators. The environment is to monitor and evaluate these operations. The represented object world should respond in clearly observable ways that are easy to interpret by the students, which will help them to relate previous experiences and informal knowledge to a semiformal representation of the knowledge covered in the environment. To facilitate intensive procedural activities, the Workers’ World provides an activity environment for students. The activity environment has basically five components: Activity Center; Activity Set; Activity Handling Tools (objects and operators); Student Records; and Interface. The Activity Center consists of tasks authored either by the classroom teacher or by other teachers but customized by the class teacher. The class teacher will prepare a set of activities and pre-store them for students’ use in accordance with students’ learning problems and requirements and these activities will be entirely tailored to students’ needs in number and type. The student activity environment also provides tools to handle and complete activity sets. Activity handling tools in the Workers’ World takes representation of the study domain into account. Hence it will be a domain representation environment which represents two basic components as screen objects, namely objects and operators. The objects, prescribed in a given activity, (workers and/or trolley and masses), are represented graphically. The operators are work-meter and road, and are intended for proceduralization of the given task. Once a trolley with or without a specified mass is pushed by a worker on a given road (distance to be specified), a particular amount of work would be produced, measured and represented in the work-meter. All these arrangements will support “making the goal of the user and representation of procedure by the learning environment close” (Schneiderman, 1998). In turn, the user will make a ready transition from his/her planned action into his/her instructions to the environment. Further, the environment makes the results of each user action visible and interpretable. It will also be able to control the users’ attention in order to accommodate exploratory and directed mode of interaction. Figure 2: A task and learner tools in Workers’-World Learning Environment 87 In Workers’-World, the interface consists of three different segments (A task presentation segment, Task objects and operators, A Work-Meter) for students to check activities (see Figure 2): The task objects together with the operator will require students to analyze the given task, to find out what to do with the provided objects and to proceduralize the given task. A presented task may be accomplished typically by the following procedures: (1) Selecting workers and/or a weight; (2) Dragging and dropping selected objects one-by-one at the back (or top) of the trolley on the distance, (3) Executing the push action to the trolley; (4) Observing the push action and reading and noting the work-meter values. The procedure that students may wish to undertake is to apply a force on an object and to measure the quantity of the force. The sequence of operations would be monitored by the environment. They function as components of procedural language which is controlling interactions between the student and the Workers’ World environment. Curriculum Authoring Center for Teachers To enhance active learning, the learning environment should provide different types of tasks. These tasks can be specified and managed through a Curriculum Authoring Center (CAC). The CAC, lesson office for teachers, will allow the learning to be contextualized and procedural in its instructional approach. It will also be capable of setting to the task-needs of a teacher who will utilize his/her experiences with students’ learning of the domain and from previous student records to author a lesson for a particular group of students. Such data exchange is needed to provide students with a set of learning activities and teachers with students’ data. To take more student data (records) into account in the task presentation in activity space and in course authoring, the CAC will exchange data with other facilities. A teacher can use the provided activity specification tools to prepare eight different modes of studying work. The number and order of activities would depend on the teachers. Hence, the teachers will need a set of easy-to-use activity specification tools. The Workers’-World Learning Environment enables a dynamic and visible authoring center with the following Activity Specification Tools (See Figure 3): A text editor 1) A task ordering unit 2) A screen object management unit 2.1) Worker management 2.2) Masses & Weights management 2.3) Labeling management 2.4) Work management 2.5) Feedback management 2.6) Screen layout management 3) A task set management 3.1) Task selection 3.2) Task browsing (downloading) 3.3) Task landing (uploading) Teacher-tool interaction in the environment is managed through a direct manipulation type of interface. A teacher can write up tasks in contexts and select order of tasks, workers, weight-objects, force and students’ screen layout through drag and drop activities. Insertion, deletion, extraction and saving of text, numbers and screen objects may be handled in the CAC. A single task or a set of tasks can be created in the lesson office or imported from the global activity center. Each instructional screen object in the CAC has the same fundamental properties. When an activity (task) is viewed by a learner, certain objects will be visible or invisible: Visibility conditions are set by the teachers. For example, whether a student should be allowed to modify the property of the road traveled by the worker in a given situation is adjusted prior to task presentation. Activities prescribed in the lesson office form the instructional curriculum for a particular group of students. The learners must manage this curriculum in an intensive interactive manner. The task (activity) specification managed through the interface of the lesson office by the teacher must be able to support tasks that are exploratory in nature or more directive in their presentation. The sequence of different mode of activities is to provide scaffolding where subsequent task relates to the students’ previous experience. Further, help or procedural hints and feedback should be attached to a particular activity. For each activity, the lesson office places its prescription on a screen that permits tagging of the problem, including text describing the problem context and guidance to the learner. The answers are also given to the system and the teacher can provide interpretation and overall task feedback by 88 using the response component. Hence, such flexibility in the environment will allow for use with individual learners, small group demonstrations, or paired studying as determined by the classroom teacher. Figure 3: Activity Specification Tools in GTP Global Activity Center and Teacher Collaboration Tools In conventional learning environments, a teacher plays an important role in determining what and how students learn through activities. Teachers are responsible for monitoring the flow of each student’s activities, playing a meta-cognitive function for the students by probing their knowledge and reasoning, monitoring participation and student engagement. Student activities must be rich and need-based so that teachers make their educational diagnosis and intervention accordingly. As student needs vary and those needs may be fulfilled with different task regimes, it would be important for teachers to have access to a large activity pool which is constructed and enriched by teachers. The Workers’ World accommodates such an activity pool managed within a Global Activity Center (GAC). Because the Internet provides a means of easy communication and information exchange platform, teachers can collaborate to construct and tailor tasks, based upon their students’ performance records. A set of online tools and resources support teachers as they, in turn, support the students through activities. This set of online resources outline a suggested sequence of activities based on what has worked in the past; each activity listed is linked to additional information regarding the purpose of the given activity, an elaboration of what the activity entails, and tips for when to intervene. It also provides a teacher with practical strategies for how to guide students’ work. To accomplish these functionalities and to make them an Internet communication medium, this study constructed a three-module collaborative work platform for teachers: Global Task Pool (GTP), Experience Repository and Learner Record Repository. Global Task Pool: It is a database of tasks prepared by teachers and uploaded to the system. It contains tasks, objects of tasks, operators to be used for tackling each task, prompt for each task and level of students for whom tasks could be used. Each submitted task may be associated with a series of other tasks, hence allowing the prescription of a task space for a (group of) student(s). Teachers can comment on tasks and share those comments. The task pool (see Figure 4) would be enriched by the contributions of other teachers. Critique of and suggestions about a particular task or task regime would enable teachers to work with better quality tasks, constructly validated by colleagues. An important characteristic of study tasks which are used in this type of 89 learning and teaching environment is the authenticity of the activity. Teachers’ collaboration will help produce more authentic activities and GTP will store them for further usage. Figure 4: Access to Global Task Pool Learner Record Repository: One of the benefits of computational learning environment is to keep record of students’ performances in a given learning activity. When those performance records are taken into account in developing new or altered activities, the learning environment is more likely to respond to students’ needs. When teachers customize learning activities, they can refer to students’ records provided by other teachers from different school(s) in the same or different regions, and have an idea of similar students’ behaviors over a series of tasks. Such a learners’ record repository would be more helpful to teachers when the records contain information about students’ learning difficulties and teachers’ experiences about overcoming those difficulties and information about student reactions. Experience Repository: The Global Task Pool provides teachers with a set of tools in order to share their experiences about student reactions to a task or task regime. Once a teacher uploads his/her tasks used in a classroom to the GTP, the teacher may also share her experiences and observations about the students’ performances and learning outcomes. These could be the result of comprehensive analyses of particular students’ learning process of work or an informal description of learner-task interaction. Further, the Experience Repository can store information about students’ task manipulation and learning styles, actions after an activity, the type of intervention they needed, and the type of additional help they required. Fast communication means of Internet may help to form an activity building community. Then, colleagues can share a rich learners’ profile and a large number of authentic tasks. The success of GAC to a large extent depends on the quality of teachers’ communication. The design of the GAC should encourage participant teachers to engage in developing meaningful practices through collaborative processes. The GAC will develop a climate where commenting on each others’ work and giving and receiving feedback are integrated and routine part of the collaborators’ work. Implementation and Evaluation of the Workers’-World Environment The entire teaching and learning environment was developed in different applications: The Student Activity Environment and the Curriculum Authoring Center were constructed using Macromedia Flash. The Global Activity Center and teacher collaboration tools were constructed with Microsoft .net platform because the GAC needs database relations and other server-side requirements. The first two systems consist of main functions and objects which control their operations. The system also works with an XML database. All the data shown to the user is extracted from the xml file with “isle” function, the file will then be parsed to the data resource, and the whole data are copied to an array written in ActionScript code. Flash platform aids with the main Keyframe construction and its definition of functions. “Execution_OK”, “Ask_Task”, “Event_Time”, and “Save_Act” are the main functions of the learning environment and are defined on the main Keyframe. For example, each task is derived from the data array through Ask_Task function. 90 Sample To evaluate the effects of the system facilities for students’ learning, a pilot evaluation was carried out with 69 seventh grade students (35 male and 34 female; average age is 14) from four different secondary schools. Four different groups of students were randomly assigned to the two task regimes. The first task regime group consisted of 39 students (20 male and 19 female) and the second task regime group consisted of 30 students (15 male and 15 female). In addition, four science teachers from four different schools participated in the study. Procedure In the evaluation study, particular attention was paid to the effect of different task regimes. Prior to the experiment with the students, the teachers met in the researchers’ office and received a two hour training about the study on how to use the Workers’ World environment online and offline, CAC and other facilities of the software set. They were also asked to select an online partner among other teachers to prepare a task regime for the Workers’ World. They were then instructed to prepare tasks in all modes of the software in use. One pair of teachers was instructed to author a task regime with items requiring qualitative thinking and worksheet support. The other pair was instructed to author a task regime with different tasks. Two different task regimes were prepared in the CAC by the teachers in two weeks time and checked by the researchers. The first task regime used only the courseware facilities, the second task regime with more qualitative items used a worksheet as a support material to the courseware facilities. In the first task regime, there were ten tasks; six of them were in essay format, three were in multiple-choice format and one was in fill-in-the blank format. The first nine tasks required employing the given software facilities for the students to solve and to proceed. The final task, fill-inthe blank, required to form the formula of work problems using the given variables. The second task regime, however, had tasks more qualitative in nature. This task regime had nine tasks; six in essay format, two in multiple-choice format and the final one was in fill-in-the blank format. A pretest and a posttest, both in multiple choice format, were prepared to measure student progress after using the learning unit. Each test contained items on the basis of the lesson objectives as to describe the relationships between; force and work, distance and work; force, distance and work; masses and work; and as to derive the formula for solving work problems. The two task regimes in the Workers’ World environment were then administered to the assigned groups. The pre and post tests and the treatment took place in computer laboratories of the schools from which the sample was drawn. Four sessions (two for each task regime) were organised. In both task regime treatments, prior to the treatment, the pretest was administered in ten minutes. Just after the pretest, the researchers explained the sample how the software facilities to be handled. Then the students were allowed to study the task regimes for an hour. Although an hour was allocated to the study of task regimes, the second task regime was studied in about fourty minutes in average. The students had to finish all of the tasks correctly by using the software facilities, even when a student predicted the answer. Therefore, all students completed all tasks in both task regimes. During the study of task regimes, no help to find the correct answer was provided. One of the researchers and one of the teachers (class teacher who was a member of task authoring group) were present during the treatments. Following the task regimes, all students were given a paper and pencil posttest similar to the pretest. In addition to the teachers’ observations of the treatments, the data for the study was the pre and post test results shown in Table 1. To see whether there is any significant difference between the students who studied the first and second task regimes, a t-test was conducted (Table 2). 39 Table 1: Pre and post test data Pretest Pretest Standard Mean Deviation 1,690 1,303 30 2,135 n First task regime: Without worksheet Second task regime: With Worksheet 1,084 Posttest Mean 2,370 Posttest Standard Deviation 1,238 2,200 1,288 The data representing the samples’ pre/post-test scores differences was indicated as mean difference in Table 2. It was found that there was a significant difference (t(2.084, 67) = 0.043, p<0.05) between seventh graders’ achievement in “work learning unit” studied through Workers-World CBL environment with worksheet support and without worksheet support. It seems that both task regimes interacted with students’ learning; however the 91 first task regime helped students more than the second task regime with a worksheet. This should not be interpreted as simply the CBL facilities are sufficient to learn the work unit, but it points to the fact that the varied use of facilities and customization of facilities to the students are important. Table 2: Pre and post test mean differences Mean Standard n Difference Deviation First task regime: Without worksheet 39 ,680 1,280 Second task regime: With Worksheet 30 ,065 1,201 Standard Error Mean ,205 ,219 Following the study, the researchers met the four teachers who collaboratively authored the task regimes, and the results of the study were shared with them. They were asked to be critical to provide information on the system so that the revisions can be made. The teachers’ comments on the sample students’ progress were positive but they pointed out that the students should be exposed to more number of both qualitative and quantitative learning tasks in a period of time longer than the study allocated. The idea of collaboration with other teachers was found interesting and useful. Information exchange and discussion specific to a learning task was found more useful than discussion lists based on general domains such as science education, where many teachers state their opinions and raise issues that generally result in superficial discussions. It was noted that thorough discussions specific to a learning task may be systematized through tools similar to the Workers’ World. However, the study with teachers should be verified with systematic tools in more comprehensive studies. Discussion and Conclusions The findings demonstrate that both task regimes have effects on students’ learning; however, the first task regime helped students more. In the first task regime, the students studied the first and second tasks to relate a force to “work” in a given construction-site work by measuring the workers’ forces. The third task aimed to reveal direct proportionality between force and work concepts; the students observed that workers’ forces are directly related to their work. The ratio relation between workers and their forces was studied in the fourth task where one to one correspondence between the variables was established by the students. The fifth task focused on work and distance relationship; to simplify the world in which only one worker, force, was given and the students had to fill out a distance-work table. Similarly, the sixth task required comparing different work values on different distances and the students completed a distance-time table. Further, the seventh task enhanced matching force, work and distance relations. The eighth and ninth task focused on the effect of mass on work, showing that machines simply transform the form of the work we do. In the ninth task, a worker had to carry 3, 5 and 12 kg masses with the help of the trolley. The students measured the work, and then had to fill in the table cells with these measurements. All required cell results were 350 Joule. Converting the relationship between the variables into an abstraction, building the equation, was studied in the last task where force, work, distance and arithmetic symbols of addition, subtraction, multiplication, and division as well as the equal sign were given. In this task, if students requested, the work mechanism was also visible. Although the sample’s progress from pretest to posttest is small, the software facilities seem to, to a certain extent, help students overcome learning difficulties and contribute to students’ learning. The Workers’ World facilities allowed students to visualize the relations between force and work, also helped most students to see that machines simply transform the work that we do. The students identified patterns of distance, force, work and mass within the environment. They further formulated the relationship between the variables. If the teachers benefited from an experience repository, a learners’ record repository and a task pool, they may have prepared different task regimes, which may help students more. The Workers’ World facilities were used by the students on an individual basis and on a limited time period. This is why the students’ communication and disposition of inquiry in classroom discourse was not possible. The only interaction students participated in was with the courseware facilities for about ten tasks. This may be a drawback of the study, however, it seems that the software facilities used even in a short period provided students with new ways of reasoning and various sense making tools. The progress in students’ scores from the pretest to the posttest and the two different results for the two different task regimes show that varying task regimes should be employed to help learning. In addition, the tools such as student-record repository and activity center enable developing the most appropriate task regimes for particular group of students. 92 The collaborative communities provide opportunities for teachers to reflect deeply and critically upon their own teaching practice on the content they teach and on the experiences and backgrounds of the learners in their classrooms (Putnam and Borko, 1997; 1247). Briscoe and Peters’ (1997: 61) findings further indicate that “collaboration facilitates change on teachers because it provides opportunities for teachers to learn both content and pedagogical knowledge from one another, encourages teachers to be risk takers in implementing new ideas, and supports and sustains the processes of individual change in science teaching”. The teacher tools designed in this study may serve the purpose as pointed out by Jauhiainen et al (2002) who reported that the courses and activities that were most valuable for day-to-day teaching were those where teachers could cooperate, reflect and plan with each other in small groups. In a recent and more comprehensive analysis of the teacher networks, Zhao and Rop (2001) report that the teacher networks were claimed to have a number of positive effects on their participants such as reduced teacher isolation, enabled curriculum development, facilitated dissemination of information, and provided easy access to curricular materials. To enable teachers to use computer mediated communication (CMC), this study showed again that not only general purpose tools like e-mail and discussion lists, but also content specific, tailorable and flexible courseware may be designed and used by teachers. Tools similar to Workers’ World would develop teacher reflective discourse communities, because the real value of CMC may lie in information sharing around a set of materials and sharing student data which could be the focus of discussion and reflections. When similar student-facilities specific to learning tasks increase in number and quality, teachers may then enrich their reflective discourse in different learning units of varying content areas. The study developed, implemented and evaluated a set of software environments with Student Activity Environment, Curriculum Authoring Center, Teacher Collaboration Tools and Global Activity Center. These components were built according to the recommendations of previous studies for collaboratively set learning environments. The studies with students and teachers validated the approach. However, the student activity environment may be enriched with new task models. The models may include script flexibility for expert teachers or programmers. The system may support external objects with predefined standards (e.g., SCORM compliant), and include a learning object or other multimedia objects as video segments, pictures or animations. Possible external objects may include updates or support for other learning units. They may also be linked to a new science unit. In this sense, the student activity environment may be used for only its core functions such as student record repository, learning object adaptivity, portability and feedback mechanism. The feedback in the student activity environment may be changed in several ways: If there is more than one type, it should be under teacher control. Further studies should investigate the different usage of the facilities and extensions. Prime interest in those research is to collect data at the operational level (e.g., to what extend are students and teachers able to use the system and do the facilities motivate them?) and at a reflection level (e.g., is the learning environment encouraging tactical thinking?) so that meta-cognitive and problem solving skills can become part of the learning objectives. Teacher collaboration tools should further be studied to increase experience-sharing among teachers in terms of similar activity center applications. Teacher actions in the center may be observed for further expansions of the system. The collaboration tools may be strengthened with the new online messaging tools. Task regimes and worksheets may be adapted for collaborative learning strategies to increase efficiency of the learning gains. Acknowledgements We would like to thank three anonymous reviewers and Dr. Gunizi Kartal for comments on earlier drafts of this paper. References Akpinar, Y., & Hartley, J. R. (1996). Designing interactive learning environments. Journal of Computer Assisted Learning, 12 (1), 33-46. Amasci, R. (2004). Errors and science myths, retrieved January 9, 2006 from http://www.amasci.com/miscon. Bell, B. (1999). Supporting educational software design with knowledge-rich tools. International Journal of Artificial Intelligence in Education, 10 (1), 46-74. 93 Briscoe, C., & Peters, J. (1997). Teacher collaboration across and within schools: Supporting individual change in elementary science teaching. Science Education, 81 (1), 51-65. Cadmus, R. R. Jr. (1990). A video technique to facilitate the visualization of physical phenomena. American Journal of Physics, 58 (4), 397-399. Chandler, P. (2004). The crucial role of cognitive processes in the design of dynamic visualizations. Learning and Instruction, 14 (3), 353-357. Collins, A., & Brown, J. S. (1988). Computer as a tool for learning through reflection. In Mandl, H. and Lesgold, A. (Eds.), Learning Issues for ITSs, Berlin, Germany: Springer Verlag, 1-18. Draper, S. W., Driver, R. J., Hartley, R., Hennessy, S., Mallen, C., Mohamed, R., O’Malley, C., O’Shea, T., Scanlon, E., & Twigger, D. (1991). Design considerations in a project on conceptual change in science. Computers and Education, 17 (1), 37-41. Dunlap, D. R., Neale, D. C., & Carroll, J. M. (2000). Teacher collaboration in a networked community. Educational Technology and Society, 3 (3), 1-17. Escalada, L. T., Grabhorn, R. P., & Zollman, D. A. (1996). Applications of interactive digital video in a physics classroom. Journal of Educational Multimedia and Hypermedia, 5 (1), 73-97. Hasebrook, J. P. & Gremm, M. (1999). Multimedia for vocational guidance: Effects of individualised testing videos and photography on acceptance and recall. Journal of Educational Multimedia and Hypermedia, 8 (4), 377-400. Hwang, F. K. & Esquembre, F. (2003). Easy java simulations: An interactive science learning tool. Interactive Multimedia Electronic Journal, 5 (2), retrieved May 1, 2006 from http://0-imej.wfu.edu.csulib.ctstateu.edu/ articles/2003/2/01/index.asp. Jauhiainen, J., Lavonen, J., Koponen, I., & Kurki-Suonio, K. (2002). Experiences from long-term in-service training for physics teachers in Finland. Physics Education, 37 (2), 28-35. Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking, Englewood Cliffs, New Jersey, USA: Prentice-Hall. Kong, S. C., & Kwok, L. F. (2005). A cognitive tool for teaching the addition/subtraction of common fraction: A model of affordances. Computers and Education, 45 (2), 245-265. Murray, T. (2003). An overview of intelligent tutoring system authoring tools: Updated analysis of the state of the art. In Murray, T., Blessing, S. & Ainsworth S. (Eds.), Authoring tools for advanced technology learning environments, Dordrecht, The Netherlands: Kluwer, 491-510. Putnam, R. T., & Borko, H. (1997). Teacher learning: Implications of new views of cognition. In Biddle, B. J., Good, T. L. & Goodson, I. F. (Eds.), The international handbook of teachers and teaching, Dordrecht, The Netherlands: Kluwer, 1223-1296. Schneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction, Boston, MA, USA: Addison Wesley. Smeets, E. (2005). Does ICT contribute to powerful learning environments in primary education? Computers and Education, 44 (3), 343-355. Tait, K. (1994). Discourse: The design and production of simulation based learning environments. In Jong, T. de & Sarti, L. (Eds.), Design and production of multimedia and simulation-based learning material, Dordrecht, The Netherlands: Kluwer, 113-131. Thompson, P. W. (1992). Notations, conventions and constraints. Journal for Research in Mathematics Education, 23 (2), 123-147. 94 Tsai, C. C., & Chou, C. (2002) Diagnosing students' alternative conceptions in science. Journal of Computer Assisted Learning, 18 (2), 157-165. Welch, M., & Brownell, K. (2000). The development and evaluation of a multimedia course on educational collaboration. Journal of Educational Multimedia and Hypermedia, 9 (3), 169-194. Welmar, H. G. (1996). Assessing the impact of computer-based learning in science. Journal of Research on Computing in Education, 28 (4), 461-85. Zhao, Y., & Rop, S. (2001). A critical review of the literature on electronic networks as reflective discourse communities for inservice teachers. Education and Information Technologies, 6 (2), 81-94. 95 Gülseçen, S. & Kubat, A. (2006). Teaching ICT to Teacher Candidates Using PBL: A Qualitative and Quantitative Evaluation. Educational Technology & Society, 9 (2), 96-106. Teaching ICT to Teacher Candidates Using PBL: A Qualitative and Quantitative Evaluation Sevinç Gülseçen Faculty of Sciences and Informatics Department, Istanbul University, 34452 University, Istanbul, Turkey gulsecen@istanbul.edu.tr Arif Kubat Faculty of Economics and Administrative Sciences, Inönü University Malatya, Turkey ABSTRACT The idea underlying this study is that the prospective teachers develop their Information and Communication Technology (ICT) skills throughout the learning process supported by the Problem Based Learning (PBL), a method that produces independent learners who can, then, continue to learn on their own in their chosen careers. The major goal of the study, conducted in two phases during two academic years, is to investigate the differential effects of PBL and conventional teacher-centred instruction on cognitive and a group of affective variables which have been coined “technophobia”. Phase I of the study, entirely qualitative in character, covered a case study in which a total of 111 students from three departments of the Faculty of Education took part. Phase II of the study was quasi-experimental in nature with 79 students taking part. Most of the outcomes from Phase II were based on statistical measurements. At the end of the instruction process, the findings indicated that, while there was no significant difference in the anxiety levels of the two groups, the number of successful students had registered a considerable increase. Furthermore, the students, actively involved in the learning process, solving real problems, viewed the PBL as an effective learning tool rich in motivation. Within this framework, the motivating role of ICT as a method of learning is automatically recognised. Keywords Problem Based Learning, Information and Communication Technology, Cognitive Tools, Improving Clasroom Teaching. Introduction In general, when the related literature is reviewed, it appears that several international and national studies point in the direction of the central role teachers play in the implementation of educational change. In particular, however, the main concern is the integration of Information and Communication Technology (ICT) into the teaching process. According to Fullan (1991), “Educational change depends on what teachers do and think – it is as simple and as complex as that.” On the other hand, while only a few teachers have had the opportunity to develop more than the basic skills in the use of ICT, even fewer have developed confidence and competence in the integration of ICT into their classrooms (Orhun, 2002a; Albion, 2003). A 1992 OECD report entitled, “Education and New Information Technologies: Teacher Training and Research” emphasizes in its conclusion that, “...the potential of the new ICT for improving learning and teaching will not be realised unless teachers are well trained and retrained in the pedagogical use of technology in the classroom (Orhun, 2002b: Yıldırım & Kiraz, 1996). While the ICT receives wider acceptance in the field of education, computer anxiety remains to be a challenging concern. Generally speaking, some teachers still exhibit a certain degree of anxiety toward computers as a tool to be employed in the fields of education and learning. Yıldırım and Kiraz (1996) state the fact that even the utilisation of e-mail is adversely effected by the negative approach of some teachers towards it. It is obvious that the teachers themselves must have necessary training and support to properly integrate the related technology into their practices even before employing methods and techniques to improve the process of teaching and learning. In particular, the teachers are to be healed of the anxiety of computers utilised in such processes. Moreover, the virtues of self-confidence and positive approach toward computers, as well as belief in the ability to employ them in an effective and efficient manner should be instilled in the teachers. Computer-based cognitive tools have a higher potential to foster meaningful learning, which requires a constructivist view of learning (Jonassen, 2000; Gibson & Silverberg, 2000). According to Jonassen, Howland, Moore and Marra (2003), constructivism, like nearly all contemporary theories of learning (situated learning, social cognition, activity theory, distributed cognition, ecological psychology and case-based reasoning) share ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 96 convergent beliefs about how people naturally come to know. One of the best exemplars of a constructivist learning environment is Problem Based Learning (PBL) as described by Barrows who was involved in the early stages of the development of PBL at McMaster University in Canada (Savery & Duffy, 1995; de Graff & Kolmos, 2003). He defines the concept in terms of its specific attributes such as being student-centred, taking place in small groups with the teacher acting as a facilitator, and being organised around problems. Liu (2004) suggests that the aspect of anchoring learning in real world contexts in PBL has been found to provide opportunities for transferring knowledge and skills from the classroom to workplace more easily. According to Albion (2003) teachers’ use of ICT may be influenced by their self-efficacy beliefs, that these, in turn, may be enhanced by access to examples of successful practice, and PBL may provide an especially effective vehicle for this purpose. We also believe that the attitudes of the teachers can be influenced and changed positively during the early periods at the teacher training institutions. By creating a learning environment which promotes professional development, teacher candidates should be made aware in terms of the ways in which they could benefit from existing ICT tools. The main goal of the study presented in this paper was to investigate the differential effects of PBL and conventional teacher-centred instruction on performance of teacher candidates of the Faculty of Education, Istanbul University (hereafter Faculty of Education) and a group of affective variables such as “technophobia” as coined by Rosen and Well (1992), i.e., Computer Anxiety Rating Scale (CARS), Computer Thoughts Survey (CTS) and General Attitudes Towards Computers Scale (GATCS). The performance and attitudes of students in the PBL curriculum has been compared with that of the students in the standard curriculum longitudinally over two years. The rest of the paper is organised in four sections. Based on the literature review in “General Rationale and Hypothesis” section, the basic definitions for constructivism and PBL are provided in that order, followed by the main hypothesis of the study. Subsequently, Phase I and Phase II detail a model of PBL program used in a second-year course on “Introduction to ICT” at the Faculty of Education taught by same instructor (the first author of this paper) during the academic years 2001-2002 and 2002-2003 respectively. Both phases introduce an “experiment” in developing ICT skills of the second-year students by having them design a multimedia presentation for their future students, working in groups undertaking PBL-based project work. While Phase I is entirely qualitative, Phase II is more quantitative and much of what is described in these two sections stems from students evaluation and feedback obtained over two academic years. A general discussion and conclusion follow in the last section. General Rationale and Hypothesis The Context Providing schools with ICTs promises a high return on investment. The presence of computers and Internet access raises ICT literacy and skills, better preparing the future generations to participate in the information society. To this end, the schools represent ideal access points because they cover a large part of the population, especially in developing countries. Connecting schools through technology also bring online that part of the population that can learn quickly how to use ICT (World Telecommunication Development Report, International Telecommunication Union, 2003). The education policies and systems of Turkey, a country with young population, are being revised on an ongoing basis so as to train competent and fully equipped teachers challenged to meet the needs of eight years of elementary compulsory schooling recently put in place. Orhun (2002b) stated that main part of Computer-Aided Education (CAE) project was initiated in 1990-1991 as part of the World Bank National Education Development Project. It included a program to introduce computer literacy and computer aided instruction in grade 10 at selected secondary schools. A report by the Research, Planning and Coordination Board of the Ministry of National Education (http://www.meb.gov.tr) titled, “The National Education at the Beginning of 2002”, it is stated that “... using the credit received from the World Bank, 3188 IT classrooms have been built in 2802 schools, tenders have been opened to build more of them, and to obtain software in 3000 more primary schools.” The Faculty of Education is one among several others to meet the ever increasing demand of teachers and educational specialists in Turkey. The initial teacher preparation involves the development of discipline-based knowledge, curriculum, and pedagogy skills as well as professional ethos. A course named “Introduction to ICT” has also been included as a compulsory course in the curriculum. The history of the course reaches back to 1999 97 when the Faculty began receiving students. At the end of the course, students are expected to gain specific ICT skills and knowledge which they will then transfer on to the workplace through opportunities provided. Most of the students come from the middle or lower classes with no or very little access to technology. Although the students seem to be motivated and keen to learn when introduced to the topic itself, a certain level of anxiety stemming from their fear of what is technological is also noted. It cannot be said that the overall attitudes of the students concerning computers, their application, or impact on the society is negative. Nevertheless, the fact is that the dominant feeling in most students’ approach to technology is one of anxiety based on a technophobic tendency pointed out earlier in this paper. As indicated by Weil and Rosen (1995), the problem of technophobia is fairly common. Erktin and Gülseçen (2001) define it as an important psychological factor efficacious in the use of computers in education. According to Rosen and Weil (1992), the technophobic is an individual evidencing at least one of the following, ranging from severe reactions to mild discomfort: anxiety about present or future interactions with computers or computer related technology, specific negative cognition or self critical internal dialogues during actual computer interaction or when contemplating future computer interaction. During the academic year 1999-2000, the “Introduction to ICT” course has been taught using traditional weekly lectures and laboratory sessions throughout the semester four hours a week. By the end of the semester, it was obvious that there was not sufficient time available to cover the entire content of the course; individual students were at varying levels of knowledge and unhappy with this fact; many students suffered the same level of technophobia as they did in the beginning; the instructor was dissatisfied with her performance – as there were too many students in one class, it was impossible to interact equally with every student on one-to-one basis causing an adverse affect on the outcome. Moreover, both the instructor and the students themselves were dissatisfied by the performance of the students who exhibited signs of lack of motivation. In order to alleviate the problem of unmotivated students, a plan could be brought forth to overcome the lack of time, as well as to facilitate the learning process. Designing an interactive and highly effective learning environment conducive to cooperation, self-assessment, and provision of prompt feedback would allow for better opportunities for students taking into account their personal learning preferences. According to Frank et al. (2003), self-direction, a passion for learning, and strong individual responsibility are important influences on achievement. Therefore, in order to allow the students to learn different ways in pace with their individual nature while encouraging them to evaluate their own way of learning, a more flexible approach to teaching was adopted. Such an approach requires the instructor to change as well. To wit, the teacher is to cease being the center of attention and the source of all knowledge, becoming instead a coach, a facilitator of the knowledge to be acquired. The next section explains, PBL has several distinct characteristics that may be identified and utilised in designing such environments. Constructivism and PBL Constructivism as a philosophical view on how we come to understand or know, holds that any so-called reality is the mental construction of those who believe they have discovered and investigated it. Its view is characterized in terms of three primary propositions. Knowledge is in our interactions with environment. Cognitive conflict is the stimulus for learning. Understanding is influenced through the social negotiation of meaning (Saunders, 1992; Savery and Duffy, 1995). From these three propositions, a set of instructional principles that can guide the practice of teaching and the design of a learning environment emerged. These principles are, anchoring all learning activities to a larger task or problem, supporting the learner in developing ownership for the overall problem or task, designing an authentic task and the learning environment to reflect the complexity of the environment, giving the learner ownership of the process used to develop a solution, designing the learning environment to support and challenge the learners’s thinking, encouraging testing ideas against alternative views and alternative context providing opportunity for and support reflection on both what is learned and on the learning activity (Savery and Duffy, 1995). As stated in the literature since its first prominence in the late 1970s, PBL has lent its increasingly important voice to the ongoing debate on how to organize teaching and learning at the universities. It restructures traditional instructor/student interaction to emphasize active, self-directed learning by the student, rather than didactic, teacher-directed instruction (Harland, 2002; Maxwell, Bellisimo, & Mergendoller, 2001). According to Driessen and Vlueten (2000), PBL is characterized by problem-orientation, interdisciplinary work and selfdirected learning and focuses on interpersonal and professional skills. Having taken its place among the curricular innovations hotly debated in the circles of the higher education over the last 30 years, learners exposed the PBL are gradually allowed to acquire more and more responsibility, who, then, become increasingly independent of the teacher in terms of their own education. Providing educational materials and guidance that facilitate learning are the responsibilities of a PBL instructor. Several studies have shown that PBL is a 98 successful method compared with the more traditional curricula with regard to internal motivation and long-term retention of the learned knowledge. It brings about procedural changes in a direction that can support a shift in perspectives from teaching to learning (Dahlgren, 2003). The PBL covers any learning environment in which the problem drives learning. That is to say, before students acquire knowledge, they are given a problem so posed that the students discover that they need to learn something new before they can solve the problem at hand. Tse and Chan (2003) assert that in the PBL environment, students act as professionals and confront problems as they actually occur – with fuzzy edges, insufficient information, and a need to determine the best solution possible by a given date. The primary distinction is the focus on introducing concepts to students by challenging them to solve a real world problem. Duch (1997) states that in contrast to the more traditional approach of assigning an application problem at the end of a conceptual unit, the PBL uses problems to motivate, focus and initiate student learning. A critical factor in the success of PBL is the problem itself. PBL problems should strive to induce students to learn at the higher Bloom levels (see Table 1), where they analyze, synthesize and evaluate rather than simply define and explain. A Level 3 problem is a good PBL problem, at Bloom’s Analysis, Synthesis or Evaluation levels. It is related to the real world, drawing the students into the problem. Table 1. Bloom’s cognitive levels and student activities (Bloom, 1956) Bloom’s Cognitive Level Student Activity Evaluation Making a judgment based on a pre-established set of criteria Synthesis Producing something new or original from component parts Analysis Breaking material down into its component parts to see interrelationships/hierarchy of ideas Application Using a concept or principle to solve a problem Comprehension Explaining/interpreting the meaning of material Knowledge Remembering facts, terms, concepts, definitions, principles The characteristics that may be identified and utilized in designing a PBL curriculum are as follows (Savery & Duffy, 1995; Tse and Chan, 2003): 1. Reliance on problems to drive the curriculum – the problems do not test skills; they assist in development of the skills themselves. 2. The problems are authentically ill-structured – there is not meant to be one solution and, as new information is gathered in a reiterative process, perception of the problem, and thus the solution, changes. 3. Students solve the problem – lecturers are coaches and facilitators. 4. Students are only given guidelines for how to approach problems – there is no one formula for student approaches to the problem. 5. Authentic, performance-based assessment – this is a seamless part and end of the instruction. Hypothesis During the first meeting with the students at the beginning of academic year 2001-2002, a female student (Cigdem) felt extremely insecure with the idea of using a computer and said, “I am afraid to break it?” Also the students encountered difficulties stemming from lack of basic technical skills such as using the keyboard. One of the male students (Hasan Ali) expressed this difficulty as, “I will never be able to get familiar with all keys on this keyboard!” The fear and anxiety reached to such a degree that some students were compelled to ask instructor’s guidance during what is their first meeting with the computer. The reactions of Cigdem and Hasan Ali, also observed in most of the students during their first experience with the computers, was the moment the instructor made the decision to leave conventional teaching and shift to the PBL model of instruction to provide the students with a more effective learning environment. Phase I Method Participants The sample of the study consists of a total of 111 sophomore students (28 female and 83 male at average age of 21) from the Department of Primary Education (Division of Classroom Teaching and Division of Science 99 Teaching) and Department of Foreign Languages Teaching (Division of English Language Teaching) respectively. At the beginning of the course, it was observed that the students, exhibiting various levels of anxiety concerning the usage of computers, were also at different levels of basic computer knowledge. In order to measure the said knowledge, a simple test comprised of ten multiple choice questions has been applied. In accordance with the results of the test, students have been classified as “Computer Illiterate”, “Moderately Computer Literate” and “Fairly Computer Literate.” The percentage of the “Fairly Computer Literate” was only 3%. As stated before, high levels of anxiety has been observed in vast majority of students who were, then, classified on such basis as “Technophobics” and “Non-technophobics”. Procedure During the first two weeks of the course, the introductory concepts of ICT (basics of computer hardware and software, familiarisation with Windows, and a word processor) were taught at what could be described as conventional lectures. One of the objectives of the said lectures was to eradicate the vast differences of basic technical knowledge determined to have existed among the students followed by the application of the PBL. In order to encourage the students to assume the responsibility and take ownership for the problem, it was presented as a Level 3 problem, classified as “good problem” at Bloom’s higher levels presented in the previous section. The students were, then, confronted with a real world scenario (Tse and Chan, 2003) through a seemingly authentic correspondence by utilizing a Power Point presentation for their future students on a topic relevant to the academic field they have chosen, e.g. language or science teaching, by working in groups of 3 to 5. In the last 10 years, Power Point presentations have become the most prevalent form of multimedia in education as the students prefer it to presentations from transparencies (Bartsch & Cobern, 2003; Jonassen et al., 2003). Not all the information needed was divulged concerning the problem at hand. Students were provided only with enough information about the goals and the process they would go through during the program. They were given a four week free study time to complete the work. De Graf and Colmos (2003) suggest that the PBL education is based on students’ background, expectations, and interests. Thus the topic selection is left to the students themselves without imposing any limit at all with the supposition that the groups would be more enthusiastic about studying a topic of their own choosing. The students were also informed that they needed to research, discover new material, arrive at judgements and decisions based on the information accessed through print (books, papers and magazines), human (other instructors from the Faculty of Education, classmates, and other experts), and electronic information resources (CD-ROMs and Internet web sites). They have especially been encouraged to browse Internet resources to obtain contemporary electronic material (pictures, images, sound effects, visual effects and video clips) and to integrate them into their presentations. Knowledge concerning Power Point presentation was not imparted to the students. They, were, however, encouraged to access computers whenever they can from computer laboratories within the university campus, Internet cafés, student dormitories and homes. During this period, tasks such as collecting materials, learning the related software, preparing the initial design of the presentations and getting feedback from the instructor were successfully performed. At the end of the four week period the presentations were examined and the deadline was extended for two weeks in order to let students to make the necessary modifications and to finalise the presentations. During that time the role of the instructor was that of being a coach and facilitator. Measurements Measurements were based on observations, interviews and group portfolios. Also a test was performed in order to assess the academic achievement of students. At the end of the semester, a performance-based assessment of final presentations was made. Findings and Discussion At the end of the course, all groups were able to complete their presentations from which the following were chosen as the most successful ones: “Substances”, “Holiday”, “Communication Devices”, “Population of our Country” (Classroom Teaching); “Ants”, “Genetics”, “Our Mountains” and “Harmful Insects” (Science Teaching); “Solar System”, “Indians”, “The Ancient Egypt”, “Earthquake” and “Seasons” (English Language 100 Teaching). Assessment of presentations and data that was collected by the instructor using measurements mentioned above revealed a number of themes that have to be discussed. Once students began working with the problem they are confronted with, they gradually identified what they needed to learn. Most of them were engaged in a self-directed study to gather the information necessary to solve the problem. Some visited Internet cafés, others met at the homes of their group-mates with computers. A couple of students visited primary schools in order to obtain information from the school teachers. One group (the “Our Mountains” group) interviewed a science teacher in preparation of their presentation and discovered that she got into trouble with the section describing the mountains because all third graders tried to memorize the text in the book. Another group (the “Communication Devices” group) visited the main post office of the city where there is a museum and obtained a large amount of material about early predecessors of contemporary communication devices in order to insert them into their presentation. In a PBL model of instruction, the focus is on learners as constructors of their own knowledge within a context similar to the context in which they would apply that knowledge. As expected, during the self-study period, students taught both critically and creatively and monitored their own understanding. Most of the groups prepared several additional slides including an explanation of initial preparations and characteristics of the presentation imparted (see Table 2). It was observed that the students worked interactively rather than dividing the workload on an individual basis. The intimacy of learning in small-group setting was determined to be part of the pleasure that intrinsically exists in an integrated educational process. During the meetings in the class, it was determined that the anxiety observed at the beginning of the course had decreased rapidly and the desire and motivation to learn had pulled ahead. The reality that “Ants” and “Substances” presentations were made by the students with the highest level of anxiety (Cigdem and Hasan Ali) was an important sign of the success of the PBL. Smiling faces with bright eyes conveyed enthusiasm and motivation of the students. At the end of the course, when the performance-based assessment was conducted, most of the students tried to behave as teachers. Self-confidence was another important characteristic observed during the assessment process. Most of the students expressed their feelings with such sentences as, “I can perform better” and “I feel proud of myself”. Table 2. Explanations about the presentation “Seasons” Characteristics Explanation We worked on vocabulary teaching. We try to teach the new words by direct association. That Purpose means we try to make a relation between words and pictures. This method is being used since the direct method. We tried to adapt it to our presentation. The pictures are very colourful. We wanted to take the students’ attention to the presentation. In our work, colours are very important. Each season is depicted in different colours. The Colours seasons and their colours are relevant. For example, white for winter, green for spring etc. In each season, there are three parts. In the first part, we examine the nature; in the second, people; and in the third, the fruits and vegetable in each season. The verbs we used are in the present tense generally. The pictures are suitable for the verbs. Pictures We try to animate the verbs with the pictures. The words in the sentences (for example; dark, thick clothes) can be related with the pictures: ski on the mountains, wear dark and thick clothes, have a picnic, pick flowers, plants start growing, go swimming. Graphics are used to make the students recognise the comparisons. Graphics The language is relatively simple; it is for lower grades. We want the students to understand Language the sentences. Games could be designed in this unit to make use of interaction while learning. Additional Phase II Rationale and Hypothesis The findings from the Phase I have encouraged us to apply the PBL program for the students taking the same course during the academic year 2002-2003. This time our aim was to investigate the differential effects of the PBL and teacher-centred instruction on cognitive and a group of affective variables which have been coined technophobia. We had to teach at least one class in conventional teacher-centred way. Our hypothesis was that the class taught by utilizing the PBL would be more successful and experience a decrease in their anxiety level more rapidly than with the class receiving teacher-centred instruction. 101 Method Design A quasi-experimental study was designed to evaluate the cognitive and affective outcomes of the PBL program. Sample consisted of university students from the Division of Classroom Teaching (experimental group) and from the Division of English Language Teaching (control group). Pre-test and post-test results for the experimental and the control groups on cognitive and affective variables were compared. The differences in the scores of the experimental and the control groups were also reported. Participants The experimental and control groups taking the same course consisted of 49 and 30 sophomore students respectively. The SPSS-X for Windows programme was used to explore the demographics and the frequency of responses. The experimental group consisted of 15 (31%) male and 34 (69%) female students. The mean age was 20.5 with a standard deviation of 1.3. About 25% of the students possessed a PC. Twenty six percent of the students reported they had taken a computer course before attending the university, and 74% had not. According to the demographics, 68% of mothers had elementary school degree, while 32% have middle/high school or university degree. Also 68% of mothers were housewives and 32% working somewhere. Forty percent of fathers had elementary school degree, 60% graduated from middle/high school or university. Forty six percent were workers and 54% were civil servants. The control group consisted of 8 (27%) male and 22 (73%) female students. Mean age was 20 with a standard deviation of 0.4. In this group, about 40% of the students possessed a PC. Thirty three percent of the students reported that they had taken a computer course before attending the university and 67% had not. From the control group demographics it was determined that 50% of mothers had elementary school degree and 50% middle/high school or university degree. While 54% of them were housewives, 46% have worked somewhere. Of fathers, 27% had elementary school degree and 73% graduated from middle/high school or university. The amount of workers among fathers was 43% while the amount of civil servants was 57%. The experimental group demonstrated low competency level on using computers. Procedure The same instructor taught in both classes. A PBL program was to be used in the experimental group. The control group was given the usual teacher-centred instruction. The PBL program was employed by assigning a group project to the class based on creativity designing a Power Point presentation for their future students as in Phase I, to be completed in a month. As it was mentioned in previous sections the critical factor in the success of PBL is the problem itself. It was presented as a Bloom’s Level 3 problem and it was similar to those students would face in their future profession. This made the students even more enthusiastic than instructor and they immediately realise that they had to make analysis, synthesis and evaluation. Instruments and Measurements After the instruction the students were given the same tests and statistical analyses were run to compare the corresponding performances of the two groups on all the tests. The pre and post tests were also compared for the two groups separately. Both classes were administered the achievement test, CARS, CTS and the GATCS prior to the instruction. Of the two classes, the one with the lower mean score on the achievement pre-test (39.8) was chosen to be the experimental group. The achievement test consisted of 16 items based on the objectives of the course. For the reliability of the scale Cronbach alpha was calculated to be 0.7775. Technophobia was measured by three scales, namely CARS, CTS and GATCS (Rosen and Weil, 1992). The scales were translated into Turkish. CARS consists of 20 items. Higher scores represent more situations for which the respondent feels anxious. The results of the item analysis for the Turkish form were deemed to be satisfactory. For the internal consistency of the Turkish form, Cronbach alpha was calculated to be 0.8975. The second scale was based on the translations of the items of the CTS. Higher scores represent more positive cognition concerning computers. Cronbach alpha was 0.7487 for the 102 Turkish form. For attitudes towards computers, the GATCS, the Cronbach alpha correlation coefficient was determined to be lower (0.6551) than the other two scales. But, this result paralleled other reliability studies conducted by Rosen and Weil on GATCS (1992). In order to gather adequate information concerning the change of students’ attitudes and their level of knowledge, other data sources were used. Among these were, the observations made throughout the course, the group interviews during the performance-based assessment of the presentations, and the final test at the end of the semester examining all the learning objectives covered over the whole semester. Results Considering the sample sizes, non-parametric tests were used for comparing the experimental and control groups as well as the pre-test and the post-test results. The two groups were compared in pre and post measures of achievement (variable ACH) and affective variables (variables CARS, CTS and GATCS) by Independent Samples T-Test. Results of the comparisons of the two groups are presented in Table 3. Paired Samples T-Test was run for the experimental and control groups separately as a non-parametric counterpart of the t-test for dependent groups. Result of the comparisons of the pre and the post test measures are presented in Table 4. When the achievement pre-test results were compared, they indicated a difference at .001 level between the two groups. The control group mean score was significantly higher than the experimental group mean score prior to the instruction. When the test was run for the achievement post test, no significant difference was determined between the scores of the two groups. The group given the PBL instruction was disadvantaged as to the cognitive entry behaviours in the beginning of instruction. At the end of the instruction, the difference disappeared. When the pre and the post test measures of achievement were compared for the two groups separately, the difference in the experimental group scores was about 0.38 points and significant at 0.0001 level, whereas, the difference in control group was .08 points, significant at 0.05 level. We concluded from these findings that PBL instruction applied in our experiment can be claimed to be just as good as the teacher centred instruction for achievement, if not better. Then tests were run for the affective variables. The results of the test for anxiety CARS pre-test given in Table 3 showed no significant differences in the anxiety levels of the two groups although the mean of the control group was a little higher. Similarly, for the post test of anxiety no significant differences existed between the scores of the two groups. When the pre and post measures of computer anxiety were compared, no significant differences were observed, although for both groups, the level of anxiety was lower after instruction. There were no significant differences in the computer cognition measured by CTS in the two groups. But, when pre and post measures were analyzed separately, the cognition of the group which was given the teacher-centred instruction was significantly more positive after the instruction. The change in cognition was not significant after PBL instruction. The results of the pre-test on GATCS showed no significant difference between the attitudes of the two groups as shown in Table 3. The results of the post-test however showed a near significant difference in the attitudes of the experimental and the control groups where the attitude of the control group was more positive. The pre and post test differences in the attitudes of the control group was significant as observed in Table 4. The attitude of the control group was more positive at the end of the instruction, while there was no change in the attitudes of the PBL group. Exp./ Contr. Diff. ACH CARS CTS GATCS Mean Exp./Contr. 2.1645/2.4792 2.0959/1.9750 2.9684/2.9067 3.3418/3.4417 Table 3. Independent Samples t-test Results PRETEST St.Dev. t Sign. Mean Exp./Contr. Exp./Contr. 0.6382/0.4803 -1.821 .04 2.5421/2.5563 0.6415/0.5663 .849 .398 2.0286/1.9667 0.4323/0.3352 .668 .506 2.9071/2.9367 0.5381/0.2736 -.943 .349 3.3051/3.4650 POSTTEST St.Dev. t Exp./Contr. 0.3883/0.3059 -.170 0.5762/0.6469 .442 0.3798/0.3598 -.342 0.6679/0.2761 -1.245 Sign. .886 .660 .733 .217 103 Pre/ Post Diff. ACH CARS CTS GATCS Table 4. Paired Samples t-test Results EXPERIMENTAL Mean St.Dev. t Sign. Mean Pre/Post Pre/Post Pre/Post 2.1645/2.5421 0.7629/0.3883 -3.783 .0001 2.4792/2.5563 2.0959/2.0286 0.6415/0.5762 -.755 .454 1.9750/1.9667 2.9684/2.9071 0.4323/0.3798 1.051 .299 2.9067/2.9367 3.3418/3.3051 0.5381/0.6679 .481 .633 3.4417/3.4650 CONTROL St.Dev. t Pre/Post 0.7157/0.3059 -.746 0.5663/0.6469 .114 0.3352/0.3598 -.555 0.2736/0.2761 -.502 Sign. .461 .910 .583 .619 General Discussion and Conclusion The motivational role of ICT in learning is widely recognised. How well prospective teachers learn to use ICT will profoundly influence their effectiveness as future educators. This paper demonstrates how we employed the PBL method in the course titled “Introduction to ICT”, providing the students at the Faculty of Education with a lively and effective learning environment. As researchers, we are interested in determining whether or not such an environment helps the students improve their academic performance. Another point of interest, as significant as the preceding one, is to note the difference in the level of anxiety about present or future interaction with the computers, if indeed there is any, in comparison with traditional teaching and learning environment. In both phases of the study, it was rewarding to see the students so interested in and deriving pleasure from their learning experience with the PBL. It seemed obvious that they were more motivated, working much harder with this model compared with the students to which the traditional methods were applied. As Milliken and Barnes (2002) states, there is a recognition that traditional, university teaching methods have served the interests of lecturers and educational institutions more than they have served the interests of students. It is possible to state that with the application of the PBL, the interests and learning needs of students are more fully and sufficiently acknowledged and satisfied. The attitude towards the computers were consistently determined to be an important variable in educational computing research playing a pivotal role in student success in computer-related tasks. Hong, Ridzuan, and Kuek (2003) stated that the students with positive attitudes towards the computers are also favorably disposed towards them as an instrument of learning. On the other hand research in medical education indicate that the students develop improved attitudes towards learning when received PBL training in advance (Lim, 2002; Liu, 2004). In our study we observed that, in addition to pedagogical benefits obtained as a byproduct, the learning process was driven by the problem initially posed: the learning process became interactive involving the students and the teachers in unison. By working in groups, the students developed collaborative work skills (Grabinger et al., 1997) and learned how to work within complex environments. Although groups spent a great deal of time on PBL work usually breaking the time limit, they took longer than expected to complete the presentations. Two factors seemed to account for this situation from which the second one might be considered as a limitation for this study. Firstly, they found that conducting Internet searches for specific information was quite different and generally needed more time. Secondly, they had limited computer access out of class hours and this made the process of working together slower. As (Tynjälä, 1999) and Willis (2002) state, student learning is strongly steered by its assessment tool. Traditional methods of assessment have often used only “objective” measures such as examinations and tests to assess learning. Student’ perception of assessment requirements direct their approaches to learning and affect their learning outcomes At the beginning of the semester students from the experimental group had negative feelings about performance based assessment which is another key point in PBL, they found it encouraging by the end of the course. The level of knowledge the students reached were deemed to be satisfactory. During the time they spent together, they also tried to teach each other the more complex features of the Power Point, most of them expressing their hope of better performance in the near future. The evidence gathered from the whole process indicate that the students were inclined to self-direction. Another promising development was observed in the area of strong individual responsibility, further motivating some students to prepare presentations for other courses, e.g. “The History of Art” and “Instruction Technologies”. Thus, the generic course could be said to have achieved its aims. 104 Although our expectation was an evidence of a significant difference between the changes in attitudes of the two groups in Phase II, we were encouraged to see, the students overcame, at least to some degree, their initial feelings of technophobia. In our opinion, this particular result concerning Phase II could be explained by taking into consideration the human factor: the instructor as the source of information. The instructor is still the dominant side in the relationship between the elements that partake in the learning process where technology is both a tool and the object. A different explanation can be derived from the fact stated by Liu (2004) that the original discomfort with an initial degree of freedom as exhibited in most of the students during PBL, is an additional source of anxiety. As it was stated in Grabinger et al. (1997), creating autonomous, life-long learners, the PBL specifically teach students to manage their own learning: identify learning needs, set learning objectives, select and employ learning strategies, determine and use appropriate resources and assess the overall process. Such an approach requires that we, the instructors, change also, from being the center of attention and the source of all knowledge to the coaches and facilitators of the acquisition of knowledge sought for by the students. Although the PBL is perceived as costly, for it requires a great investment of faculty time to provide tutors, we believe that teaching through this method produces independent learners who can continue to learn on their own in their chosen careers. Acknowledgements We wish to acknowledge the assistance of Çiğdem Selçukcan, Fidan Bayraktutan and Ceyda Cimilli in conducting interviews with the students and making classroom observations for this study. Appreciation is extended to Hulusi Gülseçen for his support during all stages of the study. We also thank the reviewers of “Educational Technology and Society” who gave constructive and helpful comments. References Albion, P. R. (2003). PBL + IMM = PBL2: Problem-based learning and interactive multimedia development. Journal of Technology and Teacher Education, 11 (2), 243-257. Bartsch, R. A., & Cobern, K. M. (2003). Effectiveness of Power Point presentations in lectures. Computers & Education, 41, 77-86. Bloom, B. (1995). Taxonomy of Educational Objectives, New York, USA: McKay. Dahlgren, M. A. (2003). PBL through the looking-glass: comparing applications in computer engineering, psychology and physiotherapy. The International Journal of Engineering Education, 19 (5), 672-681. De Graaf, E., & Kolmos, A. (2003). Characteristics of Problem-Based Learning. The International Journal of Engineering Education, 19 (5), 657-662. Driessen, E., & Vlueten, C. V. D. (2000). Matching student assessment to problem-based learning: lessons from experience in a law faculty. Studies in Continuing Education, 22, 235-248. Duch, B. (1997). Problems: A key www.udel.edu/pbl/cte/spr96-phys.html. factor in PBL, retrieved November 25, 2005 from Erktin, E., & Gülseçen, S. (2001). Psychological factors affecting the use of IT in education. Education and Science, 26 (121), 7-11. Frank, M., Reich, N., & Humphreys, K. (2003). Respecting the human needs of students in the development of e-learning. Computers & Education, 40, 57-70. Fullan, M. (1991). The new meaning of educational change, London, England: Cassell Educational Ltd. Gibson, K. E., & Silverberg, M. (2000). A two-year experience teaching computer literacy to first-year medical students using skill-based cohorts. Bulletin of Medical Library Association, 88 (2), 157-164. 105 Grabinger, S., Dunlop, J., & Duffield, J. A. (1997). Rich environments for active learning in action: problem based learning. ALT-J, 5 (2), 3-17. Harland, T. (2002). Zoology Students’ Experiences of Collaborative Enquiry in Problem-based Learning. Teaching in Higher Education, 7 (1), 6-17. Hong, K.-S., Ridzun, A. A., & Kuek, M.-K. (2003). Students’ attitudes toward the use of the Internet for learning: A study at a university in Malaysia. Educational Technology & Society, 6 (2), 45-49. International Telecommunication Union (2003). Access Indicators for the Information Society, World Telecommunication Development Report, Geneva (ISBN 92-61-10541-6). Jonassen, D. (2000). Computers as Mindtools for Schools: Engaging Critical Thinking, Prentice Hall. Jonassen, D. H., Howland, J., Moore, J. L., & Marra, R. M. (2003). Learning to solve problems with technology: a constructivist perspective, New Jersey, USA: Pearson Education Inc. Lim, K. (2002). Impacts of personal characteristics on computer attitude and academic users information system satisfaction. Journal of Educational Computing Research, 26 (4), 395-406. Liu, M. (2004). Examining the performance and attitudes of sixth graders during their use of a problem-based hypermedia learning environment. Computers in Human Behaviour, 20, 357-379. Maxwell, N. L., Bellisimo, Y., & Mergendoller, J. (2001). Problem Based Learning: modifying the medical school model for teaching high school economics. The Social Studies, March/April, 73-78. Milliken, J. & Barnes, L. P. (2002). Teaching and technology in higher education: student perceptions and personal reflections. Computers & Education, 39, 223-235. Orhun, E. (2002a). Computer-Based Cognitive Tools. In E. Orhun, & P. A. M. Kommers (Eds.), Information and communication technologies in education: a focus on cognitive tools, İzmir, Türkiye: Ege University Press, 17-26. Orhun, E. (2002b). Computer Innovation in Education in Turkey: A Study of the Initiation Conditions in Secondary Schools in İzmir. In E. Orhun, & P.A.M. Kommers (Eds.), Information and communication technologies in education: a focus on cognitive tools, İzmir, Türkiye: Ege University Press, 343-375. Rosen, L. D., & Weil, M. M. (1992). Measuring Technophobia, A Manual for the Administration and Scoring of the Computer Anxiety Rating Scale, Computer Thoughts Survey and the General Attitudes towards Computers Scale, California State University Dominguez Hills, Computerphobia Reduction Program. Saunders, W. L. (1992). The Constructivist Perspective: Implications and Teaching Strategies for Science. School Science and Mathematics, 92 (3), 136-141. Savery, J. R., & Duffy, T. M. (1995). Problem Based Learning: An instructional model and its constructivist framework. Educational Technology, 35, 31-38. Tynjälä, P. (1999). Toward Expert Knowledge? A Comparison Between a Constructivist and a Traditional Learning Environment in the University. International Journal of Educational Research, 31, 357-442. Tse, W. L., & Chan, W. L. (2003). Application of problem based learning in an engineering course. The International Journal of Engineering Education, 19 (5), 747-753. Weil, M. M., & Rosen, L. D. (1995). The psychological impact of technology from a global perspective: a study of technological sophistication and technophobia in university students from twenty three countries. Computers in Human Behaviour, 11 (1), 95-133. Willis, S. C., Jones, A., Bundy, C., Burdett, C. R., Whitehouse, C. R., & O’Neill, P. A. (2002). Small-group work and assessment in a PBL curriculum: a qualitative and quantitative evaluation of student perception of the process of working in small groups and its assessment. Medical Teacher, 24 (5), 495-501. Yıldırım, I. S. & Kiraz, E. (1996). Distance does not really matter: a case study. Proceedings of the Türkiye First International Distance Education Symposium, Ankara: UZEV, 673-682. 106 Zheng, R. & Zhou, B. (2006). Recency Effect on Problem Solving in Interactive Multimedia Learning. Educational Technology & Society, 9 (2), 107-118. Recency Effect on Problem Solving in Interactive Multimedia Learning Robert Zheng Assistant Professor, Department of Psychological Studies in Education (PSE) College of Education, Temple University, Pennsylvania, USA robert.Zheng@temple.edu Bei Zhou Department of Curriculum, Instruction, and Technology in Education (CITE), College of Education Temple University, Ritter Hall 472, 1301 Cecil B Moore Ave, Philadelphia, PA 19122, USA Tel: +1 215 204 6372 zb713@temple.edu ABSTRACT This study investigated the impact of recency effect on multiple rule-based problem solving in an interactive multimedia environment. Forty-five college students were recruited and assigned to two groups: synchronized and unsynchronized interactive multimedia groups based on their spatial ability score. Results show that students in the synchronized interactive multimedia group outperformed their counterparts in the unsynchronized interactive multimedia group in terms of response time and test scores. Results also indicated that low spatial ability learners in the synchronized interactive multimedia showed an improvement in problem solving. Keywords Multimedia, Problem solving, Recency effect, Rule-based reasoning, Working memory Introduction Problem-solving skill is highly valued. In the last five decades, many theorists and educational institutions have placed a heavy emphasis on this ability. For example, the movement of “discovery learning” (e.g., Bruner, 1961) was spawned, at least in part, by the perceived importance of fostering problem-solving skills. This emphasis on problem solving was not associated, however, with the knowledge of cognitive resources involved in the problem solving process. That is, it focused on procedures of problem solving rather than investigating the relationship between procedures of problem solving and cognitive resources that affect such procedures (Sweller & Low, 1992; Hanley, 1987). In the last twenty years, this state of affairs has begun to change with our knowledge of relevant mechanism (e.g., working memory, cognitive load, etc.) increasing markedly. This study investigated the recency effect – a phenomenon in working memory that affects learners’ holding of information during problem solving - and how such phenomenon may affect learners’ multiple rule-based reasoning in particular, and problem solving skills in general. Literature Review Working Memory and Cognitive Load Cognitive scientists believe that the input information such as auditory, visual, and kinesthetic, etc. is processed through a temporary storage before it gets coded into the long term memory (Baddeley & Logie, 1992; Logie, 1995). This temporary storage, also called the working memory, comprises two major components. One is the executive controlling mechanism that is believed to be related to cognitive activities such as reasoning and problem-solving. The other is the visuo-spatial sketchpad (VSS) that is believed to process and manipulate visuo-spatial images (Logie, 1995; Pearson & Logie, 2000). For example, the ability to mentally manipulate 3D images by rotating them in mind is largely determined by the VSS function. The two components are closely related and interacted with each other in the process of problem solving (Bollaert, 2000; Mayer & Moreno, 2003). Studies show that the working memory is very limited in both duration and capacity. Van Merrienboer and Sweller (2005) observe that the working memory stores about seven elements but normally operates on only two or three elements. They also find that the working memory can deal with information “for no more than a few seconds with almost all information lost after about 20 seconds unless it is refreshed by rehearsal” (p. 148). When the working memory becomes overloaded with information, learning can be adversely affected (Paas, Tuovinen, Tabbers, & Gerven, 2003; Sweller & Chandler, 1994; Marcus, Cooper, & Sweller, 1996). Sweller and ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 107 Chandler (1991) studied the relationship between the cognitive load and the learning effectiveness and found that learning became improved when the extraneous cognitive load was reduced. According to Cognitive Load Theory (CLT), three types of cognitive load exist: intrinsic load, extraneous or ineffective load, and germane or effective load. The intrinsic cognitive load refers to cognitive load that is induced by the structure and complexity of the instructional material. Usually, teachers or instructional designers can do little to influence the intrinsic cognitive load. The extraneous cognitive load is referred to the cognitive load caused by the format and manner in which information is presented. For example, teachers may unwittingly increase learner’s extraneous cognitive load by presenting materials that “require students to mentally integrate mutually referring, disparate sources of information” (Sweller et al., 1991, p.353). Finally, the germane cognitive load refers to cognitive load that is induced by learners’ efforts to process and comprehend the material. The goal of CLT is to increase this type of cognitive load so that the learner can have more cognitive resources available to solve problems (Brunken, Plass, & Leutner, 2003; Marcus, et al., 1996). Cognitive Load Theory has been recently introduced into multimedia studies where researchers try to find out whether the use of multimedia can improve learning by reducing extraneous cognitive load. Tabbers, Martens and Van Merrienboer (2004) studied the effects of modality and cueing on learning in terms of cognitive load theory and found that adding visual cues to the pictures enhanced learners’ retention scores. Mayer and his colleagues (Mayer, 1997; Mayer et al., 2003) conducted a series of studies on the effect of text and image on problem solving. They noticed that when learners were presented with synchronized texts and images, they were able to better recall and answer transfer questions. However when learners were presented with unsynchronized instructional material, i.e., the text was presented first followed by an image (a filled delay phenomenon), learners’ ability to recall and answer transfer questions decreased. Mayer (1997) concluded that the synchronized multimedia could alleviate the extraneous cognitive load and provide extra cognitive resources for problem solving whereas the unsynchronized multimedia could cause cognitive overload, thus decreased performance in learning. Although the CLT theory provides insights on problem solving in terms of cognitive load, cautions must be taken in interpreting learners’ ability to solve problems. Studies show that factors such as recency effect, problem types, etc. are related to learners’ problem-solving skills (Delisle, 1997; Logie, 1995). Recency Effect and Cognitive Load While the process of problem solving draws on resources from both long-term and short-term memories, it is believed to rely heavily on the working memory for the working information during the problem-solving process (Baddeley et al., 1992; Logie, 1995). When this temporary holding of information in the working memory is interrupted, the ability to solve problems can be affected. Capitani, Della Sala, Logie, and Spinner (as cited in Logie, 1995) conducted a study on the working memory. They found a high recall by subjects immediately after the items have been presented. However if there was a filled delay before the recall was required, only the first few items on the list could be recalled. Logie (1995) described the former phenomenon as the recency effect and the latter as the primary effect. He believed that “recency reflected the operation of a short-term or primary memory system” (p. 5) which was critical to problem solving process. It is essential to distinguish between cognitive load and recency effect since both involve cognitive resources in working memory pertaining to learning. The notion of cognitive load refers to the load imposed on the learner’s working memory while performing a particular task whereas the recency effect refers to the information, specifically the most recent information that can be recalled within working memory. The cognitive load study is focused on working memory architecture and its limitations relating to the design of instruction (Van Merrienboer et al., 2005; Paas et al., 2003; Tabbers et al., 2004). The recency effect is, however, focused on the state of recalling and maintaining much needed information during problem solving process. Of particular interest to researchers is the issue of effective time period in recency effect. Studies on effective time period in recency effect have so far produced mixed results. For example, studies by Posner, Boies, Eichelman and Taylor (1969, cited from Baddeley, 1997) suggest that the information resulting from a visual trace has a 2-second decay rate. However, Glanzer and Cunitz (1966, cited from Haberlandt, 1999) observed that the recency effect was wiped out when recall was delayed by 30 seconds. Contrary to the findings of Posner et al. (1969) and Glanzer et al. (1966), Bjork and Whitten (1974, cited from Haberlandt, 1999) showed that the recency effect could survive a delay. They had subjects learn a word list in a free-recall experiment. After a 30second interval of backward counting that should have resulted in a loss of the most recent items from the shortterm memory, they found the subjects were still able to recall the most recent items in the list. Obviously, the inconsistency in the above findings raises the question of whether the effective time period should be used to measure the impact of recency effect on problem solving. Richardson (1996) states that “working memory is not a general system with unitary capacity. Rather, the capacity of working memory will vary as a function of how efficient the individual is at the specific processes demanded by the task to which working memory is being 108 applied” (p. 12) which means the measurement of the impact of recency effect on problem solving should focus on the effectiveness that learners make use of the limited working memory capacity in information processing (Perfetti & Goldman, 1976). In other words, the recency effect should be measured in terms of learner efficiency in problem solving, that is, the ability to achieve optimal results in problem solving with minimum amount of time possible. Problem Types Delisle (1997) states that problem types are related to thinking procedures in problem-solving process. For example, causal relationship problems require a linear thinking procedure that has a strong linear direction emphasizing the cause and effect whereas multiple rule-based problems involve simultaneously weighing several conditions/rules in mind in order to make a decision (Frye, Zelazo, & Palfai, 1995; Price & Yates, 1995). Zheng, Miller, Snelbecker, & Cohen (2005) assert that different types of problem may require different levels of working information in problem solving process. For instance, the multiple rule-based problem solving may require more working information than does causal relationship problem solving or single rule-based problem solving. The single rule-based problem solving, according to Frye et al. (1995), requires a straight-forward deductive thinking such as applying the rule of card sorting to the action of sorting a deck of cards whereas the multiple rule-based problem solving involves a more complex, nonlinear thinking where the learner reaches a solution by engaging in a series of cognitive thinking activities such as analyzing, synthesizing, evaluating, and so forth while holding several conditions and rules in mind within a short time framework provided by the working memory (Johnson, Boyd, & Magnani, 1994; Price et al., 1995). Obviously, multiple rule-based problem solving is likely to increase intrinsic cognitive load more than other two types of problem solving. Studies showed that the interactive multimedia can enhance students’ recalling and maintaining of working information in multiple rule-based problem solving (Zheng et al., 2005). According to the dual coding theory (Paivio, 1986), learning is more effective with two sensory channels (i.e., visual and verbal) than one channel alone. When information is processed through multiple sensory channels, learners’ ability to hold such information is improved (Zheng et al., 2005). However, this ability is also determined by the types of problem and thinking procedures during the problem solving process. Learner Characteristics One of the issues that frequently surfaced in multimedia problem-solving research is whether demographic factors such as age, education, ethnicity, etc. would affect learners’ problem solving (Forcier & Descy, 2005). Hall and Cooper (1991) report a gender difference in computer use. Passig and Levin (2000) concur that gender preferences exist in multimedia related problem solving. Another issue of interest is individual differences in multimedia problem solving. Fink and Neubaurer (2005) observed that individuals differ in speed of information processing and working memory. Mayer and Sims (1994) conducted a study on spatial ability and problem solving with multimedia and concluded that the spatial ability was critical to problem solving and “appears to enhance coordinated visual and verbal instruction” (p. 399). Hypotheses Based on the above discussion, the following hypotheses were proposed that formed the basis for this study: Hypothesis 1 Participants in the synchronized interactive multimedia group will outperform their counterparts in the unsynchronized interactive multimedia group in multiple rule-based problem-solving. Hypothesis 2 Different interactive multimedia such as synchronized and unsynchronized multimedia can significantly affect learners’ spatial ability and their performance relating to such ability. 109 Hypothesis 3 Learners’ ability to solve problems can be affected by demographic factors such as education, ethnicity, gender, and hobbies. Methodology Participants & Design Participants consisted of 45 students recruited from a large comprehensive research university on the east coast of the United States in the fall 2004. Of 45 participants, 89% (n = 40) were undergraduate teacher education majors who received course credit for participating in the study and 11% (n = 5) were graduate students who volunteered to participate in this study. Approximately 65 % (n = 29) of the participants were Caucasian, 22% (n = 10) were African-American, and 13% (n = 6) were Asian or Asian American. The average age of the participants was 21 with a range from 19 to 31. Two levels of interactive multimedia (synchronized vs. unsynchronized) were crossed with two levels of spatial ability (high vs. low) to form a 2 x 2 between subjects factorial design. Participants were blocked by spatial ability and then randomly assigned to one of the multimedia groups. The independent variables included multimedia learning (synchronized vs. unsynchronized) and spatial ability (high vs. low) with two dependent variables: response time and test scores. In this study, demographic information such as age, education, ethnicity, and hobbies was collected and served as covariates in MANCOVA analysis. All statistical tests were performed with alpha at .05. Measures Two instruments were used to assess students’ spatial ability and problem-solving skills. Guilford-Zimmerman Aptitude Survey Test 5: Spatial Orientation (Guilford-Zimmerman, 1956). The GuilfordZimmerman spatial orientation test requires participants to observe the position of the bow of a boat relative to the horizon and then to correctly identify the change in the boat’s orientation by the change in horizon relative to the bow. The purpose of the Guilford-Zimmerman test is to assess participant’s ability to mentally maneuver objects in terms of the special relationship which is consistent with the reasoning tasks set up in this study, that is, the participant came up with a solution by maneuvering the figures and determining the spatial relationship between the figures based on the conditions and rules set in the tasks. The Guildford-Zimmerman spatial orientation test has reported a high reliability (alpha = .88) (Price & Eliot, 1975). Interactive Multimedia Problem-Solving Test. The interactive multimedia problem-solving test was developed by the first author using Flash MX and Microsoft Active Server Page. The test comprised five sub-tests that included air traffic control, Tower of Hanoi, sailing boat, taking pictures, and office inspection. Sub-test 1 Air Traffic Control had a set of conditions that restricted the order and parking positions of airplanes. The subject had to consider all the conditions and decided which flight would park at which gate without violating the conditions (Figure 1). Sub-test 2 Tower of Hanoi had a set of rules restricting the movement of disks. The subject was required to move three disks from one stack to another without violating the rules. Sub-test 3 Sailing Boat described six boats anchoring at four different directions. The subject was to find out the spatial relationship between the boats based on the conditions given. Sub-test 4 Taking Pictures had a set of rules that determined who could stand next to each other in the line. The subject was required to come up with a list of persons who would take the pictures in an order that did not violate the rules set by the task. Finally, Sub-test 5 Office Inspection was about office inspections by five people who conflicted with each other in terms of the schedule and the order of inspection. The subject had to consider these conflicting conditions and decided who got assigned to inspect which office without violating the conditions/rules. For each sub-test, there was a problem (text) and an interactive multimedia with which the subject could move the figures around to help solve the problem. Two versions of tests were created: synchronized and unsynchronized. The synchronized interactive multimedia test displayed both text and interactive multimedia at the same time on the same page whereas the unsynchronized interactive multimedia test separated the text from the interactive multimedia by presenting the text first followed by an interactive multimedia. 110 The problem-solving test consisted of five sub-tests, each contained three parts: the problem with the text format, interactive multimedia, and two problem-solving questions that measured participants’ multiple rule-based reasoning skills. Each sub-test had a timer recording the start and the end of the response time (Figure 1). Figure 1. Sample of Multiple Rule-Based Problem-Solving Tasks To ensure the content validity, the instrument was reviewed by a panel of instructors and a selected group of graduate students whose feedback was carefully reviewed and changes were made based on the feedback. Next, a pilot test was conducted to a group of undergraduates (n = 10). Further changes were made based on the results of the pilot test and comments from instructors. A reliability analysis was done on test items. The Pearson correlation analysis showed that nine items had a significant correlation between sub-items (Table 1). The item reliability analysis using Cronbach alpha showed a moderate coefficient of .71. Table 1. Pearson Correlation Matrix 111 Procedure Using a blocked random sampling procedure, participants were first given the Guilford-Zimmerman Aptitude Survey Test 5: Spatial Orientation (Guilford & Zimmerman, 1956). The middle value of the distribution (median = 7.25) was determined and chosen as the cut score for defining subjects’ high and low spatial ability. Subjects were then blocked into high and low spatial ability groups based on their spatial ability test score. Subjects were randomly drawn from each of the blocked groups (i.e., high and low spatial ability) to form two separate groups: the synchronized interactive multimedia group and the unsynchronized interactive multimedia group. A paired samples t-test was performed on the synchronized and unsynchronized multimedia groups. The result indicated no significant difference between two groups, t (1, 21) = 1.00 (2-tailed), p = .329, ns. Participants were given a test number and assigned to one of the interactive multimedia groups based on the result of randomization. They were provided a URL to take the problem solving test online in a computer classroom proctored by the investigator. The participants were required to fill out a demographic information sheet online which included age, education, ethnicity, hobbies, etc. After completing the demographic information, participants began to take the problem solving test. Participants were first presented a problem that included a scenario with some conditions and rules. They were then told to use the interactive multimedia to help solve the problem by moving images or figures around to simulate various solutions until the correct solution was reached. The total score equaled 10 points. If an error was made by the participant, a point would be deducted from the total score. All participants were given a consent form to sign for their participation in the study. Results Descriptive Statistics The means and standard deviations are reported in Table 2. Table 2. Descriptive Statistics for Response Time and Test Scores With regard to the response time the synchronized group (Mean sync = 16.87) spent less time than the unsynchronized group (Mean unsync = 18.89) in problem solving test. Further, high spatial ability subjects in the synchronized group (Mean sync = 15.89) spent less response time than their counterparts in the unsynchronized group (Mean unsync = 18.64). However, low spatial ability subjects in the synchronized group (Mean sync = 17.95) 112 seemed to spend more response time than their counterparts in the unsynchronized group (Mean unsync = 17.25). With regard to test scores the synchronized group (Mean sync = 5.43) scored higher on problem-solving test than the unsynchronized group (Mean unsync = 4.41). Both high and low spatial ability subjects in the synchronized group outperformed their counterparts in problem-solving test. MANCOVA Tests A multivariate analysis of covariance (MANOVA) was conducted with spatial ability and interactive multimedia group as independent variables, the response time and the test score as dependent variables, and education, gender, ethnicity, and hobby as covariates. The Wilks’ Lambda estimate was used to determine the main effects. The results indicated that there was a main effect for multimedia group (Wilks’ Lambda = 5.642; p = .007) and for spatial ability (Wilks’ Lambda = 5.397; p = .008). There was also an overall interaction between the multimedia group and the spatial ability (Wilks’ Lambda = 6.480; p = .004). The covariance analysis indicated that none of the covariates were significant: education (Wilk’s Lambda = 1.824, p = .176), gender (Wilk’s Lambda = 2.167, p = .129), Ethnicity (Wilk’s Lambda = .736, p = .486), and hobby (Wilk’s Lambda = .944, p = .398) (Table 3). Table 3. MANOVA Tests The between-subjects analysis showed that there was a significant difference between the synchronized and the unsynchronized groups in terms of the response time (F (1, 44) = 4.345; p = .043) and test scores (F (1, 44) = 6.187; p = .017). There was a significant difference between the high and the low spatial ability people for test scores (F (1, 44) = 10.091; p = .003) but no significance was found for the response time (F (1, 44) = .482; p = .491). With regard to the multimedia group and spatial ability interaction, there was an overall interaction for the response time (F (1, 44) = 11.979; p = .001). But no interaction was found for test scores (F (1, 44) = .679; p = .415) (Table 4). Efficiency Scores The efficiency score is the total test score divided by the total response time in seconds. The total test score was calculated by adding up the correct scores from each task and the total response time was calculated by summing up the response time of all tasks. An efficient problem solver is defined as a person who achieved higher test 113 scores with less response time. Descriptive data (Table 5) and the results of univariate analysis of variance (Table 6) are reported as follows. Table 4. Tests of Between-Subjects Effects Table 5. Descriptive Statistics for Efficiency Scores Table 6. Univariate Analysis of Variance 114 Discussions The discussion of the results will focus on three hypotheses proposed earlier with an emphasis on (a) the impact of recency effect on problem-solving, (b) multimedia and spatial ability, and (c) demographic factors as covariates. Hypothesis 1 Hypothesis 1 tried to investigate whether the use of different interactive multimedia (synchronized vs. unsynchronized) would affect learners’ performance in problem solving. The results of the study showed a main effect for multimedia groups as well as an overall interaction between multimedia groups and the spatial ability, which means learners’ performance in problem solving was affected by the type of media employed. This was supported by the mean difference in test scores between the synchronized group (Mean sync = 5.43) and the unsynchronized group (Mean unsync = 4.41) (Table 1). The between-subjects tests indicated a statistical significance between two groups (F (1, 44) = 6.187; p < .05) (Table 4). However, when learners’ performance was measured by the response time instead of test scores, low spatial ability subjects in the synchronized group performed less well than high spatial ability subjects in the unsynchronized group (Table 2). There was a marginal significance between two groups (synchronized vs. unsynchronized) in terms of response time (F (1, 44) = 4.345; p = .043). Nonetheless, such difference was compromised by a small effect size (η2 = .096) and weak statistical power (.530) (Table 4), which means the difference may have little practical significance in reality. Results also show that when provided with the synchronized interactive multimedia, higher spatial ability learners had a higher efficiency score than their counterparts in the unsynchronized interactive multimedia (Table 5). There was a significant difference between multimedia groups (F (1, 44) = 11.99, p < .05) (Table 6). Since the efficiency score indicates learners’ ability to solve problems in terms of the ratio between total test scores and total response time (in seconds), it would be reasonable to assume that learners who solved more problems with less response time had more cognitive resources during problem solving process. It would also imply that such learners were more effective in recalling and maintaining critical working information while solving problems since recalling and maintaining working information is critical to problem solving (Loggie, 1995). The results of MANCOVA and univariate analysis of variance have consistently shown that the synchronized interactive multimedia facilitated learners’ ability to solve problems due to a prompt recall and retrieval of working information whereas the unsynchronized interactive multimedia decreased the learners’ ability to solve problems due to a filled delay phenomenon caused by scrolling back and forth between pages. Evidently, the synchronized interactive multimedia facilitates the recall of on-demand working information – a recency effect and enables learners to solve problems more efficiently in a short time framework provided by the working memory. The findings of this study concurred with previous studies (e.g., Mayer et al., 1994; 2003) that the synchronized multimedia enhanced students’ problem solving whereas the unsynchronized multimedia could affect students’ ability to recall and maintain working information in the working memory due to a filled delay phenomenon. Hypothesis 2 Hypothesis 2 tried to find out whether different interactive multimedia such as synchronized and unsynchronized multimedia would affect learners’ spatial ability and their performance. The MANCOVA analysis revealed an interaction between multimedia group and spatial ability (Wilks’ Lambda = 6.480; p = .004). Overall, high and low spatial ability subjects in the synchronized group outperformed their counterparts in the unsynchronized group (F (1, 44) = 10.091, p < .05). It should be noted that with synchronized interactive multimedia low spatial ability subjects outperformed their counterparts and did almost as well as the high spatial ability subjects in the unsynchronized group. This is perhaps for low spatial learners who are known for linear and abstract thinking (McGrew & Flanagan, 1998), multiple sensory inputs from the synchronized interactive multimedia can provide extra cognitive resources that enable them to recall and maintain on-demand working information while engaging in multiple rule-based reasoning. This suggests that media attributes like synchronized interactive multimedia can compensate learner’s deficit in spatial ability such as visualization (Reiser, 1994; Salomon, 1979). 115 The efficiency analysis showed that the synchronized interactive multimedia facilitated learning for both high and low spatial ability learners. High spatial ability learners in the synchronized group (Mean = .387) outperformed all three groups including low spatial ability learners in the synchronized group, high and low spatial ability learners in the unsynchronized group. The mean score of low spatial ability learners in the synchronized group (Mean = .260) was higher than its counterparts (Mean = .226) and almost the same as the high spatial ability learners in the unsynchronized group (Mean = .262) (Table 5). There was a significant difference between spatial groups in terms of efficiency (F (1, 44) = 12.60, p < .05). This suggests that with the synchronized interactive multimedia both high and low spatial ability learners were able to perform well in problem solving. Hypothesis 3 Hypothesis 3 tried to determine if demographic factors such as education, ethnicity, gender, and hobbies would affect learners’ ability to solve problems. Although studies (Forcier et al., 2005; Hall et al., 1991; Passig et al., 2000) showed that demographic factors could influence learners’ attitude, motivation, and their way of learning, the results of this study showed that none of the demographic factors that is, education, ethnicity, gender, and hobbies, were significant (Table 3), which suggested that in this particular study main effects for multimedia group and spatial ability were accounted for by the differences between multimedia groups, not affected by any of the covariates mentioned above. Conclusion As it was mentioned elsewhere in this paper, the ability to hold information in one’s working memory is critical to multiple rule-based reasoning problem solving. This study indicated that the recency effect, which is related to learners’ ability to recall and maintain working information in working memory, is related to the media employed in problem solving. The analysis of efficiency scores revealed that the synchronized interactive multimedia could facilitate recalling and maintaining on-demand information in working memory - a recency effect, thus enabled learners to solve problems more efficiently whereas the unsynchronized interactive multimedia hinders immediate information retrieval due to a filled delay phenomenon. This was supported by the observation we made in which subjects in the synchronized group were more focused on problem solving whereas subjects in the unsynchronized group frequently scrolled back and forth between the pages trying to update the information in working memory while working on the problems. This study concurred with previous studies that spatial ability is related to multimedia problem solving (Mayer et al., 1994). However, it went further to conclude that the synchronized interactive multimedia could compensate low spatial ability learners for lacking visualization in problem solving. Contrary to the findings of previous studies (Forcier et al., 2005; Hall et al., 1991; Passig et al., 2000; Fink et al., 2005), this study did not find any significant relationship between demographic factors and learners’ ability to problem-solving. Overall, we can derive some useful theoretical and practical implications from this study. On the theoretical side, the results have extended the exiting studies on multimedia such as cognitive load theory and called attention to an important cognitive phenomenon – recency effect - in the interactive multimedia problem solving. The study provides a better understanding of the relationship between spatial ability and media as well as the relationship between recency effect and multimedia presentation mode, which promotes further research on the constructs of working memory, complex reasoning such as multiple rule-based reasoning, and interactive multimedia. On the practical side, our work reveals that differences in multimedia design can result in different cognitive consequences. Therefore, teachers and other professional educators need to be aware of such issues as media attributes, problem types, and learner characteristics in the process of designing multimedia problem solving, especially in designing multiple rule-based problems. The importance of teaching problem solving to students has been widely recognized by teachers, administrators, and other educational stakeholders. The advent of computer technology, particularly multimedia learning, has changed the landscape of problem-solving instruction. Computer-based problem simulations, for example, begin to replace the traditional paper and pencil approach with more vivid, interactive approach that provides both auditory and visual information to problem solving. The psychological and cognitive benefits of using multimedia to teach problem solving are palpable: the multimedia motivates students to learn, promotes deep understanding, and engages them in problem solving (Fulford, 2001; Mayer et al., 2003; Rieber & Hannafin, 1988). This study has provided initial results on the impact of recency effect on multimedia problem solving. 116 Further research is needed to investigate the relationship between cognitive load and recency effect and perhaps include cognitive load as a possible covariate to determine the impact of recency effect on problem solving. Research in the future should include a larger, more diverse population in terms of ethnicity, age, education, social and economic status. It is suggested that a broader research agenda is needed to address the whole range of issues in the study of recency effect that includes the relationship between recency effect and its related cognitive constructs such as learner aptitudes, learning styles, mental synthesis, etc. in the context of working memory and multimedia learning. References Baddeley, A. D., & Logie, R. H. (1992). Auditory imagery and working memory. In Reisberg, D. (Ed.), Auditory imagery, Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 179-197. Baddeley, A. D. (1997). Human memory: Theory and practice, Hove, UK: Psychology Press. Bollaert, M. (2000). A connectionist model of the processes involved in generating and exploring visual mental images In Nuallian, S. O. (Ed.), Spatial cognition: Foundations and applications, Amsterdam/Philadelphia: Joint Benjamins Publishing, 329-346. Bruner, J. (1961). The act of discovery. Harvard Educational Review, 31 (1), 21-32. Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38 (1), 53-61. Delisle, R. (1997). How to use problem-based learning in the classroom, Association for Supervision and Curriculum Development: Alexandria, VA, USA. Fink, A., & Neubauer, A. C. (2005). Individual differences in time estimation related to cognitive ability, speed of information processing and working memory. Intelligence, 33 (1), 5-26. Forcier, R., & Descy, D. (2005). The computer as an educational tool: Productivity and problem solving (4th Ed.), Upper Saddle River, NJ, USA: Pearson-Merrill/Prentice Hall. Frye, D., Zelazo, P. D., & Palfai, T. (1995). Theory of mind and rule-based reasoning. Cognitive Development, 10 (4), 483-527. Fulford, C. P. (2001). A model of cognitive speed. International Journal of Instructional Media, 28 (1), 31-42. Guilford, J. P., & Zimmerman, W. S. (1956). Guilford-Zimmerman aptitude survey, Beverly Hills, CA, USA: Sheridan. Haberlandt, K. (1999). Human memory: Exploration and application, Boston, MA, USA: Allyn and Bacon. Hall, J., & Cooper, J. (1991). Gender, experience, and attributions to the computer. Journal of Educational Computing Researcher, 7 (1), 51-60. Hanley, G. L. (1987). The origin of information and its effect on problem solving. Paper presented at the Annual Meeting of American Psychological Association, New York, USA, ERIC Document Reproduction Service No. ED291791. Johnson, J. T., Boyd, K. R., & Magnani, P. S. (1994). Causal reasoning in the attribution of rare and common events. Journal of Personality and Social Psychology, 66 (2), 229-242. Logie, R. H. (1995). Visuo-spatial working memory, Hove, UK: Lawrence Erlbaum Associates. Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of Educational Psychology, 88 (1), 49-63. Mayer, R. E. (1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32 (1), 1-19. 117 Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38 (1), 43-52. Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of Educational Psychology, 86 (3), 389-401. McGrew, K. S., & Flanagan, D. P. (1998). The intelligence test desk reference (ITDR): Gf-Gc cross-battery assessment, Needham Heights, MA, USA: Allyn and Bacon. Paas, F., Tuovinen, J. E., Tabbers, H., & Gerven, P. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38 (1), 63-71. Paivio, A. (1986). Mental representations: A dual coding approach, Oxford, England: Oxford University Press. Passig, D., & Levin, H. (2000). Gender preferences for multimedia interfaces. Journal of Computer Assisted Learning, 16 (1), 64-71. Pearson, D. G., & Logie, R. H. (2000). Working memory and mental synthesis: A dual task approach. In Nuallian, S. O. (Ed.), Spatial cognition: Foundations and applications, Amsterdam/Philadelphia: Joint Benjamins Publishing, 347-359. Perfetti, C. A., & Goldman, S. R. (1976). Discourse memory and reading comprehension skill. Journal of Verbal Learning and Verbal Behavior, 15 (1), 33-42. Price, L., & Eliot, J. (1975). Convergent and discriminant validities of two sets of measures of spatial orientation and visualization. Educational and Psychological Measurement, 35 (4), 975-977. Price, P. C., & Yates, J. F. (1995). Associative and rule-based accounts of cue interaction in contingency judgment. Journal of Experimental Psychology, 21 (6), 1639-1655. Reiser, R. (1994). Clark’s invitation to the dance: An instructional designer’s response. Educational Technology Research and Development, 42 (2), 45-48. Richardson, J. T. E. (1996). Evolving concepts of working memory. In Richardson, J. T. E., Engle, R. W., Hasher, L., Loggie, R. H., Stoltzfus, E. R. & Zacks, R. T. (Eds.), Working memory and human cognition, Oxford, England: Oxford University Press, 3-30. Rieber, L. P., & Hannafin, M. J. (1988). Effects of textual and animated orienting activities and practice on learning from computer-based instruction. Computers in the Schools, 5 (1-2), 77-89. Salomon, G. (1979). Interaction of media, cognition, and learning, San Francisco, CA, USA: Jossey-Bass. Sweller, J., & Chandler, P. (1991). Evidence for cognitive load theory. Cognition and Instruction, 8 (4), 351362. Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and Instruction, 12 (3), 185-233. Sweller, J., & Low, R. (1992). Some cognitive factors relevant to mathematics instruction. Mathematics Education Research Journal, 4 (1), 83-94. Tabbers, H. K., Martens, R. L., & van Merrienboer, J. J. G. (2004). Multimedia Instructions and Cognitive Load Theory: Effects of Modality and Cueing. British Journal of Educational Psychology, 74 (1), 71-81. Van Merrienboer, J. G., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17 (2), 147-177. Zheng, R., Miller, S., Snelbecker, G., & Cohen, I. (2005). Use of multimedia for problem-solving tasks. Paper presented at the annual conference of American Educational Research Association, Montreal, Canada. 118 Pragnell, M. V., Roselli, T. & Rossano, V. (2006). Can a Hypermedia Cooperative e-Learning Environment Stimulate Constructive Collaboration? Educational Technology & Society, 9 (2), 119-132. Can a Hypermedia Cooperative e-Learning Environment Stimulate Constructive Collaboration? Mary Victoria Pragnell Faculty of Medicine and Surgery, University of Bari - P.za Giulio Cesare, 1 – 70124 Bari – Italy mpragnellv@libero.it Teresa Roselli and Veronica Rossano Department of Computer Science, University of Bari - Via Orabona,4 - 70126 Bari - Italy Tel: +39 080 5443276 Fax: +39 080 5442424 roselli@di.uniba.it rossano@di.uniba.it ABSTRACT The growing use of the Internet in learning environments has led to new models being created addressing specific learning domains, as well as more general educational goals. In particular, in recent years considerable attention has been paid to collaborative learning supported by technology, because this mode can enhance peer interaction and group work. Among the different active learning strategies, Cooperative Learning has found in the Internet and the World Wide Web the ideal technological support. In this scenario, we have implemented a web-based environment, endeavouring to reproduce within it the traditional teaching methodologies typical of cooperative learning. Herein we describe two experiments aiming to assess the quantity and quality of the interaction promoted by the system and how such factors as gender, background knowledge and role affect communication. We compared the communication and the learning gain achieved in the experimental group that worked with the system, named Geometriamo, with those achieved in the control group working in class with the teacher. The results demonstrate that the use of an environment mediating peer-to-peer collaboration can be highly beneficial, even in primary school age. A high number of messages were exchanged among all pupils and, notably, the highest learning gain was recorded in the less able students in the experimental group. Keywords: Cooperative learning, Collaborative learning, CSCL, E-learning environment, Educational technology Introduction In advanced nations, computers and the WWW in particular are having a strong impact on education. These tools must clearly be regarded as versatile aids rather than as a replacement for face-to-face teaching methods. Nevertheless, the growing use of the Internet in learning environments has enabled new learning models to be created addressing specific learning domains, as well as more general educational goals. In particular, considerable attention has been paid in recent years to collaborative learning supported by technology (Hiltz 1994), because this learning mode can enhance peer interaction and group work. Although in the past the effectiveness of collaborative learning was not widely accepted because academic excellence was defined by individual achievement, the area of research known as Computer-Supported Collaborative Learning (CSCL) has recently shown rapid growth (Johnson et al. 1998, Choen & Scardamalia, 1998, Hakkarainen et al. 1999, Hoadley & Linn 2000, Enyedy et al. 1997). One of the aims of CSCL is to increase the quality of the learning/teaching process by engaging students and teachers in coordinated efforts to build new knowledge and solve problems together (Dillenbourg 1996). This approach offers a good medium for classroom discussions that can facilitate participation and social interaction among students, and between the teacher and students (Shellens & Valcke 2005). In the face-to-face discussions which normally arise in the traditional classroom during collaborative type learning, the teacher’s role is fundamental and shifts from that of unique information source to that of communication facilitator (William & Peters 1997). ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 119 To obtain successful online learning, in both a collaborative and an individual learning situation, three types of interaction must be present, namely learner-content, learner-learner and learner-instructor (Moore 1993). The first type, learner-content, is strictly dependent on the construction and organisation of the material, a task which is traditionally carried out by the teacher. The other two forms of interaction have to do with communication processes in the strictest sense, and depend on the quality of the tools made available in the given CSCL environment. Among the different active learning strategies, much attention has in recent years been paid to Cooperative Learning, that has found in the Internet and the World Wide Web the ideal technological support for setting up collaborative activities with no space-time restrictions. There is still some terminological confusion between the two concepts, although John Myers endeavoured to clarify this, by declaring: “The cooperative learning tradition tends to use quantitative methods which look at achievement: i.e., the product of learning. The collaborative tradition takes a more qualitative approach, analyzing student talk in response to a piece of literature or a primary source in history.” (Myers 1991). Our approach addresses both aspects, analyzing both the quantitative learning gain and the qualitative communication engendered. Naturally, synchronous and asynchronous tools are not enough to ensure effective communication between learners and teacher and the interaction needs to be encouraged by appropriate stimuli leading to positive exchanges of opinion and outlook on the problem to be solved. A web-based cooperative learning environment, therefore, must help to build up a communicative network and provide an environment where student groups of varying sizes can work together to achieve common learning goals (Johnson et al. 1999). Vygotskij (Vygotskij 1978) appraised the psychological implications of cooperative learning and asserted that human beings are products of their cultures, thus people learn from interacting with others around them. According to an analogous approach to the one adopted for intelligent tutoring systems, featuring the introduction of Artificial Intelligence techniques and methods (Schank et al., 1993, Schank et al. 1995, Roselli 1995), in cooperative learning environments the role of communication facilitator can conveniently be assumed by a virtual tutor, that simulates the teacher’s role in a traditional setting. Cooperation has been found to act as a catalyst for the development of particular problem-solving abilities, because learners are able to continue to use the reasoning techniques and strategies assimilated while working with their companions or the teacher, when they later have to solve similar problems alone (van der Meijen & Veenman 2005). This is the theoretical framework within which our research work is inserted. We have implemented a web environment, endeavouring to reproduce within it the traditional teaching methodologies typical of cooperative learning. Our WWW Hypermedial Cooperative Environment, called Geometriamo, is addressed to pupils attending fifth-grade at the elementary school. The experiments we describe were conducted last year and one of the aims, whose results are illustrated herein, was to assess the quantity and quality of the interaction promoted by the system in pupils aged 10-11 years, and how factors such as gender, background knowledge and role can affect communication. Theoretical Background Considerable research has been devoted to the benefits of CSCL. In particular, CSCL environments are considered as satisfactory tools that can: facilitate task-oriented and reflective activity (Choen & Scardamalia, 1998; Hakkarainen et al. 1999, Wegerif, 2004, Gillies, 2004), encourage complex reasoning and deeper levels of argumentation (Hoadley & Linn 2000), support mathematical problem solving (Enyedy et al. 1997, Nason & Woodruff 2003), improve the use of conceptual models (Bell, 1997, Wegerif 2004) and increase students’ cognitive and metacognitive understanding (Brown, et al. 1998, Cohen & Scardamalia 1998, Shellens & Valcke, 2005). A large number of studies comparing CSCL and non-CSCL environments (Lamon et al. 1996) in high school or university settings have demonstrated that CSCL can achieve better results in terms of student learning gain (Berge 2003, Charnistski et al. 2003, Veerman et al. 2001). Only few experiences have yet been reported in the primary school setting (Lipponen 2003, Anastasiades 2003, Tapola 2001, van der Meijen & Veenman 2005), but in these, too, the value of the combination of new educational technologies, computers and the WWW, integrated with a traditional learning pedagogical model such as cooperative learning, has been stressed. However, although many studies of CSCL environments have demonstrated positive results as regards both individual and group learning, others have pointed out some limitations (Guzdial 1997, Guzdial & Turns 2000). Poor student participation has been recorded, measured in terms of the number of messages exchanged and the number of lines per message, while another side effect (Feldman et al. 1999) is that students have been found to 120 tend to indulge in discussion more as a form of social interaction than for true learning purposes. It is now recognised that in distance learning situations, characterised by geographic and physical isolation, new communication strategies need to be implemented, which were not required in face-to-face communication. Thus, the role of online tutor, be it virtual or human, must concentrate on fostering constructive, focused communication achieving a balance between learning of the domain and social exchange. In any case, considerations of a purely quantitative type may not be a sufficient metric for the assessment of the contribution of CSCL to the quality of the interaction, let alone of the efficacy of the resulting learning. As has been shown (Lipponen 2003), other conclusions may be drawn and results obtained if in addition to quantitative assessment some qualitative factors are taken into account, focusing on gender, motivation, communication direction, etc. In this context, the effectiveness of the computer as a support for group work, and its effect on social interaction, take on great importance. In CSCL environments, unlike in face-to-face discussion, communication occurs by means of text messages. Several researchers have recognized the strength of writing as a communication tool. Writing messages gives students the opportunity to reflect, to share their ideas and expertise (Collins et al. 1991), to become more aware of their knowledge and to ask and answer questions without any problem of time limits. All these factors encourage a deeper level of thinking in virtual communities. From the research standpoint, the possibility of recording discussions conducted in cooperative work sessions is of fundamental importance, as a basis for conducting in-depth analysis of the quantity and quality of such communication and its effects on the overall learning process. Cooperative Learning and the method STAD for setting up the CL Already by the beginning of the 1990s, in the USA considerable debate had arisen as to the best way to reform teaching methods in the fields of math and science. Teachers propend for the choice of integrating the traditional face-to-face formula, in which students are provided with information to be memorized, with active learning activities, in which the students are given the opportunity to build on the knowledge gained. Cooperative Learning is achieved by dividing the class up into small groups that work together to achieve the best group results by means of mutual assistance among the group members. All the members of the group must work on the task assigned by the teacher and each is aware that the success or failure of each individual will affect the results of the whole group (“your success benefits me and my success benefits you, in short we all sink or swim together here” (Johnson et al. 1999)). This fosters both a team spirit and a more questioning approach to learning. The learner-interaction variable is obviously the key to success or failure, on which all the other variables depend (motivation, cognitive processes, class organization, assessment, etc.). There are many collaborative and cooperative learning methods, which also can be considered as group learning methods and used in both classroom-based and web-based environments. One of the methods adopted for achieving Cooperative Learning is Student Team Learning (STL) defined by Slavin (Slavin 1980, Slavin 1990). It is fundamentally centred around interaction in small groups but, above all, on individual responsibility and the provision of incentives and rewards to stimulate the group’s individual and collective commitment. STL is one approach featuring various cooperative techniques, including the Student Teams Achievement Divisions (STAD) by Slavin, that we have chosen to implement in Geometriamo because it is suited to teaching disciplines such as mathematics. The focal point of this teaching strategy is interaction among the members of small groups, during which the notion of individual responsibility is reinforced by the fact that it has a strong effect on the final group assessment (Zhao, 2002). This individual accountability motivates students to do a good job of peer tutoring and explaining concepts, as the only way for a team to succeed is if all the team members have mastered the information or skills being taught. The effects achieved with this method are not confined to the cognitive sphere but include fostering interpersonal relationship skills. The teacher, in this context, is seen more as a source, rather in the light of a guide and a sounding board, than as an authoritative figure, and studies conducted by psychologists have demonstrated that this has remarkable positive character-forming effects. The STAD technique subdivides the learning/teaching process into different phases: explanation by the teacher of the topic to be studied; subdivision of the class into small heterogeneous groups; group work; individual assessment; correction of the tests and final grading. 121 Groups should consist of 4/5 students and must be heterogeneous, so that in each group the different levels (good, fair, sufficient, poor) are represented, as well as both sexes and different socio-cultural backgrounds. In the group work the students’ task is to assimilate the concepts learnt during the lesson and help their companions to do so. This work is carried out in two phases: solution of a series of simple exercises supplied by the teacher to learn the concepts studied during the theoretical lesson, and then solution of a complex exercise: the latter is the main goal of the group work. During the subsequent individual assessment tests the students may not help one another because each student must be aware that he or she is responsible for his/her own level of understanding. The individual assessment is transformed by the teacher into a group assessment by summing the individual progress marks. This system of progress assessment allows each student to contribute to the group if, and only if, he or she does his/her best and demonstrates a substantial improvement as the work develops. In the design of our cooperative learning environment the various components needed to reproduce the model in a virtual classroom were defined. Geometriamo, a www hypermedial cooperative environment The idea of building Geometriamo grew out of our awareness that nowadays the Net is very much used in schools, where an increasing number of cooperative experiences have been set up involving several different nations (Olimpo & Trentin 1993). In fact, in advanced nations the schools are equipped with online laboratories that allow large-scale exchanges of experiences at all levels. To examine the needs of schools at different academic levels, we decided to build a hypermedial environment for cooperative learning, Geometriamo, that aims to bring in virtual contact pupils from different classes and schools as a means of fostering constructive exchanges of opinion. In the design and development phases the pedagogical model determined the building of the other components, in order to achieve the ideal environment for cooperative learning according to the STAD. In addition, when implementing the domain, the teachers contributed actively by providing theoretical material and exercises, to ensure the typical content for the scholastic level addressed by Geometriamo, as well as the appropriate level of terminology. The result was found to be a versatile tool combining the features of a tutorial system and a CSCL environment. In fact, in Geometriamo the domain is plane geometry, and the tutorial system is organized in hypermedial form, while the cooperative activity involved is that of assigning an exercise to each group, to be solved together reaching a unanimous conclusion. In the tutorial part, the first knowledge transfer is in the form of “explanation by the teacher of the topic to be studied”. Each topic is organized in pages on Theory and Examples (Fig. 1a and 1b). The former contain the explanations of the various geometric figures and the relative direct and inverse formulas; the latter pose model exercises (already solved) that teach the user the method for solving a geometric exercise. The “group work” phase is set up by the cooperative environment. Each group can resort to a series of Simple Exercises (Fig. 1c) serving to apply what was explained in the Theory, and to the model exercises, as well as carrying out a series of Complex Exercises (Fig. 1d) that represent the task assigned. Each group must solve these by communicating among themselves and agreeing on the problem solving strategy to be used. At the end of the task the solution is communicated to the virtual tutor, which will provide positive feedback and reward, or suggest areas of further study. The chance to work in a group and to interact, discussing a study topic or the solution of a task, was provided by a dedicated component allowing one-to-one and one-to-many communication. To exchange opinions, ask questions and make suggestions, the members of each group can use a messaging notice board that works like email. Figure 2a shows the notice-board of the leader of Group 5. As can be noticed the group members have sent two requests for help. For each message the notice-board displays the sender, the receiver, the date and the number of responses received. Figure 2b shows the form for writing the message, the student has to fill the sender field (implemented using a drop-down menu), the subject and the text field. Moreover, in the left corner of both figures (2a and 2b) of the leader’s notice-board there are three buttons that allow the leader to ask the tutor for help for a particular student, for the group that is having difficulty in cooperating or with a difficult exercise. In the right corner there is the virtual tutor’s notice-board where all the answers sent by the tutor are 122 listed. In the example (Fig 2a) the virtual tutor invites the leader before asking for help for one of the group members, to try to discuss the problem better with him/her and to involve him/her more in the group discussion. 1a. Theory: Pentagons and Hexagons as regular polygons 1c. Simple Exercise: Calculate the perimeter of the Hexagon 1b. Example: calculation of the perimeter of a pentagon 1d. Complex Exercise: calculate the perimeter of GIGI Figure1: Some examples of pages (a)Theory, b) Example, c)Simple and d)Complex exercises) As pointed out above, one of the advantages of text-based communication is that of keeping trace of the contributions by each member. All the messages sent and received are stored and can then be used as tools for assessment of the learning process, providing useful and detailed information on the dynamics of the interaction. For this reason, each member has to register. When a sufficient number of users have registered for the system to create heterogeneous cooperative groups, the collaboration activities can be started. In addition, a leader must be individuated in each group, whose task is that of guiding and moderating the group work. In Geometriamo the leader interacts with the Tutorial Component, and asks for suggestions as to how to “promote” peer interaction. This tutorial component, implemented using Artificial Intelligence techniques, intervenes to stimulate collaborative activities, peer interaction and to suggest revision contents, as the Teacher does in the traditional collaborative classroom. This is possible because Geometriamo includes not only the typical components of an Intelligent Tutoring System (Knowledge Base, Student Module, Tutor Module and Student Interface) (Barr et al. 1982) but also a Group Modeling component (Group Module), that keeps track of the group history, providing the data for suitable intervention by the tutorial component. Moreover, Geometriamo provides an assessment area that includes the “individual assessment” and “correction of the tests and final grading”. In this phase the pupil can take a final test that assesses not only individual improvement but also and above all, the group improvement, calculated as defined in the STAD. This places the emphasis on the group rather than the individual, in accordance with the cooperative philosophy. 123 Figure 2a: Example of leader’s notice-board Figure 2b: Writing messages in notice-board Finally, there is an area devoted to the Teacher, who acts as supervisor, monitoring the work done in each group, in terms of exercises done, pages visited and assessment tests completed. The Effectiveness of Geometriamo as a tool promoting communication To assess the effectiveness of communication promoted by Geometriamo in a computer supported learning situation, two controlled experiments were conducted: a pilot study and a follow-up study. Aim of the evaluations The experiments were carried out on the one hand to assess the learning efficacy achieved in a distance cooperative learning environment and on the other, to see how successfully the electronic tool could mediate communication, and make interaction among the group members more active and productive. Learning Effectiveness Questions Both experiments aimed to answer the following questions: 1. Can children really improve their knowledge of geometric concepts by using the hypermedia? 2. Can the use of the cooperative learning system be as effective as cooperative learning in the classroom mediated by a teacher? 3. Can the use of the cooperative learning be as effective for less gifted students as for more gifted students? Promoting Collaboration Questions The follow-up study also aimed to probe the following aspects: 1. How many notes did each student send to his/her companions? 2. What is the relationship between communication and learners’ ability? 3. How much does the factor of not knowing the other members of the group affect the interaction in a cooperative learning environment? 4. What relationship is there between gender and ability to stimulate communication among group members? These questions were answered according to the method described below. Samples Two experiments were conducted: ¾ A first pilot study was carried out to see whether the environment or the content needed improvements. For this study only 24 pupils were selected. 124 ¾ A follow-up study involved a larger sample: 152 pupils. The pupils in both experiments were attending the fifth class at the primary school “XX Circolo Didattico E. De Amicis” in Bari. Learning using hypermedia was a new experience for all of them, but as we expected, some pupils were familiar with the use of PCs, primarily for playing games and for navigating on Internet to find information. In any case, the fact that some of them had no experience of computers was not a handicap because appropriate training in the use of the PC and the specific environment was given to all the participants. The pupils firstly underwent an individual pre-test, developed in collaboration with the teachers, to assess their prior knowledge of geometry. For each test the teachers assigned an overall mark corresponding to the four levels (excellent, good, fair and poor). The experimental designs Both the experiments were designed using a mixed approach, with pre-test post-test as the within-subjects factor and the between-subjects factor. The children, 24 in the pilot study and 152 in the further investigation, were divided into two groups: the members of the Experimental Group (EG) used the Geometriamo hypermedia and the members of the Control Group (CG) carried out cooperative learning in class with the teacher. The two main groups (EG and CG) were subdivided into cooperative subgroups, 6 in the pilot study and 19 in the follow-up study, each numbering 4 pupils. This number was defined on the basis of the STAD methodology, that prescribes small groups of 4/5 students, as well as convenience, as the system is currently able to manage four unit groups. When forming the groups, care was taken to ensure homogeneity within and among groups both in the experimental and control groups, using both the marks obtained in the pre-test and the teacher’s overall assessment of each pupil for the previous year. Particular care was taken to match for sex, (2 males and 2 females per group), as this factor has a strong effect on communication among 10-11 year-old pupils. In each group it was also necessary to appoint a leader to coordinate the group activities. The EG leaders could ask the virtual tutor for help in motivating the group cooperation, while the CG leaders could ask the teacher. The Experimental Group worked in the two laboratories available in the two school premises connected online to the server made available by our research lab, where the software was installed. The Control Group worked in the classroom, putting the desks of the group members together so that they could work and share materials, knowledge and skills. In the follow-up study, because of the larger sample size, we were able to create more heterogeneous groups. As Figure 3 shows, in each group half of the subjects worked in groups of four including only classmates (denominated as EGClass and CGClass) and the other half in groups of four pupils from other classes participating in the experiment (denominated as EGMix and CGMix). This enabled closer simulation of on-line cooperative learning, in which participants do not know one another. This further subdivision allowed us to assess how much this factor of not knowing the other members of the group can affect the interaction in a cooperative learning environment. Figure 3: Distribution of the pupils in the EG and CG 125 Procedures As stated above, the pupils firstly underwent an individual pre-test to assess their prior knowledge of geometry. Both experimentations consisted of three sessions, each one lasted an hour and a half, with a 2-day interval during the sessions. In the Theory session the teaching unit on the perimeter of a regular polygon was studied by the Experimental Group with the aid of Geometriamo and by the Control Group with the aid of the teacher. In the second session they worked on simple exercises, and in the third, they had to solve a complex exercise consisting of calculation of the perimeter of a compound figure (in Figure 4 the last two sessions are denominated Work Group sessions). One week after the experiment, all the pupils were given a post-test. To try and control confounding variables related to the participants’ history, no other geometric lessons were taught during the period of the experiment and the pupils were not given any homework on the topic. Finally, comparison of the results obtained by each participant, in the pre-test and in the post-test, allowed us to evaluate not only the learning gain but also how much the peer to peer interaction had contributed to improve their knowledge. Figure 4: A graphic depiction of the experimental procedures Results Data analysis yielded the following answers to the research questions listed above. Question 1. Can children really improve their knowledge of geometric concepts by using the hypermedia? From an effectiveness point of view a first important result is shown in Figure 5 and Figure 6. 8 6.4 7.5 6.2 7 6 6.5 Pre-t est 6 Post-test 5.5 Pre-test 5.8 Post-test 5.6 5 5.4 4.5 4 EG CG Figure 5: Average pre-test and post-test score in pilot study 5.2 EG CG Figure 6: Average pre-test and post-test score in follow-up study 126 Moreover, Geometriamo provides an assessment area that includes the “individual assessment” and “correction of the tests and final grading”. In this phase the pupil can take a final test that assesses not only individual improvement but also and above all, the group improvement, calculated as defined in the STAD. This places the emphasis on the group rather than the individual, in accordance with the cooperative philosophy. Question 2. Can the use of the cooperative learning system be as effective as cooperative learning in the classroom mediated by a teacher? Table 1 and Table 2 report the t-test analysis, which was applied to the results of the post-test to test the difference between the performances of the experimental and control groups. The predetermined alpha level adopted for hypothesis testing was 0.05. The results confirmed the experimental hypothesis claiming an equal improvement over time in the two experimental conditions. This implies that all the children learned during the experiment and that this learning was almost identical in both groups. Therefore we can assert that no significant difference due to the mode of instruction was found between the two groups in either experiment. Table 1 t-test analysis in pilot study EG CG N Mean learning gain Standard Deviation 12 12 1.33 1.17 2.67 1.59 Total Standard Deviation = T(22) = t= 2.198 0.186 1.717 Table 2. t-test analysis in follow-up study N EG CG 76 76 Mean learning gain 0.66 0.21 Total Standard Deviation = T(150) = t= Standard Deviation 2.30 3.08 2.722 1.007 1.67 Question 3. Can the use of the cooperative learning be as effective for less gifted students as for more gifted students? Scattering of data evinced an interesting finding reported in Figure 7, which shows that in the EG group pupils with a lower score in the pre-test had a higher learning gain. These results seem to confirm what previous research had pointed out: the use of the computer may promote better and freer communication among members of a group, reinforcing the inherent characteristics of the cooperative learning model. Very likely, the less directly personal confrontation occurring online encourages students with greater learning difficulties to voice their opinions and doubts, and this freer sounding-board effect benefits their overall understanding of the concepts under discussion. Furthermore, the use of text-based communication gives students more opportunities for reflection and exploration of their own knowledge and this encourages deeper levels of thinking, a point which is particularly useful in less gifted students, who can thus realize what it is that they don’t know. The final result seems to be that of reducing the gap between less and more able students. Question 4. How many notes did each pupil send to his/her companions? The above results were confirmed by the number of messages exchanged during interaction with Geometriamo. The data collected in the Follow-up study demonstrate that the messaging component was amply used. In fact, during the three working sessions the 20 groups exchanged an average of 100 messages. Consistent use of the messaging function is confirmed by analysis of the individual messages sent (Figure 8), which showed that only 7 of 76 pupils sent less than 10 messages per session, while all the others had taken full advantage of the communication tool. 127 30 5% 25 4% Less able 3% More able 2% Number of students 6% 20 15 10 5 1% 0 0% 0-10 Pilot Study Follow-up Study 11-20 21-30 31-40 41-50 51-60 61-70 Number of messages sent by each student Figure 7. Learning gain of less gifted and more gifted pupils in Pilot study vs. Follow-up study Figure 8: Number of messages sent per individual Question 5. What is the relationship between communication and learners’ ability? Moreover, analysis of the number of messages exchanged according to the classification of the learner’s ability (excellent, good, fair and poor) showed that as expected, the group leaders exchanged the highest number of messages because they had been made responsible for the promotion of communication, and more gifted pupils also communicated very extensively, as shown in Figure 9. On the face of it, and comparing these with the results discussed in Question 3, this would seem to imply that a greater level of interaction does not therefore result in a true learning gain. In fact, however, even if more gifted pupils sent more messages than less gifted ones, the learning gain was actually higher in the latter pupils. It would appear that even if they tended to experience the discussion rather more passively, less gifted students in any case felt themselves to be involved in the developing debate and were able to absorb the growing knowledge. In fact, the rare messages they sent tended to be requests for further explanation. This tendency was less marked in the EGClass than in the EGMix, showing that familiarity does, to a certain extent, smooth out such communication difficulties. In any case, even if messages were not directly addressed to them, the most gifted students took part in subsequent explanations, thanks to the stimulation provided by the Tutorial Component, that had the express task of promoting peer interaction and reviving it in case of difficulties. In short, closer analysis shows that the high level of communication by the leaders is a demonstration of the achievement of another objective of the cooperative activity, that of encouraging more gifted students to help their less gifted peers. Furthermore, it is interesting to observe that the tendency towards a higher number of messages exchanged among students in the EGClass than in the EGMix is reversed in the group of students classified as poor, reinforcing the concept that they are less overawed in an online and therefore more impersonal setting. 40 12% 35 10% 30 8% 25 20 EG Class 15 EG Mix 6% Class Group 4% Mix Group 2% 10 5 0% 0 -2% Excellent Good Fair Poor CG EG Figure 9: Number of messages exchanged per class of ability Figure 10 Learning gain in the class groups and mixed groups in CG and EG 128 Question 6. How much does the factor of not knowing the other members of the group affect the interaction in a cooperative learning environment? Further analyses in this direction yielded particularly interesting results as shown in Figure 10, illustrating the comparison of the results obtained in the follow-up study, in the two groups (EG e CG) in their respective subdivisions (EGClass and CGClass versus EGMix and CGMix). As expected, even if all the groups showed an overall improvement in the EG, this was higher in the class groups than in the mixed classes groups, because children find it easier to work with their classmates. Instead, in the CG, although the class groups improved, the mixed class groups had negative results. One of the factors which could explain the results obtained is the communication medium used. In fact, while face-to-face communication inhibited pupils in the control group mixed classes, especially the shy ones, in the experimental group the mediation of the electronic system tended to overcome the psychological barriers created in mixed cooperative situations (a fear of asking for help, jealousy, pride in one’s own abilities, etc,…). Question 7. What relationship is there between gender and ability to stimulate communication among group members? During group work sessions, the figure of the leader took on a fundamental importance in the task of stimulating communication by encouraging cooperation among his/her companions. Analysis of the messages sent by each leader revealed that female leaders appeared to be more able to involve the group and encourage cooperation among members than their male counterparts (see Figure 11). However, this first impression was revised when the type of messages sent was considered in greater depth. Figure 12 compares the number of messages sent (quantity) with their type (quality). In this analysis we assessed how many of the messages sent by each group leader were directly related to the domain topic. The results were found to be approximately equal for male and female group leaders. In short, the difference in the results for overall communication seems to be due to a greater number of messages having been sent for relational purposes by female group leaders. Again, this result is not unexpected in the age group involved in our experimentation. In any case, overall, the results show that regardless of gender, the group leader is absolutely essential as the fulcrum around which cooperation can be built up and a miniature learning community created. Are girls or boys m ore successful as group leader? 30 25 20 Girls 15 Girls Boys Boys 10 5 0 Figure 11 Correlation between gender and ability to stimulate communication in the group Relationship messages Topic messages Figure12. Ratio of relational versus topic-related messages sent by leader Conclusions and future works A multiplicity of different learning methods using the web as the communication tool, many of which are labelled as “non traditional”, are now available at all levels of education, and addressing all types of learners. They are now so widespread, and the quality of the learning promoted is so strongly supported by experimental evidence, that it is no longer justifiable to consider distance learning facilities purely as an alternative method for stimulating motivation in learners or for solving space-time problems. The ability to develop metacognitive skills fostered by CSCL environments is a very important aspect. The experiments conducted with our cooperative environment also aimed to address the synergic aspects of cooperative and collaborative learning; they demonstrate that a number of benefits can be gained with the use of an environment that mediates peer-topeer communication even in primary school age. Collaboration skills are, after all, indispensable in our modern world, where a multidisciplinary approach is becoming the norm. After the first sense of diffidence toward the different communication tool had worn off, the experimental subjects appreciated the chances offered to talk to 129 pupils in other buildings and to share not only their knowledge and skills in the field of geometry but also the pleasure of building relationships with new, unknown peers. Our experimental results confirm the value of our environment as a tool for stimulating collaboration in a cooperative learning situation. A high number of messages was sent in all EG groups. Comparison between the communication that took place in the EG and in the CG showed that although the efficacy results (Roselli 2003) demonstrated a learning gain in both groups, Computer-Mediated Communication was more successful than classroom communication for the purposes of fostering collaborative learning. Indeed, we found that the less directly personal confrontation occurring online seems to encourage students with greater learning difficulties to voice their opinions and doubts, and this freer sounding-board effect benefits their overall understanding of the concepts under discussion. Even if students classified in the lower aptitude groups tended to send less messages, and hence to experience the discussion rather more passively, in any case the less gifted students appeared to feel themselves involved in the developing debate and were able to absorb the growing knowledge. Another interesting aspect, namely analysis of the communication from the gender standpoint, was that while boys tended to send less messages than girls, their messages were more topic-related, while girls also sent many messages having a more relational purpose. Overall, the topic-related communication was comparable for the two genders. All these results have encouraged us to extend research in this area. We are at present carrying out a reorganisation of the teaching content, modifying the environment to make it suitable for use with high school students. Further investigations will also be made of Geometriamo in the current version but in different contexts, for example by using it with children of different nationalities, to see whether the results obtained in the present experience are confirmed in other cultural milieus. References Anastasiades, P. S. (2003). Distance learning in elementary schools in Cyprus: the evaluation methodology and results. Computers & Education, 40 (1), 17-40. Barr, A. & Feigenbaum, E. (1982). The handbook of Artificial Intelligence (Vol. 1 & 2), Los Altos, CA, USA, Morgan Kaufmann. Bell, P. (1997). Using argument representations to make thinking visible for individuals and groups. In Koschmann, T., Hall, R., & Miyake, N. (Eds.), CSCL II: Carrying Forward the Conversation, NJ, USA: Lawrence Erlbaum Associates, 10-19. Berge, Z. L. (2003). Interaction in Post-Secondary Web-Based Learning, retrieved November 29, 2005 from http://www.saskschools.ca/~parkland/interaction.htm. Brown, A. L., Ellery, S., & Campione, J. C., (1998). Creating zones of proximal development electronically. In Greeno, J. G. & Goldman, S. V. (Eds.), Thinking practices in mathematics and science learning, NJ, USA: Lawrence Erlbaum Associates, 341-367. Charnistski, C. W., Molinaro, J. A., Corabi, J., & Nolan, K. (2003). Comparing Student Achievement in a Graduate Level Research Methods Course Using Face-to-Face and Web-based Instruction: Result of a Pilot Study. In Rosset, A. (Eds.) World Conference on E-Learning in Corp., Government., Health., & Higher Education (ELEARN), VA, USA: Association for the Advancement of Computing in Education (AACE), 174177. Choen, A., & Scardamalia, M. (1998). Discourse about ideas: monitoring and regulating in face-to-face and computer-mediated environments. Interactive Learning Environments, 6 (1-2), 93-113. Collins, A., Brown, J. S., & Holum, A. (1991). Cognitive apprenticeship: making thinking visible. American Educator, 15 (6-11), 38-46. Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C (1996). The evolution of research on collaborative learning. In Reimann, P. & Spada, H. (Eds.), Learning in Humans and Machines: Towards an interdisciplinary learning science, UK: Elsevier, 189-211. 130 Enyedy, N., Vahey, P., & Gifford, B. R. (1997). Active and supportive computer-mediated resources for student to student conversations. In Koschmann, T., Hall, R. & Miyake, N. (Eds.), CSCL II: Carrying Forward the Conversation, NJ, USA: Lawrence Erlbaum Associates, 27-36. Feldman, A., Konold, C., Coulter, B., Conroy, B., Hutchison, C., & London, N. (1999). Network science, a decade later. The internet and classroom learning, NJ, USA: Lawrence Erlbaum Associates. Gillies, R. M. (2004). The effects of cooperative learning on junior high school students during small group learning. Journal of Learning and Instruction, 14 (2), 197-213. Guzdial, M. (1997). Information ecology of collaboration in educational settings: influence of tool. In Koschmann, T., Hall, R. & Miyake, N. (Eds.), CSCL II: Carrying Forward the Conversation, NJ, USA: Lawrence Erlbaum Associates, 83-90. Guzdial, M., & Turns, J., 2000. Effective discussion through a computer-mediated anchored forum. Journal of the Learning Sciences, 9 (4), 437-469. Hakkarainen, K., Lipponen, L., Jarvela, S., & Niemivirta, M. (1999). The interaction of motivational orientation and knowledge-seeking inquiry in computer-supported collaborative learning. Journal of Educational Computing Research, 21 (3), 261-279. Hiltz, S. R. (1994). The virtual classroom: Learning without limits via computer networks, NJ, USA: Ablex Publishing Corporation. Hoadley, C. M., & Linn, M. C. (2000). Teaching science through online peer discussions: speakeasy in the knowledge integration environment. International Journal of Science Education, 22 (8), 839-857. Johnson, D. W., Johnson, R. T., & Smith, K. A. (1998). Cooperative learning returns to college. Change, 30 (4), 26-35. Johnson, D. W., & Johnson, R. T. (1999). Learning together and alone: Cooperative, competitive and individualistic learning (5th Ed.), Boston, USA: Allyn &Bacon. Lamon, M., Secules, T., Petrosino, A., Hackett, R., Bransford, J., & Goldman, S. (1996). Schools for thought: overview of the project and lessons learned from one of the sites. In Schauble, L. & Glaser, R. (Eds.), Innovations in learning: new environments for education, NJ, USA: Lawrence Erlbaum Associates, 243-288. Lipponen, L. Rahikainen, M., Lallimo, J., & Hakkarainen, K. (2003). Patterns of participation and discourse in elementary students’ computer-supported collaborative learning. Journal of Learning and Instruction, 13 (5), 487-509. Moore, M. (1993). Three types of Interaction. In Harry, K. Hohn, M & Keegan, D (Eds.), Distance Education: New Perspective, London, UK: Routledge, 19-24. Myers, J. (1991). Cooperative learning in heterogeneous classes. Cooperative Learning, 11 (4). Nason, R., & Woodruff, E. (2003). Fostering Authentic, Sustained, and Progressive Mathematical KnowledgeBuilding Activity in Computer Supported Collaborative Learning (CSCL) Communities. Journal of Computers in Mathematics and Science Teaching, 22 (4), 345-363. Olimpo, G., & Trentin, G. (1993) La telematica nella didattica: come e quando. Tecnologie Didattiche, 2, 4-17. Roselli, T. (1995). Artificial Intelligence Can Improve Hypermedia Instructional Technologies for Learning. ACM Computing Surveys, 27 (4), 624-626. Roselli, T., Faggiano, E., & Rossano, V. (2003) A Further Investigation On The Effectiveness Of A WWW Hypermedial System for Cooperative Learning. In Tin Lee, K. & Mitchell, K. (Eds.), International Conference on Computer in Education, VA, USA: Association for the Advancement of Computing in Education (AACE), 440-442. 131 Schank, R. C., Davenport, G., Ghislandi, P., Spohrer, J. C., & Wilson, K. (1993). A.I. Multimedia and Education. In Bajcsy, R. (Ed.), Thirteenth International Joint Conference on Artificial Intelligence. CA, USA: Morgan Kaufmann Publishers, 1667-1672. Schank, R. C., Korcuska, M., & Jona, M. (1995). Multimedia Application for Education and Training: Revolution or Red Herring? ACM Computing Surveys, 27 (4), 633-635. Shellens, T., & Valcke, M. (2005). Collaborative learning in asynchronous discussion groups: What about the impact on cognitive process? Computers in Human Behavior, 21 (6), 957-975. Slavin, R. E (1980). Using Student Team Learning, Baltimore, MD: The John Hopkins University, Center for the Social Organization. Slavin, R. E. (1990). Cooperative Learning: theory, research, and practice, Englewood Cliffs, NJ, USA: Prentice-Hall. Tapola, A. (2001). Participation and motivation in computer-supported collaborative learning. In proceedings of World Conference on Educational Multimedia, Hypermedia, and Telecommunications, VA, USA: Association for the Advancement of Computing in Education (AACE), 1873-1878. van der Meijen, H., & Veenman, S. (2005). Face-to-Face versus computer-mediated communication in a primary school setting. Computers in Human Behavior, 21 (5), 831-859. Veerman, A., & Veldhuis-Diermanse E. (2001). Collaborative learning through computer-mediated communication in academic education, retrieved March 20, 2005 from http://www.ll.unimaas.nl/eurocscl/papers/166.doc. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes, Cambridge, MA, USA. Harvard University Press. Williams, V., & Peters, K. (1997). Faculty Incentives for the preparations of Web-based Instruction. In Khan, B. H. (Ed.), Web-based Instruction, NJ, USA: Educational Technology Publications. Wegerif, R. (2004). The role of educational software as a support for teaching and learning conversations, Computers & Education, 43 (1-2), 179-191. Zhao, J., McConnel, D., Li, K., & Akahori, K. (2002). STAD Modeling Method for Computer Supported Group Learning. In Kinshuk, Lewis, Akahori, Kemp, Okamoto, Henderson & Lee (Eds.), International Conference on Computer in Education, Los Alamitos, CA, USA: IEEE Computer Society Press, 344-348. 132 Chan, H., Tsai, P., & Huang, T.-Y. (2006). Web-based Learning in a Geometry Course. Educational Technology & Society, 9 (2), 133-140. Web-based Learning in a Geometry Course Hsungrow Chan Associate Professor, Department of Mathematics Education, National Pingtung University of Education No.4-18, Minsheng Rd., Pingtung City, Pingtung County 900, Taiwan (R.O.C.) hchan@mail.npttc.edu.tw Tel: +886-8-7226141#6200 or 6210 Fax: +886-8-7239860 Pengheng Tsai Doctoral Student, Department of Industrial Technology Education, National Kaohsiung Normal University No.58, Lane 59, Longsing St., Cianjhen District, Kaohsiung City 806, Taiwan (R.O.C.) pengheng.tsai@msa.hinet.net Tel: +886-7-7510048 Fax: +886-7-7510445 Tien-Yu Huang Professor, Department of Computer Science, National Pingtung University of Education No.4-18, Minsheng Rd., Pingtung City, Pingtung County 900, Taiwan (R.O.C.) tyhuang@mail.npttc.edu.tw Tel: +886-8-7226141#3107 or 3110 Fax: +886-8-7215034 ABSTRACT This study concerns applying Web-based learning with learner controlled instructional materials in a geometry course. The experimental group learned in a Web-based learning environment, and the control group learned in a classroom. We observed that the learning method accounted for a total variation in learning effect of 19.1% in the 3rd grade and 36.5% in the 6th grade. The main factor in the difference was attributed to the students’ ability to apply information technology. We further found that students need more communication with peers, teachers, and lecture materials when learning via the Internet. We are encouraged that progress in the technological advances of the Internet will make it more convenient and more effective for use in Web-based learning. Keywords: Classroom learning, Geometry curriculum, Learner control, Van Hiele level of geometric thought, Web-based learning Introduction Elementary students in Taiwan usually learn geometric concepts by recitation and through practice. Most of them fail to learn basic geometric concepts and geometric problem-solving, so they obviously don’t view themselves as being successful in geometry courses. Some elementary geometry students have a lot of difficulty with ordering properties logically and general definitions. This situation gets worse later in Junior and High School. Taiwan is not alone in this. Much learning of geometric concepts by U.S. students has been by rote; they frequently do not recognize components, properties, and relationships between properties (Clements & Battista, 1992). Many mathematics educators have studied children’s geometric conceptions. The three dominant lines of inquiry concerning children’s geometric conceptions are based on the theories of Piaget, the van Hieles, and cognitive psychologists (ibid). While the Piagetian and cognitive psychology studies were not grounded in educational concerns, the van Hielian research, by contrast, was so-grounded (Clements, Swaminathan, Hannibal, & Sarama, 1999). Thus, this study is based on their work. According to the van Hiele theory, students progress through levels of thought in geometry when aided by instruction (van Hiele, 1986). Thinking on geometric concepts develops from an initial, Gestalt-like visualization through increasingly sophisticated levels: descriptive and analytic, abstract and relational, formal deductive, and mathematically rigorous. The five van Hiele levels are thought of in the following ways. 1. The visual level: Children identify shapes according to appearance, recognizing them as visual gestalts using visual prototypes, saying, for instance, that a given figure is a circle because “it looks like a moon.” 2. The descriptive and analytic level: Students reason about geometric concepts by means of an informal analysis of component parts and attributes. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 133 3. 4. 5. The abstract and relational level: Students order properties logically and begin to appreciate the role of general definitions. They can form abstract definitions and distinguish between the necessity and sufficiency of a set of properties in determining a concept. The formal deductive level: The role of axioms, undefined terms, and theorems is fully understood, and original proofs can be constructed. The mathematically rigorous level: Students can compare various axiomatic systems based on various axioms, and study various geometries in the absence of concrete models. Helping children move through these levels may be taken as a critical educational goal (van Hiele, 1986). The main aim of geometric education is to change students’ geometric conceptions through teaching activities, and move students from one level of van Hiele thinking to the next. Using an elementary geometry curriculum, the first purpose of this study was to find a way to help elementary students use the learning effect to advance their van Hiele levels of geometric thought. Previous lines of inquiry into children’s geometric conceptions have provided useful foundations, but have also left gaps that impede curriculum development and teaching improvements. And as briefly explained above, much learning of geometric concepts by Taiwan students has been by recitation and through practice. We should allow students to explore geometric concepts and shapes, and must provide experiences in informal geometry for students throughout their school years. However, because of limited classroom space, teaching time, and inability to interact, classroom teaching and learning contexts in traditional schools haven’t been successful in geometry courses, and students always treat the geometric models as toys. Moreover, instructional control and program control are administered by teachers in classrooms, and interactive activities are few or even nonexistent. So even though teachers perform quite well, no instructional records are preserved and it is impossible for students to review the instructional interactive process. The concept of learner control, defined as an individual ability to sequence the content while engaged in an instructional activity, is now extended to include allowing students to control the pace of instruction (Mager & Clark, 1963; Gay, 1986; Reiser, 1984). Several studies suggest that learner control of instruction can result in more positive student attitudes, higher levels of student engagement, and less anxiety (Fowler, 1983; Hannafin, 1984). Accordingly, this study involved using the concept of learner control in conducting a Web-based geometry course. However, implementation did entail some risk. Learner-controlled instructional materials may improve students’ performance and attitudes toward the subject matter, and increase efficiency, but they also produce large variations in students’ performance that may result from instructional tasks, instructional designs or assessment procedures (Hannafin, 1984). Lepper (1985) suggests that providing learner control increases the relevance of instruction especially if the activity involves the external environment. Through careful combinations of course exposition text, images, expository videos, and 3D virtual geometric objects, the prototypic module strives to simulate instructional activities and geometric mental manipulation action. Our creation of Web-based materials had the ultimate goal of generating complete support for vertical and horizontal integration in the geometry course. Our experimental design embedded the instructional material scaffolding in the Internet, and allowed students to arrange their programs, manipulate 3D virtual geometric objects, and discuss online with teachers and peers. That is, instructional control and program control were returned to students. Here, teachers served as counselors and guides. Hence, the second purpose of this study was to promote the learning effect and level of geometric thought of elementary students with Web-based learning. This study designed and applied Web-based cube and cuboid materials for 3rd grade students, and prism, pyramid, cylinder, and cone materials for 6th grade students. Upon completion, we used our own test to assess the learning effect and the van Hiele level of geometric thought. We also interviewed the sampled students to determine how thoroughly they grasped the geometric concepts, how well they accepted this different learning method, and their perceptions on using Web-based learning. Method Participants The experimental group, which received Web-based instruction, included thirty-five 3rd grade students and thirty-nine 6th grade students. The control group, which received classroom instruction, included thirty-four 3rd grade students and forty 6th grade students. All subjects were from an elementary school in Kaohsiung City, Taiwan. 134 Web-based learning environment This study applied the Web-based learning environment whose structure is shown in Figure 1. The environment is divided into a teachers’ management interface, students’ learning interface, and system support interface. Teachers’ management interface: Enables teachers to manage all parts of the courses, design Web-based materials, record lecture videos, administer teaching files, and test students. Students’ learning interface: Students logging on to this environment can choose studying the geometry course, controlling the instruction program and content, asynchronous and real-time communication, and taking tests. In particular, they can manipulate 3D virtual geometric objects using a mouse. System support interface: This provides course management, including page setting, personal e-mail, and discussion and assignment areas so that instruction activities proceed smoothly. These interfaces are integrated by the database management function which, includes the user, curriculum, Webbased materials, and instruction activities record databases. Figure 1 Web-based learning environment Achievement tests The aim of this study was to find a way to help promote learning and van Hiele levels of geometric thought among elementary students. We developed our pre- and post-tests according to the criterion-referenced van Hiele theory of content validity. Each test consisted of three parts: 1. identifying and denominating geometric shapes on level 1; 2. informal analysis of geometric component parts and geometric shape attributes on level 2; 3. forming abstract definitions and distinguishing geometric shapes on level 3. The Pearson correlation coefficients for the pre- and post-tests were 0.8054 for the 3rd grade and 0.736 for the 6th grade as alternate-form reliability, and the Cronbach coefficient for the pre- and post-tests were 0.8289 for the 3rd grade and 0.8482 for the 6th grade, with a 0.05 confidence level. Results Learning effect Table 1 shows descriptive statistics for both groups’ pre- and post-tests. We found that learning effects in both groups were promoted in the geometry course. 135 Table 1. Summary of means; SD: the learning effect determined by achievement test 6th Grade 3rd Grade Pre-test Post-test Pre-test Post-test Group n Group n Means SD Means SD Means SD Means SD Ex 35 66.86 10.42 78.03 13.36 Ex 39 46.77 18.99 72.21 23.23 Con 34 71.94 9.96 80 16.89 Con 40 46.98 17.88 61.93 22.54 Ex = experimental group, Con = control group Table 2. Summary of ANCOVA: the learning effect of different learning methods, *p<.05 Sources of variation df Mean Square F Sig. Type Ⅲ Sum of Squares Between groups 37.061 1 37.061 0.2 0.656 3rd Grade (learning method) Error 12214.466 66 185.068 Eta Squared = 0.003, Observed Powerα =0.073, r2 = 0.191 Between groups 1756.824 1 1756.824 5.136 0.026 6th Grade (learning method) Error 25994.814 76 342.037 Eta Squared = 0.063, Observed Powerα =0.609, r2 = 0.365 Grade After removing the influence of the covariance, F = 0.2 for the 3rd grade in Table 2 is below the 0.05 significance level, which means there was no significant difference in learning effect between the Web-based and classroom learning environments. Taking a closer look at the strength of association (r2 = 0.191), we see that learning method accounted for 19.1% of the total variation in learning effect in the 3rd grade class, but in the 6th grade class, F = 5.136 reached the 0.05 significance level, which means the learning effect in the Web-based learning environment differed significantly from the learning effect in the classroom learning environment. Checking the strength of association (r2 = 0.365), we see that the learning method accounted for 36.5% of the total variation in learning effect in the 6th grade class. Tabl e 3 . van Hiele Level Distribution on Pre-test and Post-test 3rd Grade 6th Grade Test Level Ex (n＝35),％ Con (n＝34),％ Ex (n＝39),％ Con (n＝40),％ Prism, Pyramid, Prism, Pyramid, Cube Cuboid Cube Cuboid Cylinder Cone Cylinder Cone 30.8 20.5 42.5 30 * 0 0 0 0 48.6 45.7 26.5 44.1 30.8 48.8 30 50 1 Pre51.4 54.3 70.6 53 35.9 28.2 22.5 15 2 test 0 0 2.9 2.9 2.5 2.5 5 5 3 7.7 5 12.5 12.5 * 0 0 0 0 8.6 11.5 2.9 5.9 17.8 28.2 15 30 1 Post88.6 85.7 79.4 76.4 41 33.4 62.5 47.5 2 test 2.8 2.8 14.7 14.7 33.4 33.4 10 10 3 * = unreached Level 1 Table 3 shows that the van Hiele level of geometric thought in both groups was promoted from level 1 to level 2 or level 2 to level 3. The promotion percentage in the 3rd grade control group was greater than that of the experimental group, however, in the 6th grade, the promotion percentage was greater in the experimental group than in the control group. Group interview Students were questioned in a group interview about the various learning settings, with the interviewer recording the students’ answers (I = interviewer). After the interview, the students were asked to make two perspective drawings. Six students from each group were sampled and answers were coded as follows: E = experimental group, C = control group, 3 = 3rd grade, 6 = 6th grade; score intervals are: 1 = high score, 2 = middle score and 3 = low score, and serial numbers are 1 and 2. Thus, code E312 represents the 2nd sample in the high score interval of 3rd grade students in the experimental group. Students’ verbal statements are given below: 136 Experimental group I: Just tell me about the activities of the geometry course via the Internet? E312: I watch the videos and the Web, then I discuss with teacher and classmates in the chat room. E321: I do the same things, and I think the 3D things in the screen are interesting. E331: But, sometimes I don’t know what to do. E611: I can receive the reply from teacher by e-mail or bulletin board. E622: When I click the hyperlink, I can explore the information of the geometry course. E631: Yes, if I can’t understand the geometry course, I can watch the videos again. Control group I: Just tell me about the activities of the geometry course in the classroom? C311: Teacher asks us to open the textbook, we go page by page. C322: And we make some paper models and play with them. C332: I don’t know what teacher says, I forget. C612: We read and listen to what teacher teaches. After school, I do homework and complete exercises. C622: At times, we make too much noise, and teacher silences us. But that is not all my fault. C631: I don’t know who I can ask about the geometry problems. Our teacher just teaches all the time. I: Can you tell me how you play with the 3D animations? E311: I can rotate and move them with the mouse. E332: I can find their surfaces, sides and corners. E612: I can operate them easily with the mouse, their solid shapes are clear. E632: I have seen 3D things in video games, so I know how to use them. I: Can you tell me how you play with the paper models? C312: I can make the paper models and observe them. C331: The paper models are like building blocks, I can use them to build a house. C611: Teacher tells us how to make them, I can understand more about geometry. C632: I can’t make the paper models. I: How do you feel the Internet discussions went? E311: I can discuss with teacher and classmates in the chat room, and send my questions. I can answer others’ questions, too. E322: I don’t worry about making too much noise on the Web, or teacher asking me to be quiet. E331: I can see what I don’t understand while they talk. E612: It’s perfect. I can communicate with teacher and classmates by typing. E621: And we share our thinking on the bulletin board or by e-mail. E632: Although I type slowly, everyone can see what I type in the list. I: How do you feel the classroom discussions went? C312: I can answer every question teacher asks. C321: Teacher doesn’t notice when I raise my hand most times. C331: Nobody asks me, and I don’t want to reply to any questions. C612: In my group, they don’t listen to me. How can I complete the task? C621: I don’t need to do too much. Jay and Lin usually will do the whole job. C632: I don’t know how to join them, they are too smart. I: What is the most difficult for you on the Internet? E312: Nothing at all! I feel it is interesting. Will we have another course? E321: I don’t see a real teacher. E332: Typing is the most difficult thing for me. I can’t keep up with others. E611: Too many others want me to help them and reply to them. I don’t like it. E622: People saying something dirty in the chat rooms is unlovely. E631: I can’t type fast, because it is hard for me to spell well. I: What is the most difficult for you in the classroom? C311: Nothing is difficult for me in the classroom, I do whatever teacher says. C322: I can’t solve some homework exercises of the, and I can’t find any help. C332: The geometry course is the most difficult for me anyway. C611: I think I don’t have enough ability to be the leader of the team. C622: I can’t join discussions. Some people are uncooperative. C631: Everything is difficult for me. I wish I could understand geometry better. Interpretation of the students’ responses to the questions above helped to explain the quantitative results obtained with the ANCOVA. The group interview results can be classified into three main viewpoints on the Web-based and classroom-based learning settings. The first viewpoint is that students in high score intervals could complete most of the learning objectives and solve problems in both 3rd grade and 6th grade settings. The second viewpoint is that students in the middle score interval of control groups of both grades hardly gained any help when they faced problems with geometry questions. Students in the middle score interval of the 6th grade experimental 137 group could acquire help from the chat room, bulletin board, or e-mail, but not the 3rd grade. The third viewpoint is that the main difficulty students in the low score intervals of both grades’ control groups had was in the content of the geometry course. Another difficulty students in the low score intervals of both grades’ experimental groups had was inability to use computers and the Internet. In particular, they all complained of slow keying-in and spelling. The Perspective drawings Code E311 Cube Table 4. Perspective drawings by 3rd grade and 6th grade students Cuboid Code Cube C311 E312 C312 E321 C321 E322 C322 E331 C331 E332 C332 Code E611 Prism or cylinder Pyramid or cone Code C611 E612 C612 E621 C621 E622 C622 E631 C631 E632 C632 Prism or cylinder Cuboid Pyramid or cone 138 Students need both spatial and drafting ability. From Table 4, we can see that the students with higher van Hiele levels made better drawings. However, while subject C632 had better sketching ability, he couldn’t reach level 1 of the van Hiele theory. Discussion The results above show that learning method accounted for a total variation in learning effect of 19.1% in the 3rd grade and 36.5% in the 6th grade. And the promotion percentage in van Hiele level of geometric thought in the 3rd grade control group was greater than the promotion percentage in the experimental group, but the promoted percentage in van Hiele level of geometric thought in the 6th grade experimental group was greater than the promotion percentage in the control group. The group interview results explain why learner-control instruction indeed improved students’ achievements, but also produced large variations in students’ performance. Those variations may come from the instructional task, instructional design, or assessment procedure results. In other words, students in the experimental groups thought that learning in the Web-based learning environment was interesting, helpful, and convenient, but some had difficulty typing and manipulating computers. A few also felt that the lecture video wasn’t real. The influence of these problems was more serious than the shortcomings of classroom learning in the 3rd grade, however, the reverse was true in the 6th grade. That is, the ability to learn with computers and the Internet were the important factors in learner-controlled geometry instruction in the Web-based learning environment. For reasons mentioned above, the 6th grade were possessed of better computer skills and experiences than 3rd grade, so they could benefit more learning effect than the 3rd grade in a geometry course via the Web-based learning environment of learner control. Conclusion The Web-based geometry course in this study was implemented with learner controlled instructional materials. We observed that the 6th grade students’ ability to learn with computers and the Internet was sufficient to handle problems and promote their van Hiele levels of geometric thought. But this instructional technology seemed to threaten 3rd grade students’ learning. Our research goals were confined by the 3rd grade students’ ability to apply information technology. Research into the application of the Internet has been an amazing journey and led to some interesting observations. The evidence from the group interview pointed out that students need more communication with peers, teachers, and lecture materials, but this need wasn’t satisfied for students in the control group. However, students in the experimental group could be included in a navigational scheme that reflected a balance between instructional control and program control based on learner control. This balance could let students in the experimental group meet the requirements of curricular discussion. Thus, it will help them to construct their geometric concepts and get greater achievement. We realize that this is but one small step in a long and challenging journey. The ever-growing presence of the Web should facilitate necessary levels of communication to continue our research and development. The technological advances in Internet progress will make it more convenient and more effective for use in Web-based learning. References Clements, D. H., & Battista, M. T. (1992). Geometry and spatial reasoning. In Grouws, D. A. (Ed.), Handbook of reasoning on mathematics teaching and learning, New York, USA: Macmillan, 420-464. Clements, D. H., Swaminathan, S., Hannibal, M. A. Z., & Sarama, J. (1999). Young children’s concepts of shape. Journal for Resaerch in Mathematics Education, 20 (2), 192~212. Fowler, J. F. (1983). Use of computer-assisted instruction in introductory management science. Journal of Experimental Education, 52, 22-26. Gay, G. (1986). Interaction of learner control and prior understanding in computer-assisted video instruction. Journal of Educational Psychology, 78, 225-227. Hannafin, M. (1984). Guidelines for using locus of instructional control in the design of computer-assisted instruction. Journal of Instructional Development, 7, 6-10. 139 Lepper, M. R. (1985). Microcomputers in education: Motivational and social issues. American Psychologist, 40, 1-18. Mager, R. F., & Clark, C. (1963). Explorations in student-controlled instruction. Psychological Reports, 13, 7176. Resier, R. A. (1984). Reducing student procrastination in a personalized system of instruction course. Instructional-design Theories and Models: An Overview of Their Current Status, Hillsdale NJ, USA: Lawrence Erlbaum Associates. Van Hiele, P. M. (1986). Structure and insight. A theory of mathematics education, Orlando, FL, USA: Academic Press. 140 Askar, P., Usluel, Y. K. & Mumcu, F. K. (2006). Logistic Regression Modeling for Predicting Task-Related ICT Use in Teaching. Educational Technology & Society, 9 (2), 141-151. Logistic Regression Modeling for Predicting Task-Related ICT Use in Teaching Petek Askar Department of Computer Education and Instructional Technology Faculty of Education, Hacettepe University, 06800, Beytepe, Ankara, Turkey Tel: + 90 312 2977180 Fax: + 90 312 2977176 paskar@hacettepe.edu.tr Yasemin Kocak Usluel Department of Computer Education and Instructional Technology Faculty of Education, Hacettepe University, 06800, Beytepe, Ankara, Turkey Tel: +90 312 2978569 Fax: + 90 312 2977176 kocak@hacettepe.edu.tr Filiz Kuskaya Mumcu Department of Computer Education and Instructional Technology Faculty of Education, Hacettepe University, 06800, Beytepe, Ankara, Turkey Tel: +90 312 4207924 Fax: +90 312 4207274 filiz.kuskaya@tbmm.gov.tr ABSTRACT The main goal of this study is to estimate the extent to which perceived innovation characteristics are associated with the probability of task related ICT use among secondary school teachers. The tasks were categorized as teaching preparation, teaching delivery, and management. Four hundred and sixteen teachers from secondary schools in Turkey, completed a questionnaire, which was designed to determine the taskrelated usage and the perceptions of the teachers in regard to ICT. Logistic stepwise regression analysis showed that complexity or ease of use was found to be a common perceived innovation characteristic for teaching delivery, preparation and managerial tasks in schools. Another result of this survey lead one to conclude that observability is a perceived attribute in teaching delivery in some specific tasks performed during the class period whereas relative advantage and compatibility are for teaching preparation tasks. Keywords Innovation characteristics, Logistic regression, ICT, Secondary schools Introduction The way of the influence on education system exerted by innovations in information and communication technologies (ICT) had previously been confined to training individuals for technology–literacy. This function, has gradually changed, and developed into a new dimension, affecting the learning–teaching processes in a direct way. Indeed, when looked at the National Educational Technology Standards (NETS) of the International Society for Technology Education (ISTE), it is seen that the skills required for teachers were no longer limited to knowing the basic processes and concepts related to technology, but developed into a wider spectrum, comprising the integration of technology into education, and knowing and implementing to ethical principles related to the use of these technologies (ISTE 2004). Apart from these skills that teachers are expected to have, it has also been put forward through researches that teachers, in connection with the use of ICT in classes, have developed their own principles, ideas and judgements, and have influenced their implementation (Galanouli, Murphy, & Gardner, 2004; Cope & Ward, 2002; Mumtaz, 2000). For this reason, it is seen that teachers have important role in ICT use in schools. Furthermore, the fact that educational changes rely largely on the adoption of this change by teachers (Van den Berg, Vandenberghe, & Sleegers, 1999; Fullan, 1991; Hall & Hord 1987), is also indicated in literature study. Diffusion of ICT in Education Use of ICT in schools for the purpose of teaching and learning is a kind of diffusion process in which ICT is an innovation which is defined by Rogers (2003, p.12) as “any idea, practice or object that is perceived as new by ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 141 an individual or other unit of adoption”. In fact, ICT as a relatively new building block in the educational system, causes innovations which ranges from way of communications, and interactions to teaching methods, and materials (Askar & Usluel, 2005). Rogers’ (2003, p. 11) research indicates, “Technological innovations are not always diffused and adopted rapidly, even when the innovation has obvious, proven advantages”. Along with a process for adoption, Rogers provides a theory of how the innovation itself can affect this process. He identified five innovation characteristics that influence the decision to adopt an innovation. The literature on diffusion of innovation highlights the factors that may influence a teacher’s likelihood of utilizing computers for instructional purposes. Dooley and Murphrey (2000) stated that how teachers perceive and react to these technologies is far more important than the technical obstacles in influencing implementation and use. Four main elements in the diffusion of innovation are stated by Rogers (2003) as innovation, communication channels, time, and social system. The characteristics of an innovation, as perceived by the members of a social system, determine its rate of adoption. Five attributes of innovations (Rogers, 2003) are; relative advantage, compatibility, complexity, trialability, and observability. In other words, when an innovation is perceived by users as having greater relative advantage, compatibility, trialability, observability, and less complexity, the innovation will be adopted more rapidly. The first characteristic, relative advantage of an innovation, as perceived by the members of a social system, is positively related to its rate of adoption. Relative advantage is the degree to which an innovation is perceived as better than the idea it supersedes. If the technology provides some type of increased effectiveness or efficiency, then individuals are more likely to adopt the technology (Rogers, 2003). The nature of the innovation largely determines what specific type of relative advantage is important to adopters expressed as economic profitability, social prestige, decrease in discomfort, low initial cost, time savings, savings in effort, and immediacy of rewards, or other benefits. The degree of relative advantage is often expressed in economic profitability and in status giving. This suggests the need to focus on the specific pedagogical advantages of the instructional technology over more conventional teaching tools in the in-service training program (Bennett & Bennett, 2003). So, relative advantage is the degree to which a teacher perceives a new technology as superior to existing substitutes. The second characteristic, compatibility of an innovation, as perceived by the members of a social system, is positively related to its rate of adoption. Compatibility is the degree to which an innovation is perceived as being consistent with the existing values, past experiences, and needs of potential adopters (Rogers, 2003. p. 15). Socio-cultural values and beliefs, previously adopted ideas and the needs determine compatibility or incompatibility. So, compatibility is the degree to which ICT in education is perceived as being consistent with the teachers’ values, past experiences and needs. The third characteristic, complexity of an innovation, as perceived by the members of a social system, is negatively related to its rate of adoption. Complexity is the degree to which an innovation is perceived as difficult to understand and use (Rogers, 2003). Complexity refers to the extent to which an innovation is difficult to understand; the greater the difficulty, the more reluctant potential adopters will be to embrace the change. To ensure that the fear of technical complexity does not present itself as an obstacle, it is important that the content and outcomes of the development and training program be consistent with the knowledge, skills, and abilities of the teacher involved. Complexity often found to be inversely related to the diffusion of an innovation, while simplicity, or ease of use, makes for wider and more rapid acceptance (Serow & Zorowski, 1999) The fourth characteristic, trialability of an innovation, as perceived by members of a social system, is positively related to its rate of adoption. Trialability is the degree to which an innovation may be experienced with on a limited basis. New ideas which go through the experimental process can be adopted faster (Rogers, 2003). The last characteristic, observability in the results of an innovation is positively related to the rate of adoption. Observability is the degree to which the results of innovation are visible to others (Rogers, 2003). If the technology has a high degree of observability, it will be relatively easy for the teacher to learn about it and judge its potential benefits. This, in turn, can increase the likelihood of adoption. Bennett and Bennett (2003) studied the perceived characteristics of instructional technology that may influence a faculty members’ willingness to integrate it in his/her teaching. They have expressed that the most important barrier that teachers face using technology is not lack of technology or funds but teachers’ lack of willingness and their belief that technology is not useful. Butler and Sellbom (2002) examined the factors affecting teachers 142 in adopting new teaching technologies and barriers emerging during adoption. Surveys have been mailed to 410 teachers, about 30% have responded. As a result of the research, trust in technology has been identified as the most important factor in teachers’ decision whether or not to adopt. Know-how, difficulty in learning and time required to learn appear as the second most important factor in adoption. Believing that technology enriches and improves education, difficulty using technology and management support appears as other factors affecting adoption. Another study investigating the rate of adoption with respect to computers in three primary schools was conducted by Askar and Usluel (2003). During the research, the factors such as perceived attributes of computers and some characteristics of schools which have an effect on the rate of adoption were taken into consideration. For this purpose, in the year 2000, 27 teachers and in 2002, 31 teachers from the same schools were interviewed. Changes in the ratio of computer use were identified over two years. Four variables which might be effective in the differences were: relative advantage, observability, factors which are encouraging and hindering the use of computers in the schools. Mumcu (2004) highlighted in her research on the diffusion of ICT in vocational and technical schools that the most critical obstacles were found as insufficient budget, hardware and in-service training. In addition, a positive relationship between relative advantage, compatibility and visibility with the use of ICT were reported. Dooley, Metcalf, and Martinez (1999) explained that public schools are installing computer technology in classrooms at an alarming rate. However, the training for this infusion of technology does not always transfer to integration of the technology into the curriculum. With the introduction of computers in schools, there are significant changes in the school organization and the roles of the teachers, administrators, parents and students. For institutional change, it is imperative that school personnel understand the diffusion process and its implications for success or failure of innovations. In addition to ICT usage, Internet adoption is attracting the researchers. Jebeile and Reeve (2003) reported the findings of a study of teacher adoption of Web technology in a secondary college. The results showed that the innovation adoption variables of relative advantage, compatibility, visibility, ease of use, results demonstrability, and trialability should be considered by school administrators seeking to increase the use of e-learning within their organizations. Braak (2001a) examined the factors influencing the use of computer-mediated communication (CMC) by teachers in secondary schools in Brussels. The survey compared a group of CMC users with non-CMC users. It was demonstrated that language teaching was the best predictor for the use of CMC. The main reason for this is that education policy within the area under investigation has developed a specific CMC project that is primarily oriented towards a target group of language teachers. A second predictor of CMC use was the degree of technological innovativeness. This instrument is a measure of the willingness of the teacher to adopt technological innovation in his own teaching practice. A third predictor was perceived CMC attributes. This instrument indicates the degree to which users observe any congruence between the characteristics of CMC as a medium and their own teaching practice. Martins, Steil, and Todesco (2004) conducted a survey on 92 language schools in Brazil. Results revealed that the Internet is adopted in 55% of the schools analyzed. Both the model of linear multiple regression and the model of logistic regression predicted 77% of the cases of adoption and, therefore, represented satisfactorily the data from the questionnaire used. The variables observability and trialability were found to be the two most significant predictors of adoption. In one of the studies (Braak, 2001b), logistic regression analysis results showed that technological innovativeness, teaching a technology-related subject, and computer experience were more important in explaining the computer use in the classroom than the computer attitude, general innovativeness, age, and gender. In summary, as Surry and Gustafson (1994) concluded, compatibility, complexity and relative advantage can be important considerations when introducing an innovation into instructional settings. In parallel Bussey, Dormody, and VanLeeuwen (2000) stated that the strongest predictor of the level of adoption of technology education was the perception of the teacher of the attributes of technology education. The researchers also concluded that Rogers’ theory of perceived attributes can be a valuable tool for instructional developers working to increase the utilization of their products. ICT in schools in Turkey ICT was first introduced to schools in Turkey in 1984. Since then, Ministry of National Education has allocated considerable amount of budget for the diffusion of computers in the teaching and learning process. For the diffusion of ICT into schools, so many efforts have been undertaken such as in-service training of teachers and administrators, courseware and educational material development and training of computer coordinators. 143 The body of authority is the key policy makers in the Ministry of National Education for all levels of schools in Turkey. However the implementation of any innovation, including ICT is under the responsibility of the schools. In a longitudinal and qualitative study (Askar & Usluel, 2003) held in three primary schools, the change in the situation of teachers related to their use of computers were observed throughout the two years (2000 and 2002) with respect to each school; and it was seen that there were differences, with respect to each school, in the use of computers by teachers. Therefore the adoption rate varies due to schools and teachers. To install a computer lab or to get a trained teacher (computer coordinator) don’t imply that adoption is successful even it is an authority innovation-decision. Indeed, as a result of the analysis of the data obtained through questionnaire from 638 teachers in 27 primary schools with the aim of identifying the purposes of Turkish teachers in using ICT, it was noticed that computers in schools have become widely used in administrative work – such as, preparation of lecture plans and unit plans, organizing scores and reports of students, writing official letters. However, this wide use of computers has not yet been observed in the instructional activities – either using a presentation tool during class or using computers for experiments (Usluel & Askar, 2002). In addition it was observed that instructional usage could be grouped as teaching preparation and delivery (Askar & Usluel, 2003). It is supported by literature that ICT have entered the lives of teachers and are used in administrative tasks at schools, but not in instructional tasks. Studies pointed out that teachers’ use of ICT for instructional purposes is insufficient (Martins, Steil, & Todesco, 2004; Askar & Usluel, 2003; Szabo & Suen, 1998; Proulx & Campbell, 1997). Hence task-related analyses in the framework of innovation characteristics (Rogers, 2003) would be important for developing strategies in the diffusion of ICT in schools. Research Questions This research investigates the adoption of task-related ICT by teachers. The task-related ICT usage was modeled by the four perceived attributes: relative advantage, compatibility, complexity, and observability. The trialability was not considered to be a differentiating attribute to ponder on, since almost all teachers respond to trialability questions affirmatively. Do the four ICT innovation characteristics discriminate the users and non-users on the following tasks? 1. Preparation for a lesson such as developing worksheets or internet search for course content (Teaching preparation) 2. Preparation of lesson plans, unit plans and yearly plans (Teaching preparation) 3. Use of educational multimedia while giving lectures, doing practices reviewing lessons (Teaching delivery) 4. Using a presentation tool during class. (Teaching delivery) 5. Using ICT for experimental study in the laboratories (Teaching delivery) 6. Preparation of examination or organizing student scores (Management) 7. Writing and saving official letters (Management) Methodology This research was based on causal-comparative research design. In causal-comparative research, investigators attempt to determine the cause or consequences of differences that already exist between or among groups of individuals (Fraenkel & Wallen, 2003). The concern of this study was to identify of membership of users and non users given in the research questions according to the perceived attributes. Since the outcome variable was dichotomous the binary logistic regression model was used. Participants The participants of the study were 416 teachers from 8 secondary schools in Ankara, capital of Turkey. There are 8 counties in the provincial capital of Ankara. One county has been chosen among these, and efforts have been exerted to reach all secondary school teachers. There were 710 teachers in the secondary schools of that county; 548 of them have been reached through the survey, 425 of them have replied and 416 questionnaire forms have been evaluated. The demographical information is shown on Table 1. 144 Years in this school Years in teaching Age Gender Table 1. Demographical information of the teachers TOTAL Demographical Information f Female 286 % 68.8 Male 130 31.2 20-29 age 30-39 age 40-49 age 50-59 age 1-5 years 6-10 years 11-15 years 16-20 years 21-25 years 26 years and more 1-5 years 6-10 years 11-15 years 16-20 years 21-25 years 44 170 174 28 42 74 119 82 69 30 191 94 80 48 3 416 10.6 40.9 41.8 6.7 10.1 17.8 28.6 19.7 16.6 7.2 45.9 22.6 19.2 11.5 0.7 100 TOTAL Data Collection and Analysis The questionnaire were developed by the researchers and administered to the teachers one by one. Since the items in the questionnaire were used individually in the data analysis, the total score was meaningless. The questionnaire was divided into three sections. Section 1 consisted of demographical information dealing with gender, age, educational qualifications, years in teaching, and years in this school concerning the respondents. Section 2 included tasks eliciting the views of the teachers toward the use of ICT in teaching preparation, teaching delivery, and management. Section 3 consisted of the views of the teachers regarding the innovation characteristics toward the use of ICT on the tasks. An example was given for section 2 and for section 3; Section 2 “Do you use educational multimedia while giving lectures, doing practices or reviewing lessons?” The responses were taken as ‘yes’ or ‘no’. Section 3 “Rate your opinion whether the following task is observable or not.” Use of educational multimedia while giving lectures, doing practices reviewing lessons. The responses were taken as ‘not observable’, ‘undecided’, and ‘observable’. “Rate your opinion whether the following task is complex or not.” Preparation of lesson plans, unit plans and yearly plans The responses were taken as ‘not complex’, ‘undecided’, and ‘complex’ “Rate your opinion whether the following task is compatible with your job.” Using a presentation tool during class. The responses were taken as ‘not compatible’, ‘undecided’, and ‘compatible’ Since the outcome variable is dichotomous (binary) the binary logistic regression model was used. It was defined as Y=0 (non-use), or Y=1 (use). X denotes the vector of independent variables or predictors. The independent variables are perceived attributes of innovation; relative advantage, compatibility, complexity and observability. The binary logistic regression is stated in terms of the probability that Y=1 given X: 145 P (Y = 1 | X ) = 1 1 + Exp(− βX ) Ζ = βX is called the linear predictor and stands for β0 + β1X1 +……….+ βpXp . By solving this equation Y, the form for the binary logistic regression model is obtained: ln P (Y = 1 | X ) = logit (Y ) = β X (Košmely & Vadnal, 2003). P (Y = 0 | X ) The recommended sample size for this kind of research is calculated by taking into consideration the minimum ratio (sample size/ predictor variables) of 10 to 1, with a minimum sample size of 100 or 50, plus a variable number that is a function of the number of predictors (Peng, Lee & Ingersoll, 2002). It was satisfied in this study. Findings Task 1: Preparation for a lesson such as developing worksheets or internet search for course content (Teaching preparation) The binary logistic stepwise regression was conducted of four innovation characteristics: relative advantage, compatibility, complexity and observability on the dependent variable task 1 (use or not use). Out of 407, 175 teachers coded as users and 232 non-users. Variables in the equation are complexity and relative advantage (for complexity β=-1.014, SE=0.141, Wald’s χ 2=51.401, p=0.000 and for relative advantage β=0.789, SE=0.397, Wald’s χ 2= 3.942, p=0.047). The equation for task 1 is Z = (-0.492) + (-1.014) * Complexity + 0.789 * Relative Advantage. The Nagelkerke R2 was reported for each predictor (R2 =0.208, R2(complexity)=0.195). Figure 1 shows the probability of a teacher using ICT while preparing a lesson as a function of complexity and relative advantage. 0.8 Estimated Probability 0.7 0.6 0.5 0.4 Relative Advantage 0.3 Yes 0.2 0.1 Undecided Not 0 Not Complex Undecided Complex Complexity Figure 1. The likelihood of a teacher using ICT for task 1 by perceived complexity and relative advantage Complexity and relative advantage are predictors that distinguish between the teachers who do and who do not benefit from ICT in preparing instructional activities. The highest probability of usage is the category of teachers who found computers for this task advantageous and who found it easy to use (P=0.7028). The second highest probability is the category of teachers who couldn’t decide whether it is advantageous or not, but found it easy to use (P=0.5180). The lowest probability is the group who thought that it had no advantage to him (or her) and it was complex to use. 146 Task 2: Preparation of lesson plans, unit plans and yearly plans (Teaching preparation) The binary logistic stepwise regression analysis for this task showed that two variables were significant for discriminating the users and non-users. Out of 406, 147 were non-users, 259 were users. Variables in the equation were complexity and compatibility (for complexity β=-1.376, SE=0.177, Wald’s χ 2=60.674, p=0.000 and for compatibility β=1.330, SE=0.364, Wald’s χ 2=13.368, p=0.000). Estimated Probability The equation for task 2 is Z = (-1.305) + (-1.376) * Complexity + 1.330 * Compatibility (R2=0.354, R2 (complexity) =0.307). Figure 2 shows the probability of a teacher using ICT while preparing of lesson plans, unit plans and yearly plans experiments in the laboratories as a function of complexity and compatibility. 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Not Compatible Not Complex Undecided Undecided Compatibility Compatible Complex Complexity Figure 2. The likelihood of a teacher using ICT for task 2 by perceived complexity and compatibility Preparing lesson plans, which is one of the teaching preparation tasks, indicates the highest usage probability. In this task, as well, compatibility and ease of use come to the foreground. The highest probability of usage is the categories of teachers who found computers for this task compatible and who found it easy to use (P=0.7873). The second highest is the teachers who had no idea for compatibility, but found it easy to use (P=0.4947). Estimated Probability Task 3: Use of educational multimedia during lecturing, practices and reviewing lessons (Teaching delivery) 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 Observability Observable Undecided Not Observable Not Complex Undecided Complex Complexity Figure 3. The likelihood of a teacher using ICT for task 3 by perceived complexity and observability The binary logistic stepwise regression analysis for this task showed that two variables were significant for discriminating the users and non-users. Out of 406, 296 were non-users, 110 were users. Variables in the 147 equation were complexity and observability (for complexity β=-0.871, SE=0.186, Wald’s χ 2=21.851, p=0.000 and for observability β=0.449, SE=0.155, Wald’s χ 2= 8.412, p=0.004). The equation for task 3 is Z = (-0.680) + (-0.871) * Complexity + 0.449 * Observability (R2 =0.173, R2 (complexity) =0.145). Figure 3 shows the probability of a teacher using ICT for educational multimedia during lecturing, practices and reviewing lessons as a function of complexity and observability. The probability of using multimedia during lessons has been found lower than the probability of using computers for preparation of instructional activities. This result indicates that diffusion of using ICT for preparation of lesson is more rapid than using ICT during lesson. The highest probability of usage is the category of teachers who found computers non complex for this task and who observed its usage (P=0.4491). However still the probability is lower than 0.50. While ICT usage during lesson is related to the teachers’ skills in one respect, it may also be related to the infrastructure, technology, and physical features of classrooms. Task 4: Using a presentation tool during class (Teaching delivery) For this task out of 407, 323 were non-users, 84 were users. The variables in the equation were complexity and compatibility (for complexity β=-1.447, SE=0.314, Wald’s χ 2=21.300, p=0.000 and for compatibility β=4.565, SE=0.570, Wald’s χ 2=7.533, p=0.006). The equation for task 4 is Z = (-3.777) + (-1.447) * Complexity + 1.565 * Compatibility (R2=0.291, R2 (complexity) =0.248). Figure 4 shows the probability of a teacher using a presentation tool during class as a function of complexity and compatibility. 0.4 0.35 Compatible 0.3 Estimated Probability 0.25 0.2 0.15 0.1 0.05 Compatibility Undecided Not Compatible 0 Not Complex Undecided Complex Complexity Figure 4. The likelihood of a teacher using ICT for task 4 by perceived complexity and compatibility The two innovation characteristics which distinguish between the teachers who use presentation program during lesson and who do not are ease of use and compatibility. The group with highest probability is the one who find it easy to use and compatible (P=0.3707). What draws attention here is presence of compatibility characteristic together with the ease of use. As is known, one indication of compatibility is the habits and how compatible they are to the way of life. The reason why the probability of use of presentation program by teachers in classes is high is that they likely perceive the presentation program as the extension of the white-board and overhead projector. Task 5: Using computers for experiments in the laboratories (Teaching delivery) The binary logistic stepwise regression analysis showed that complexity and observability were significant for discriminating the users and non-users (for complexity β=-1.614, SE=0.284, Wald’s χ 2=32.360, p=0.000 and for observability β=0.602, SE=0.170, Wald’s χ 2=12.562, p=0.000). Out of 407, 313 were non-users, 94 were users. 148 The equation for task 5 is Z = (-0.257) + (-1.614) * Complexity + 0.602 * Observability (R2=0.308, R2 (complexity) =0.266). Figure 5 shows the probability of a teacher using ICT for experiments in the laboratories as a function of complexity and observability. 0.6 Observable Estimated Probability 0.5 0.4 0.3 0.2 0.1 Observability Undecided Not Observable 0 Not Complex Undecided Complex Complexity Figure 5. The likelihood of a teacher using ICT for task 5 by perceived complexity and observability The highest probability of usage is the categories of teachers who found computers non complex and who observed its usage (P=0.4837). In this task, observability arises as a discriminating variable with ease of use. The highest probability for this task is category of teachers who find it observable and ease of use. This finding indicates that teachers should be given chances to observe the ICT integration done by their associates in the classroom. Task 6- Preparation of examination or organizing student scores (Management) and Task 7- Writing and saving official letters (Management) The binary logistic stepwise regression analysis for task 6 and 7 showed that complexity is the only discriminating variable. For task 6, out of 407, 128 were non-users, 279 were users (for complexity β=-1.416, SE=0.183, Wald’s χ 2=59.658, p=0.000). For task 7, out of 407, 164 were non-users, 243 were users (for complexity β=-1.416, SE=0.183, Wald’s χ 2=59.825, p=0.000). While the probability of ICT usage by teachers who find it easy are 0.7835 (task 6), and 0.7164 (task 7), the probability of ICT usage by them who don’t find it easy are 0.1756 (task 6) and 0.1107 (task 7). The equation for task 6 is Z = 2.702 + (-1.416) * Complexity (R2 =0.245). The equation for task 7 is Z = 2.432 + (-1.505) * Complexity (R2 =0.264). Conclusion & Discussion The role that perceptions play in the adoption of an innovation has been examined by different researchers (Martins, Steil, & Todesco, 2004; Bennett & Bennett, 2003; Jebeile & Reeve, 2003; Braak, 2001; Bussey, Dormody, & VanLeeuwen, 2000; Serow & Zorowski, 1999; Surry & Gustafson, 1994). These researchers also confirm the importance of the study of perceived attributes and rate of adoption. As is seen, complexity comes out as the predictor variable in all tasks. While complexity and relative advantage comes forth in preparation for lesson activities, being predictor of compatibility together with complexity in presentation during lesson and preparation of lesson plans, and also coming forth of observability together with complexity in the use of multimedia during lesson and experiments in the lab, prove that perceived attributes could vary according to tasks. Complexity or ease of use was found to be a common perceived innovation characteristic for teaching delivery, preparation and managerial tasks in schools. As a predictor variable, complexity (ease of use) explains the user/non user between 15% and 31% according to the tasks. Complexity as defined before is the degree to which 149 the technology is difficult to understand or use. The strategies for decreasing the perceived complexity are highly important for diffusion of ICT in schools. The design of staff development and the ongoing and immediate technical and educational support could be key considerations. The barriers teachers stated in several studies are likely to be related to the perceived complexity. Another result of this survey lead one to conclude that “observability” is a perceived attribute in teaching delivery in some specific tasks performed during the class period. Besides professional development of teachers, the classroom activities should be open for all teachers in the school. Teachers should be given chances to observe the ICT integration done by their associates in the classroom. Therefore, it is easy to learn about it and judge its potential benefits. The importance of observability indicates how critical it is that the instructional technology be demonstrated in the in-service training program. Ideas that can easily be observed and communicated to others will be adopted more quickly than ideas that are more difficult to see and communicate. It is recommended that the best practices in ICT integration in schools should be shared by the teachers with active participation are more useful than the traditional in-service teacher training programs. A program that focuses on the interactive instructional properties of the technology would be of greater interest to this teacher than one that failed to discuss how the technology is consistent with his/her teaching philosophy. Unfortunately, in many circumstances, the introduction of instructional technology will require the rejection of one set of values and ideas about education and the adoption of a new set with regards to what constitutes effective pedagogy (Bennett & Bennett, 2003). It can be argued that installing ICT for schools solely as product doesn’t carry much meaning. For this reason besides of the nationwide solutions, school based solutions will be more realistic in the diffusion of computers for instructional purposes. One of the results of this research is that, complexity comes out as the predictor variable in all tasks, and that other variables vary according to the tasks. School administration should focus their efforts on decreasing complexity. Technical support, on going teacher professional development, early familiarity with the ICT, sharing best practices as well as barriers and difficulties in real teaching-learning environments could be some strategies in the diffusion process. On the other hand, the importance of school’s implementations according to a phased plan by way of determining its priorities with respect to ICT, and shaping it according to the task, cannot be ignored. Further detailed studies on the ICT diffusion in schools will illuminate this subject. References Askar, P., & Usluel, Y. K. (2003). A longitudinal study related to the rate of adoption of computers: Comparison of three schools. Hacettepe University Journal of Education, 24, 15-25. Askar, P., & Usluel, Y. K. (2005). The diffusion of computers in the schools: A comparison of three cases on the basis of complex systems. In Berg, G. A. (Eds.), Encyclopedia of International Computer-based Learning: IDEA accepted for publication. Bennett, J., & Bennett, L. (2003). A review of factors that influence the diffusion of innovation when structuring a faculty training program. Internet and Higher Education, 6, 53-63. Braak, J. Van (2001a). Factors influencing the use of computer mediated communication by teachers in secondary schools. Computers & Education, 36, 41-57. Braak, J. Van (2001b). Individual characteristics influencing teachers class use of computers. Journal of Educational Computing Research, 25 (2), 141-157. Bussey, J. M., Dormody, T. J. & VanLeeuwen, D. (2000). Some factors predicting the adoption of technology education in New Mexico public schools. Journal of Technology Education, 12 (1), retrieved May 1, 2006 from http://scholar.lib.vt.edu/ejournals/JTE/v12n1/bussey.html. Butler, D. L., & Sellbom, M. (2002). Barriers to adopting technology. Educause Quarterly, 2, 22-28. Cope, C., & Ward, P. (2002). Integrating learning technology into classrooms: The importance of teachers’ perceptions. Educational Technology & Society, 5 (1), 67-70. 150 Dooley, L. M., Metcalf, T., & Martinez, A. (1999). A study of the adoption of computer technology by teachers. Educational Technology & Society, 2 (4), 107-115. Dooley, K. E. & Murphrey, T. P. (2000). How the perspectives of administrators, faculty, and support units impact the rate of distance education adoption. Online Journal of Distance Learning Administration, 3 (4), retrieved May 1, 2006 from http://www.westga.edu/~distance/ojdla/winter34/dooley34.html. Fraenkel, J. R., & Wallen, N. E. ( 2003). How to design and Evaluate Research in Education (5th Ed.), New York: McGraw-Hill. Fullan, M. G. (1991). The new meaning of educational change (2nd Ed.), New York, USA: Teachers College Press. Galanouli, D., Murphy, C., & Gardner, J. (2004). Teachers’ perceptions of the effectiveness of ICT-competence training. Computers & Education, 43, 63-79. Hall, G. E., & Hord, S. M. (1987). Change in Schools: Facilitating the Process, USA: Suny Press. ISTE (2004). The National Educational Technology Standards, retrieved November 30, 2005 from http://www.iste.org/standards/. Jebeile, S., & Reeve, R. (2003). The diffusion of e-learning innovations in an Australian secondary college: Strategies and tactics for educational leaders. The Innovation Journal, 8 (4), retrieved May 1, 2006 from http://www.innovation.cc/peer-reviewed/jebeile-reeve-elearning.pdf. Košmely, K., & Vadnal, K. (2003). Comparison of two generalized logistic regression models: A case study. Paper presented at the 25th International Conference on Information Technology Interfaces (ITI) 2003, June 1619, 2003, Cavtat, Croatia. Martins, C. B. M. J., Steil, A. V., & Todesco, J. L. (2004). Factors influencing the adoption of the internet as a teaching tool at foreign language schools. Computers & Education, 42, 353-374. Mumcu, F. K. (2004). Diffusion of information and communication technologies in vocational and technical schools, MS Thesis, Hacettepe University, Turkey. Mumtaz, S. (2000). Factors affecting teachers’ use of Information and Communications Technology: A review of the literature. Journal of Information Technology for Teacher Education, 9 (3), 319-333. Peng, C. J., Lee, K. L., & Ingersoll, G. M. (2002). An introduction to logistic regression analysis and reporting. The Journal of Educational Research, 96 (1). Proulx, M., & Campbell, B. (1997). The professional practices of faculty and the diffusion of computer Technologies in university teaching. Electronic Journal of Sociology, retreived November 30, 2005 from http://www.sociology.org/content/vol002.003/proulx.html. Rogers, M. E. (2003). Diffusion of innovations (5th Ed.), New York, USA: The Free Press. Serow, R. C., & Zorowski, C. F. (1999). Diffusion of instructional innovations in engineering education. Paper presented at ICEE’99, Czech Republic, Paper No. 258. Surry, D. W., & Gustafson, K. L. (1994). The role of perceptions in the adoption of computer-based learning, Education Resources Information Service (ERIC), ED 374 788. Szabo, M., & Suen, C. (1998). A study of the impact of a school district computer technology program on adoption of educational technolog, retreived November 30, 2005 from http://www.quasar.ualberta.ca/edmedia/Suenszabo.html. Usluel, Y. K., & Askar, P. (2002). Teachers and use of computers. The Proceedings of BTIE -5, Informatics Association Of Turkey Ankara-Turkey, 169-180. Van Den Berg, R., Vandenberghe, R., & Sleegers, P. (1999). Management of innovations from a cultural individual perspective. School Effectiveness and School Improvement, 10 (3), 321-351. 151 Kiraz, E. & Ozdemir, D. (2006). The Relationship between Educational Ideologies and Technology Acceptance in Preservice Teachers. Educational Technology & Society, 9 (2), 152-165. The Relationship between Educational Ideologies and Technology Acceptance in Pre-service Teachers Ercan Kiraz Middle East Technical University, Faculty of Education, Department of Educational Sciences, Ankara, Turkey Tel: +90-312-210 40 37 Fax: +90 312 210 12 54 ekiraz@metu.edu.tr Devrim Ozdemir Virginia Polytechnic Institute and State University, 220 War Memorial Hall 24061 Blacksburg, VA, USA Tel: +1-540-239-37 61 dozdemir@vt.edu ABSTRACT After the evaluation of numerous technology integration programs in school districts and universities, it is recognized that the existence of technology does not guarantee its utilization in the classroom environment. Although many models and theories have tried to explain the contributing factors in technology acceptance, most of the models and theories have focused on technology-related factors. This study focused on educational ideology, a factor not related to technology that also affects decisions in terms of educational applications. Based on the literature review, we hypothesized a new model of technology acceptance which includes educational ideology as an external factor. We attempted to create a model that was compatible with our hypothesized model by collecting data from surveys completed by 320 pre-service teachers. Structural Equation Modeling was employed to create the path analytic model. The variables used in the path analytic model were the components of the original Technology Acceptance Model and six different educational ideologies. The results showed that the new model was consistent with the hypothesized model. Therefore, the results illustrate that different educational ideologies may have different effects on teachers’ technology acceptance. Keywords Educational Ideologies; Technology Acceptance; Technology Adoption, Teacher Education Introduction With the swift advent of technology in previous decades, Information and Communication Technologies (ICT) have pervaded the workplace and fostered modern corporations along with providing governments with a proficient infrastructure. Besides these dramatic changes in many aspects of society, education remained by and large a traditional craft (Perkins 1992, p.3).. According to Strommen (1992), “technological changes that affected society left educational systems largely unchanged” (as cited in Semple, 2000, p. 21). Since the users are imperative and have played a major role in the utilization of technology, factors affecting their technology use became an important concern for researchers. This is because merely the existence of technology in the classroom does not guarantee the utilization of that technology. Teachers are less likely to integrate technology into their instruction unless they accept the notion of the requirement of educational technology use in their classroom environment. (Stetheimer and Cleveland, 1998; Weiss, 1994). The central questions with regard to technology acceptance are how individuals perceive technology and which factors contribute to the lack of utilization (Rogers, 1995; Surry, 2000). Technology Acceptance It is commonly accepted that today’s teachers may benefit from educational technology to a great extent. Therefore, failure in the usage or acceptance of technology is an important issue. At this point, it is necessary to define the term ‘technology acceptance’ to determine the factors affecting the actual use of educational technology in the classroom environment. Davis, Bagozzi and Warshaw (1989) defined significant factors affecting technology acceptance in their Technology Acceptance Model (TAM). The TAM stems from behavioral theory, and is a well-known model that has undergone significant developments since its conception. A substantial number of factors have been added to TAM as possible significant determinants of technology use. However, only some of the factors remained as major determinants of technology use. In order to understand the rationale in the TAM model, it is necessary to mention the supporting theories. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 152 One of these behavioral theories is the Social Cognitive Theory (SCT) (Bandura, 1986). The SCT outlines three reciprocal factors, the person, behavior, and the environment and explains the relationship among these factors. In this reciprocity, personal and environmental factors affect behavior and vice versa. SCT mentions two different channels, outcome expectation and self-efficacy, between the two factors, the person and behavior. According to this theory, people ask two questions before behaving, ‘Can I do it?’ and ‘Should I do it?’ If they believe that they are able to perform the behavior and also if they will attain good outcomes afterwards, they are in close proximity to perform. Although the SCT model showed how the two main factors, outcome expectation and self-efficacy, affected behavior, attitudinal factors were left unmentioned. In 1975, Fishbein and Ajzen produced the Theory of Reasoned Action (TRA) by focusing on attitudinal factors on behavior. TRA posited behavioral intention as a measure of one’s intention to perform a specific behavior and represented the primary predictor of actual behavior (as cited in Brosnan, 1999). Therefore, according to TRA, self-efficacy and outcome expectation does not directly affect the actual behavior; however, it affects behavioral intention. In 1988, the TRA model was modified and the theory was changed to the Theory of Planned Behavior (TPB). According to the TPB, behavior is influenced solely by behavioral intention and behavioral intention in turn is influenced by attitudes toward behavior, by subjective norms and by perceived behavioral control (Bajaj and Nidumolu, 1998). Finally, in 1989, Davis, Bagozzi and Warshaw defined a model, named the “Technology Acceptance Model” (TAM) which covers some significant factors affecting the use of technology (Davis et al., 1989). Similar to the TRA, the TAM explained how one primary factor, the behavioral intention to use, affected actual system use. Although the behavioral intention to use is the only factor related to actual system use, recent studies confirmed that this is only a mediating factor - a hypothetical factor and not used in most of the studies. Corresponding to the TPB, the TAM explains attitudinal factors affecting the behavioral intention to use focusing on the factor, attitudes towards the use of technology. In addition, similar to the SCT, the TAM demonstrates the effects of self-efficacy and outcome expectation on attitudes toward use of technology. The TAM defined self-efficacy and outcome expectation factors as the perceived ease of use of technology and perceived usefulness of technology which are related to self-efficacy and outcome expectation. In TAM, all other factors related to technology acceptance are called external variables. In summary, technology acceptance is related to four main factors: the perceived ease of use of technology; the perceived usefulness of technology; the attitudes toward the use of technology; and the frequency of use of technology. For instance, if people perceive that technology is easy to use and useful in their field, this builds positive attitudes toward the use of that technology. Positive attitudes toward the use of technology have a positive relationship with the behavioral intention to use the technology. In this sense, four main factors of the TAM, shown in Figure 1, have been used as major determinants of technology acceptance in this study. Percieved Usefullness External Variables Attitude Towards Behavioural Intention to use Actual System Use Percieved Ease of Use Figure 1. Original Technology Acceptance Model (Source: Legris et. al., 2003) Thus far, many external variables have been tested in the model and few studies have explained more than 40% of the variance in technology use (Legris, Ingham & Collerette 2003). In addition, Legris, Ingham and Collerette stated that “TAM is a useful model, but has to be integrated into a broader model which would include variables related to both human and social change processes, and to the adoption of the innovation model” (p.191). In accordance with this claim, the main question of this study was “which factors influence the decision directly or indirectly”. This study suggests that educational ideology may affect one’s decisions directly or indirectly, since every individual has a belief system. 153 Educational Ideologies Educational ideologies are rooted in political ideologies and affect decisions related to education. In 1981, William F. O’Neill grouped educational ideologies into six different groups under two main categories. O’Neill categorized these ideologies as conservative and liberal educational ideologies with three subgroups for each category. The conservative educational ideologies are educational fundamentalism, educational intellectualism, and educational conservatism. The liberal educational ideologies are educational liberalism, educational liberationism, and educational anarchism. O’Neill (1990) mentioned that educational ideologies have an impact on individuals’ beliefs with regard to the overall goals of education, the objectives of the school, the child as a learner, administration and control, the nature of the curriculum, instructional methods and evaluation and classroom control. However, O’Neill did not mention the effects of educational ideologies on the acceptance of educational technology. At this point, it would be beneficial to delineate what educational ideologies are. The information below is summarized from “Educational Ideologies: Contemporary Expressions of Educational Philosophy” (O’Neill, 1990). In educational fundamentalism, knowledge is a tool for reconstructing society in pursuit of a predetermined pattern of moral excellence where man is a moral agent. The approach is tacit anti-intellectualism and is opposed to the critical examination of preferred patterns of belief and behavior. Education is considered as moral regeneration and the ideology focuses on the original purposes of the existing social traditions and institutions, placing emphasis on a return to the past as a corrective reorientation. In educational intellectualism, knowledge is viewed as an end to itself and truth has an intrinsic value, where man is man. That is, man’s universal nature transcends specific circumstances. The approach is traditional intellectualism (stressing reason and speculative wisdom). Education is an orientation to life in general. It focuses on the intellectual history of a man, generally identified with the dominant Western intellectual tradition of classicism. In educational conservatism, knowledge is for social utility and a means of realizing existing social values. Man is a citizen, who finds his highest fulfillment as an effective member of the established social order. This approach is based on reasoned conformity and reliance on the best answers of the past as the most trustworthy guide to present action. Education is considered as socialization to the established system. The ideology focuses on existing social traditions and institutions and places emphasizes on the present situation, viewed in a relatively shallow historical perspective, that is, conventionalism. In educational liberalism, knowledge is a necessary tool used in practical problem solving. The individual is a unique personality, who finds his greatest satisfaction in self-expression in response to changing conditions. The approach is effective thinking (practical intelligence), and the ability to solve personal problems effectively. Education is the development of personal effectiveness. In educational liberationism, knowledge is a necessary tool for required social reforms. Man is a product of culture, who finds his highest fulfillment along the lines defined and controlled by the existing social system. The approach is based on the objective (rational- scientific) analysis and evaluation of existing social policies and practices. Education is the fullest realization of each person’s unique potentialities as a distinctive human being. This ideology focuses on social conditions that block the fullest realization of individual potentialities, and emphasizes on the future (that is, on changes in the present system required to bring about a more humanistic and humanizing society). The purpose of individual is to bring about immediate large scale changes within the existing society. The ideology stresses the significant changes that affect the basic nature and conduct of the established social system In educational anarchism, knowledge is a natural by-product of daily living. Individual personality is a value that transcends the requirements of any particular society. The approach is based on free choice and self determination in a sane and humanistic social setting. Education is considered as a natural function of everyday living in a rational and productive social environment and the ideology focuses on the development of an “educational society” that either eliminates or radically minimizes the necessity for formal schools and other such institutional constraints on personal behavior, and emphasizes on a post-historical future in which people function as self-regulating moral beings. The purpose of the ideology is the continuous change and self-renewal within a constantly emerging society; stresses the need for minimizing and / or eliminating institutional restraints on a personal behavior (deinstitutionalization). 154 It is clear that each ideology focuses on different definitions of knowledge and of the individual, and each ideology has different approaches to reach different goals related to education. In this sense, comparing the differences in the general characteristics of educational ideologies, perceptions and attitudes regarding educational technologies may change depending on ideology. While some teachers may think that educational technology is crucial to reach the goal of education, some others may think that educational technologies are not as useful to reach the goals, as the goal of technology may oppose the ideology of the individual. The aforementioned shed light on the effect of factors such as the teachers’ perceptions, attitudes, and beliefs on the use of educational technology. Therefore, using educational ideologies and attitudes, beliefs, and perceptions as meaningful categories, this study investigated whether educational ideologies of pre-service teachers have an effect on technology acceptance. Method Based on the literature, this study hypothesized a model including educational ideologies as an external variable of the Technology Acceptance Model. The methodology section of this study outlines the sample, the instrument, the data analysis stages, and presents the stages necessary to obtain a model compatible with the hypothesized model Sample Since this study is related with technology acceptance and educational ideologies, whether participants have a well formed educational philosophy is an important issue. Participants of the study were graduates of Teachers’ High Schools and are currently attending the Faculty of Education in the Middle East Technical University (METU), Ankara, Turkey. They were selected through nation wide examinations and are training to be teachers. During both their high school and university educations, they have taken courses relating to teaching and learning, such as “Introduction to Educational Sciences”, “Introduction to the Teaching Profession”, “Instructional Planning and Evaluation”, “Development and Learning”, “Introduction to Computer Education”, “Instructional Materials Development”, “Classroom Management”, and so forth. They have had exposure to subjects such as philosophy, educational philosophy, terrains of philosophy, schools of philosophy, and the relationship between philosophy and educational practices. They have also had work experience during their undergraduate practicum period, a requirement for all teacher candidates set by the Faculty of Education. Therefore, it can be assumed that these students have satisfactory theoretical and practical backgrounds regarding the formation of a philosophical context. The sample consisted of 320 students. In terms of their departments, the majority of these students were attending the Department of Elementary Education (50.3%) followed by the Department of Foreign Language Education (38.4%). Furthermore, most of the students (43.4%) who responded to the survey were freshmen, 14.1% of the students were sophomore, 22.2% of the students were junior, and 17.2% of the students were at the senior level. The majority of the sample was female (66.9%). Finally, 50.3% of the students were between the ages of 15 and 20, while 41.3% of the students were between the ages of 21 and 25. Instruments The data collection instrument consisted of five sections − demographics, educational ideologies, perceived ease of use, perceived usefulness, attitudes toward computer use and the frequency of use. The second section was adapted from O’Neill (1990) in order to identify the educational ideologies. Perceived ease of use and perceived usefulness sections were adapted from Legris et al (2003). The Attitudes toward computer use section was adapted from a questionnaire developed by Dusick & Yildirim (2000). A committee approach was used in the adaptation process of the instrument. The items were translated by eight different instructors from the Department of Foreign Language Education at METU. An expert familiar with both American and Turkish cultures was responsible for the appropriate translation while subject matter experts adapted the translations. A pilot study was conducted to correct some misunderstandings. The researchers also used a pilot study to check the validity of the items in the instrument. The educational ideology part consisted of 104 questions that measured the two general and six specific educational ideologies. 14 questions were used to identify each of the six specific educational ideologies and 10 155 questions for each general educational ideology. Given that this study investigated the effect of specific educational ideologies, a total of 84 questions related to specific educational ideologies appeared in the instrument. The perceived ease of use and perceived usefulness sections included 10 questions on each topic. The attitudes toward educational technologies section included 23 questions. Finally, the actual use section had only one question. With the exception of the question related to actual use, all of the questions utilized a fivepoint Likert-type scale with responses ranging from Strongly Agree to Strongly Disagree. The question in the actual use section utilized a six-point Likert-type scale ranging from “more than once a day” to “less than once a month”. Data Analysis The data analysis of the study was conducted in three sections: item analysis, factor analysis, and path analysis. Item Analysis: For the item analysis, the statistical program SPSS 10.0 (Statistical Package for Social Sciences) was used. Considering the dispersion of the data in each item and the corrected total item-scale correlations, the items were analyzed and the valid and reliable items were selected for the factor analysis stage of the study. Factor Analysis: Factor analysis was a necessary step to confirm the assumption of the structural equation modeling, that all of the variables in the hypothesized model are independent from each other. Therefore, the results of the factor analysis will prove that the items selected for the path analysis are discriminatory for the factors and that all these variables are independent factors in the model. This part of the analysis includes two subsections: exploratory and confirmatory factor analysis. According to the principal component analysis results, 21 out of 84 questions in the educational ideologies section, 10 out of 10 questions in the perceived usefulness of technology section, 7 out of 10 questions in the perceived ease of use section, 9 out of 23 questions in the attitudes toward use section, and the 1 question in the actual use section were determined to be useable for this study. The details of the exploratory and confirmatory factor analyses will be presented in the next section. The study first analyzed the dimensionality of the 84 items selected from the questionnaire. The purpose in this stage was to find out which questions statistically represent the six different educational ideologies. Since each participant in the sample may have different ideas stemming from the different ideologies as mentioned above, the educational ideology questions do not divide the sample into groups clearly. Moreover, there are some commonalities among the educational ideology questions. For exploratory factor analysis, principal component analysis was used to determine the discriminatory educational ideology questions. The items determined to be theoretically and statistically separate from the other factors were evaluated using confirmatory factor analysis. After the confirmatory factor analysis, 21 educational ideology questions were chosen for further analysis. Six factors representing educational ideologies were retained for further analysis. The validity values of the factors (eigenvalues) were 3.66, 2.45, 1.44, 1.38, 1.14, and 1.11. Table 1 presents items grouped as a result of principal component analysis, with their respective factor loadings. Six factors explained 53% of the total variance in this particular analysis. The first factor represents educational fundamentalism, the second educational conservatism, the third educational liberationism, the fourth educational liberalism, the fifth educational intellectualism, and the sixth educational anarchism. The numbers in bold text indicate significant factor loading of each question. Educational Ideologies Section Considering the factor structure as indicated in Table 1, we formed latent variables and educational ideologies, for the path analytic model. In this process, two important criteria were used. Firstly, we attempted to keep the number of observed variables to a minimum of three (Schumacher & Lomax, 1996); secondly, we gave greater preference to the items with greater factor loadings. The first factor in the table above represents the educational fundamentalism ideology and includes questions 90, 101, 97, 87, and 62. The second factor, educational conservatism includes questions 4, 72, 46, 12, and 83. The third factor, educational liberationism includes questions 84, 99, and 69. The fourth factor, educational liberalism includes questions 77, and 53. The fifth factor, educational intellectualism includes questions 2, 23, and 14. The sixth factor, educational anarchism includes Questions 6, 45, and 39. These factors were used as latent factors for path analysis. 156 Table 1: Factor Loadings of Educational Ideologies Items for Principal Component Factor Analysis Factor Loading Items 1 2 3 4 5 90. The school should encourage a return to the simple and straightforward virtues of an earlier day, to the older and better ways. 101. The schools should emphasize the virtues of the historical past as a way of correcting the existing overemphasis on the present and the future. 97. The individual finds his greatest fulfillment in a voluntary subordination to the ends of the State. 87. A central purpose of education should be to revive and reaffirm an almost religious commitment to certain profound national goals. 62. The history of this nation is preeminently a spiritual history guided by Providence. 4. In the final analysis, human happiness derives from adapting oneself to prevailing standards of belief and behavior. 72. Students should be trained to be good citizens in terms of prevailing cultural views about the nature of good citizenship and proper conduct. 46. The basic value of knowledge is its contemporary social utility; knowledge is primarily a means of adapting successfully within the existing social order. 12. The school should encourage an appreciation for time – tested cultural institutions, traditions, and processes. 83. Schools should be run in a manner consistent with the conventional wisdom (the common sense beliefs) of society at large. 84. The schools should stress the critical analysis and evaluation of prevailing social beliefs and behavior. 99. The schools should encourage students to recognize and respond to the need for particular kinds of liberalizing social reforms. 69. The teacher should be a model of intellectual commitment and social involvement. 77. The teacher should be basically an organizer and expediter of learning activities and experiences. 53. Knowledge is ultimately a tool, a means to be used in solving the problems of everyday living. 14. Secondary education should provide the student with an orientation to life in general, emphasizing his role as a human being rather than training him for any particular social role or position. 23. The schools should place their basic emphasis on man as man; that is, on the sort of abiding human nature which all individuals share. 2. The most valuable type of knowledge is that which involves symbolism and abstract thinking. 6. Individual differences (physical, psychological, and social) are so significant that they dictate against the wisdom of prescribing the same or similar educational experiences for all people. 45. Problems associated with student control and discipline are frequently caused by a society which blocks the development of personal responsibility by over controlling everyone, including students. 39. Conventional teaching ordinarily subverts the child’s capacity for self – learning. -.192 .781 .758 .662 .274 .568 .236 .132 .492 .135 -.361 .245 .189 -.147 -.401 .186 -.138 .256 .734 -.125 .128 .719 .160 .134 .655 .246 .176 .276 .540 .125 .164 .215 .447 -.318 .235 .734 .139 -.124 .486 .102 .139 .158 .175 .156 .180 .625 .273 .119 6 .284 .118 .787 .176 -.132 .714 .206 .610 .253 -.140 .594 .132 .244 -.121 .148 .180 .130 -.165 .211 .592 -.286 -.130 .699 .332 .598 .535 Note: Loadings below 0.10 were suppressed in the table. 157 Technology Acceptance Section This section of the questionnaire consisted of 4 different subsections: the perceived usefulness of technology, perceived ease of use, attitudes toward use, and actual use sections. Using exploratory factor analysis, it was ensured that each subsection included only one factor. Therefore, three different exploratory factor analyses were conducted for the perceived ease of use, perceived usefulness, and attitudes toward use sections at this stage. Since the actual use section included one question, exploratory and confirmatory factor analysis was not conducted for this section. Ten questions in the perceived usefulness of technology section, seven questions in the perceived ease of use section, nine questions in the attitudes toward use section, and one question in the actual use section were determined to be used for this study. Table 2: Principal Component Analysis Results for Technology Acceptance Factor Loading Items Perceived Usefulness 8. Computers enhance my educational effectiveness. 6. Computers increase my performance in education. 5. Computers increase my productivity in education. 10. Overall, I found computers to be useful in education. 3. Computers enable me to accomplish tasks more quickly in education. 9. Computers make my work easier in education. 2. Computers give me greater control over my work in education. 1. Computers improve the quality of the work I do in education. 4. Computers support the critical aspects of my work in education. 7. Computers allow me to accomplish more work than otherwise be possible in education. Perceived Ease of Use 8. My interaction with the computers are clear and understandable. 3 Interacting with the computers is often frustrating. 2. Learning to operate computer applications is easy for me. 4. I find it easy to get the computers to do what I want to do. 10. Overall, I find computers easy to use. 6. It is easy for me to remember how to perform tasks using the computers. 5. The computers are rigid and inflexible to interact with. Attitudes toward use 9. I feel anxiety when I am using computers. 4. I have self-confidence in using computers. 7. I feel terrible while my friends are talking about computers. 5. It is hard to learn new computer applications. 3. I am not the one who can work with computers. 2. It is easy to learn how to use computer software. 10. I fell confused while I am working with computers. 20. The use of computers in classrooms are useful and it is worth of endeavor. 18. I am also using computers out of school. 1 .845 .824 .810 .801 .777 .774 .767 .721 .672 .648 .712 .685 .673 .661 .649 .633 .488 .792 .741 .725 .717 .688 .663 .593 .555 .507 Note: Loadings below 0.10 were suppressed in the table. Table 2 presents items grouped as a result of the principal component analysis, with their respective factor loadings. Each variable has one factor in this analysis. The perceived usefulness section explained 59%, the perceived ease of use section explained 42% and the attitudes toward use section explained 45% of the total variance in this particular analysis. The questions in the Table 2 represent the three factors of technology acceptance. Table 3 indicates the Lambda-x estimates and standard errors as obtained for each of the observed variables from the confirmatory factor analysis, with their abbreviations, the names of the latent variables, response modes, and respective item means. Lambda-x values, which are the loadings of each observed variable on the respective latent variable, indicate reasonable sizes to support the idea of using these latent variables in the proposed path analytic model to explain significant factors in educational technology acceptance. After the confirmatory factor analysis of each questionnaire, 21 questions in the educational ideologies section, five questions in the perceived usefulness section, four questions in the perceived ease of use and attitudes toward use section and one question for the actual use section were selected for the path analysis. Considering the factor structure as indicated in Tables 1 and 2, we have formed latent variables for the path analytic model. In 158 this process, the two important criteria were used again. First, the number of observed variables was kept to a minimum of three (Schumacher & Lomax, 1996); second, preference was accorded to the items with greater factor loadings. Table 3: LISREL Estimates, Standard Errors for Confirmatory Factor Analysis and Item Means with Response Modes Latent and Observed Variables Educational Fundamentalism Question 62 Question 87 Question 90 Question 97 Question 101 Educational Intellectualism Question 2 Question 14 Question 23 Educational Conservatism Question 4 Question 12 Question 46 Question 72 Question 83 Educational Liberalism Question 53 Question 77 Educational Liberationism Question 69 Question 84 Question 99 Educational Anarchism Question 6 Question 39 Question 45 Perceived Usefulness of Technology Question 2 Question 4 Question 7 Question 8 Question 9 Perceived Ease of Use Question 2 Question 6 Question 8 Question 10 Attitudes toward Use Question 2 Question 4 Question 5 Question 16 Lambda-x SE Mean 0.48 0.44 0.82 0.58 0.68 0.06 0.06 0.05 0.06 0.05 2.97 3.18 2.54 2.63 2.36 0.44 0.47 0.54 0.07 0.07 0.07 2.96 4.02 3.86 0.60 0.57 0.59 0.76 0.49 0.06 0.06 0.06 0.05 0.06 3.33 3.57 4.05 3.76 3.02 0.69 0.65 0.07 0.07 3.87 3.87 0.53 0.68 0.63 0.06 0.06 0.06 4.05 3.87 3.94 0.26 0.43 0.66 0.07 0.07 0.09 3.46 3.81 3.93 0.73 0.65 0.74 0.94 0.84 0.05 0.05 0.05 0.04 0.05 4.03 3.87 3.89 4.11 4.23 0.61 0.62 0.74 0.63 0.05 0.05 0.05 0.05 3.25 3.41 3.56 3.44 0.69 0.81 0.70 0.62 0.05 0.05 0.05 0.05 2.87 3.41 3.43* 4.05 Response Mode 1 (Strongly Disagree) to 5 (Strongly Agree) * This item was reversed for the analysis. Path Analysis: In the study, LISREL 8.30 for Windows (Joreskog & Sorbom, 1999) with SIMPLIS command language was used to analyze data for path analysis with latent variables. The maximum likelihood estimation method was used in all of the LISREL analyses. For the model data fit assessment, Standardized Root Mean Squared Residual (SRMR), and Root-Mean-Square Error of Approximation (RMSEA) were used in the study. In this study, we determined a 0.073 Standardized Root Mean Square (SRMR) and a 0.080 Root Mean Square (RMSEA) index. These indices were deemed adequate to treat the respective item groups as distinct latent variables in the path analytic model. The alpha reliability coefficients for the latent variables were 70 for fundamentalism, 40 for intellectualism, 69 for conservatism, 52 for liberalism, 56 for liberationism, 37 for anarchism, 83 for perceived usefulness of technology, 69 for perceived ease of use of technology and 76 for 159 attitudes toward use. These reliability coefficients are significant to indicate that the questions under each category are reliable. Based on the evidence from the literature, it would be possible to claim that educational ideologies might affect educators’ perceptions on the usefulness of technology and their attitudes toward technology directly. Therefore, they might affect technology use indirectly. This study has focused on educational ideologies as possible external factors of the TAM. In this study, educational ideologies and perceived ease of use are exogenous variables, the use of educational technology is endogenous variables and finally, perceived usefulness and attitudes toward use are both exogenous and endogenous variables. Edu. Fund. Edu. Intel. Perc. Useful. Edu. Cons. Edu. Lib. Edu. Libt. Attitudes Use Edu. Anar. Perc. E. U. Figure 2: Hypothesized Model Results In order to predict the most compatible model, t-test results for both exogenous and endogenous variables and model data fit indexes (such as SRMR and RMSEA) were taken into consideration. The paths that indicated nonsignificant t-values were deleted from the model. In accordance with the LISREL analysis, some minor modifications were made to the model to obtain the most compatible model. Finally, the model in Figure 3 was obtained with 0.073 SRMR and 0.080 RMSEA fit index values. These values were deemed adequate to interpret the significant relationship among the latent variables. Table 4 shows Lambda-x estimates, t-values and standard errors for the educational technology acceptance model. The Lambda-x estimates and t-values reflect the reliability of the items in the final model. None of the t-values are below 0.30 which means that all these questions could be kept for the best-fit model. Table 4: LISREL Estimates, t-values, and Standard Errors for LISREL Model Latent Variables EDUCATIONAL FUNDAMENTALISM EDUCATIONAL INTELLECTUALISM EDUCATIONAL CONSERVATISM Observed Variables Question 62 Question 87 Question 90 Question 97 Question 101 Question 2 Question 14 Question 23 Question 4 Question 12 Question 46 Question 72 L Lambda-x 0.49 0.45 0.82 0.58 0.68 0.36 0.42 0.58 0.59 0.57 0.55 0.78 t 8.40 7.88 15.51 10.23 12.34 5.10 6.09 8.07 10.55 10.14 10.21 14.95 SE 0.06 0.06 0.05 0.06 0.05 0.07 0.07 0.07 0.06 0.06 0.05 0.05 160 EDUCATIONAL LIBERALISM EDUCATIONAL LIBERATIONISM EDUCATIONAL ANARCHISM PERCEIVED USEFULNESS OF TECHNOLOGY PERCEIVED EASE OF USE ATTITUDES TOWARD USE Question 83 Question 53 Question 77 Question 69 Question 84 Question 99 Question 6 Question 39 Question 45 Question 2 Question 4 Question 7 Question 8 Question 9 Question 2 Question 6 Question 8 Question 10 Question 2 Question 4 Question 5 Question 16 0.49 0.63 0.59 0.22 0.72 0.66 0.26 0.40 0.66 0.75 0.66 0.77 0.93 0.84 0.59 0.63 0.75 0.60 0.71 0.84 0.69 0.33 8.39 10.21 9.56 8.56 11.89 11.07 3.71 5.56 7.77 11.54 10.42 11.81 13.58 12.64 10.66 11.56 14.53 10.94 5.06 5.11 5.03 4.18 0.06 0.06 0.06 0.07 0.06 0.06 0.07 0.07 0.08 0.06 0.06 0.06 0.07 0.07 0.05 0.05 0.05 0.05 0.14 0.16 0.14 0.08 Table 5 presents the Beta estimates, which are the coefficients among attitudes toward computers, perceived usefulness, and frequency of use. The table also presents the Gamma estimates, which are the coefficients among the endogenous and exogenous variables and t-values. Table 5: LISREL Estimates and t-values for LISREL Model Latent Variables Attitudes toward use & Perceived Usefulness of Technology Perceived Usefulness & The Frequency of Use Perceived Ease of Use Educational Fundamentalism & Attitudes toward Technology Use Educational Conservatism Educational Liberalism Educational Anarchism & Perceived Usefulness of Technology Educational Liberationism Educational Intellectualism Beta 0.43 0.25 - Gamma 0.96 -0.15 0.26 -0.32 -0.33 0.49 0.38 t 4.01 4.24 4.28 -1.95 1.97 -2.22 -2.49 3.73 4.33 Table 5 and Figure 2 both indicate the structural model of technology acceptance developed. In this model, the standardized path coefficients varied between –0.33 and 0.96 in the fitted model. Cohen (as cited in Kline, 1998) made some suggestions about the interpretations of the absolute magnitudes of path coefficients. Cohen (1988) explained that standardized path coefficients that have absolute values less than 0.10 might indicate a “small” effect; whereas values around 0.30 indicate a “medium,” and values above 0.50 indicate a “large” effect, respectively (Kline, 1998). In accordance with these suggestions, the path coefficient from attitudes towards use to the perceived ease of use could be considered as a high effect in the model. All the other path coefficients indicated medium effect sizes in the model. The model developed in this study has some similarities as well as differences in regard to the TAM. Similar to the hypothesized model, the fitted model in figure 3 shows that there is a direct effect of perceived usefulness on the actual use, and there is also a direct affect of perceived ease of use on attitudes. However, unlike the TAM, the fitted model shows that attitudes towards use affect perceived usefulness, and there is no significant effect of attitude towards actual use. Also, the fitted model shows that there is no significant relationship between perceived ease of use and perceived usefulness. PEU: Perceived Usefulness of Technology, ATT: Attitudes toward Technology Use, PU: Perceived Usefulness U: The Frequency of Use F: Educational Fundamentalism, I: Educational Intellectualism, C: Educational Conservatism, L: Educational Liberalism, LB: Educational Liberationism, A: Educational Anarchism. 161 Figure 3 shows that there is a strong positive relationship between pre-service teachers’ perceived ease of use of educational technology and their attitudes toward use. Therefore, pre-service teachers who are competent with technology in a classroom environment have highly positive attitudes towards the use of educational technologies. In addition, fundamentalist and liberalist pre-service teachers have low values for attitudes towards the use of educational technologies. While educational fundamentalists and educational liberalists demonstrate negative linear relationships in relation to attitudes towards use, pre-service teachers who embrace conservative educational ideologies demonstrate high attitude values towards the use of educational technologies. Moreover, pre-service teachers who believe in educational anarchism suppose that educational technologies are not useful in the classroom environment. However, educational liberationists and intellectualists believe that educational technologies are very useful in the classroom environment. Figure 3: Structural Model of Educational Technology Acceptance Model Integrated with Educational Ideologies Arguments over the integration of technology in schools have met with many obstacles. Most decision makers or administrators want to benefit from technology in educational practices. For this reason, teacher education institutions as well as training programs have placed a strong emphasis on technology education courses in both pre-service and in-service education. However, the literature shows that self-efficacy of the teachers, in other words the perceived ease of use, is not sufficient to allow teachers to use technology in education. This study shows that the perceived ease of use may not directly influence the frequency of use of computer technologies in education. This result might give some clues about the ineffectiveness of computer literacy courses and technology education courses in pre-service and in-service education and training. Based on the results of this study, we assert that there is a medium effect of perceived usefulness of technology on the frequency of use of technology in education. This result is consistent with the suggestions of Moore et al (1999) who suggested that teachers’ technology competencies consist of four major categories: prerequisite technical skills, technical skills, instructional uses, and professional roles (Moore et al, 1999). They explained that teachers react negatively to courses that emphasize technical skills without practice. For this reason, instructional technology use must be related with teaching practices in actual professional implementation. In this sense, we can say that prospective teachers consider educational technologies to be more useful in real educational settings, which also promotes the use of technology in education effectively. Besides the effect of the perceived ease of technology use on the frequency of use, perceived ease of use positively affected the attitudes toward use. According to our model, attitudes towards use are the only 162 determinant of perceived usefulness. The pre-service teachers who perceived the use of technology in education to be crucial probably have high positive attitudes towards use. Although the perceived ease of use is the primary determinant of attitude, three educational ideologies, fundamentalism, conservatism, and liberalism, also have significant effects on attitudes towards use. For example, educational fundamentalist ideology has a negative affect on attitudes towards the use of technology. This result might be stemming from the characteristics of educational fundamentalism, since it tends to implement uncritical acceptance and urges individuals to conform to the existing social order. However, computer technologies, especially the Internet with access to various different sources, provide opportunities for increased critical thinking. From a broader vision, students might get a chance to question common sense by accessing different sources, and consequently, this might lessen the authority of the teacher in the classroom environment. In this aspect, fundamentalist pre-service teachers might see technology as possible threat to their existing routines. Conservatism has a positive affect on attitudes towards use. According to the model, the educational conservative view appears more moderate to change when compared with the fundamentalist view. As Skolnik (1998) stated, “It has been pointed out that since computers first appeared, predictions have been made about how they would revolutionize education, but that has not happened” (p. 644). Based on the literature, it could be appropriate to say that technology is a tool for changing society. Nevertheless, belief, understanding and acceptance of technology illustrate a complex pattern in relation to the implementation of educational technology. Therefore, there may be a consistency between attitudes towards the use of technology and the conservative educational view. As opposed to conservatism, the educational liberalist ideology has a negative affect on the attitude towards use. In fact, liberalist pre-service teachers do use technology, especially computer technology, for a significant amount of time and become accustomed to technology as a part of their college life. Interestingly, even though they use technology, they may not be conscious concerning the need of technology in their prospective job settings. In 2001, Steel and Hudson conducted a study regarding academics’ perceptions of educational technologies and found that some of the academics thought the use of educational technology was a potential threat to meaningful face-to-face interaction. Whether liberalists assume technology as potential threat or not should be studied in-depth. All of the ideologies mentioned above have effects on attitudes toward use. Moreover, educational anarchism, educational liberationism, and educational intellectualism have an effect on the perceived usefulness of technology. Educational Anarchism has a negative effect on the perceived usefulness of technology. Some applications of computer technology intend to reach a wide population to provide them more a systemized education via synchronous or asynchronous technologies. This is contradictory with what anarchists deem. Because, preservice teachers who might think of educational technology as a mediator of a centralized and institutionalized society might perceive educational technology as not useful since some of the technology integration programs create centralized and institutionalized society, such as the use of learning management systems and learning content management systems. There is a positive relationship between educational liberationism and the perceived usefulness of technology. It is an expected result since the literature supports the consistency between the goal of education from an educational liberationist perspective and the general perceived role of technology in education. The educational liberationist belief emphasizes the fullest development of each person’s unique potentialities as a human being. Steel and Hudson (2001) also support this notion by underlining the introduction of technology into the learning process as a potential quality-enhancing act. Educational Intellectualism has positively affected the perceived usefulness of technology. Since educational intellectualism seeks to change existing educational practices in order to make them conform more perfectly to some established and essentially unvarying intellectual or spiritual ideal, technology in education may be seen as a tool to change existing educational practices to be more consistent with those purposes. Discussion This study has indicated that attitudes towards use and the perceived usefulness of technology differentiate depending on the educational ideologies of pre-service teachers. These attitudes and perceptions about the 163 usefulness of technology directly or indirectly affect the frequency of use of technology. Technology education or computer literacy courses in pre-service and in-service teacher education is not enough to increase the frequency of use of educational technologies if they focus only on technical skills, rather than including the instructional use and emphasizing the professional roles of teachers in the environment integrated with technology (Inan, F., Yildirim, S. & Kiraz, E. 2004; Yildirim, S. & Kiraz, E. 1999). According to a report published by UNESCO (2002), there is continuum of approaches through which schools and education systems proceed in their adoption of ICT. In the first stage of technology adoption, namely the applying approach, teachers and school administrators start using ICT for daily tasks and they adopt curriculum to create more space for ICT in teaching. In the infusing approach, ICT is integrated throughout the school curriculum. School personnel start using more advanced and customized software to increase their productivity as well as their professional competencies. Finally, in schools that demonstrate the transforming approach, ICT becomes an internal part of the curriculum and it is used as a catalyst for school reform. Teachers take more initiative to move from teacher-centered teaching approach to student-centered learning activities, and schools serve as learning centers for the surrounding communities (p. 15-16). It should be noticed that technology integration in schools largely depends on individuals, especially teachers, since they are obviously gatekeepers for all kind of innovation introduced to the education system. Thus, it is crucial to scrutinize teacher’s ICT utilization and reveal factors that contribute to their skeptical practices of teaching with technology. Unquestionably, this study revealed the fact that educational ideologies cultivate the type and extent of teacher’s technology use and thus it has a paramount effect on a successful integration of technology throughout the system. There could be two different approaches according to the findings of this study. Firstly, educational ideology is a system of belief that has established through a long period of time and it is hard to change. If the truth is that educational technology is useful for every educational practice, its benefits should be emphasized for different individuals rather than imposing specific educational technologies. Therefore, stakeholders and teacher educators should try to familiarize educators by considering their various educational beliefs. Secondly, this study shows that the use of educational technology and the benefits to education is an arguable matter since there is a contradiction between the overall goals of some educational ideologies and the factors of technology acceptance. Two questions arise from the findings of this study: is the acceptance of educational technology necessary for every single person? Or can we achieve success in education without using educational technology in every situation? The answer is complex, as none of the ideologies is superior to the others. Each ideology is effective in its own scope and has a possible explanation for accepting or rejecting technology. Therefore, some pre-service teachers may not consider the use of educational technology necessary in some situations since their educational goals might be attainable without using the means of technology. References Bajaj, A, & Nidumolu, S. R. (1998). Techniques A feedback model to understand information system usage. Information & Management, 33 (4), 213-224. Bandura, A. (1986) Social Foundations of Thought and Action: A Social Cognitive Theory, Englewood Cliffs: NJ: Prentice Hall. Brosnan, M. J. (1999). Modeling Technophobia: a case for word processing. Computers in Human Behavior, 15, 105-121. Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Management Science, 35 (8), 982-1003. Dusick, D., & Yıldırım, S. (2000). Faculty Computer Use and Training: Identifying Distinct Needs for Different Populations. Community College Review, 27 (4), 33-47. Inan, F., Yildirim, S., & Kiraz, E. (2004) The Design and Development of an Online Learning Support System for Preservice Teachers: A Discussion of Attitudes and Utilization. Journal of Interactive Instruction Development, 17 (1), 3388-3393. Kline, R. B. (1998). Principles and practice of structural equation modeling, New York: Guilford. 164 Joreskog, K. G., & Sorbom, D. (1996). LISREL 8 user's reference guide, Chicago, Illinois, USA: Scientific Software International. Legris, P., Ingham, J., & Collerette, P. (2003). Why do people use information technology? A critical review of the technology acceptance model. Information & Management, 40, 191-204. Moore, J., Knuth, R., Borse, J., & Mitchell, M. (1999). Concepts and Procedures. Paper presented at the 10th International Conference of Society for Information Technology and Teacher Education, February 28 - March 4, 1999, San Antonio, Texas, USA. O'Neill, W. F. (1990). Educational Ideologies Contemporary Expressions of Educational Philosophy, Dubuque, Iowa: Kendall / Hunt Publishing Company (Original work published 1981). Perkins, D. (1992). Smart schools: Better thinking and learning for every child, New York: The Free Press. Rogers, E. M. (1995). Diffusion of Innovations (4th Ed.), New York: Free Press. Schumacher, R. E., &. Lomax, R. G. (1996). A beginner's guide to SEM, Manwah, New Jersey, USA: Lawrence Erlbaum Associates. Semple, A. (2000). Learning theories and their influence on the development and use of educational technologies. Australian Science Teachers Journal, 46 (3), 21-28. Skolnik, M. L. (1998). Higher Education in the 21st Century. Futures, 30 (7), 635-650. Steel, J., & Hudson, A. (2001). Educational Technology in Learning and Teaching: The Perceptions and Experiences of Teaching Staff. Innovations in Education and Teaching International, 38, 103-111. Stettheimer, T. D., & Cleveland, A. D. (1998, November). Modeling Utilization of Planned Information Technology. Paper presented at The 1998 American Medical Informatics Association Annual Symposium, November 7-11, 1998, Walt Disney World Resort, USA. Surry, D. W., & Land, S. M. (2000). Starategies for motivating higher education faculty to use technology. Innovations in Education and Training International, 37 (2), 145-153. UNESCO (2002). Information and Communication Technology in Education: A curriculum for schools and programme of teacher development, Paris: UNESCO. Weiss, J. (1994). Keeping up with the research. Technology & Learning, 14 (5), 30. Yildirim S., & Kiraz, E. (1999). Obstacles in Integrating Online Communication Tools into Preservice Teacher Education: A Case Study. Journal of Computing in Teacher Education, 15 (3), 23-28. 165 Serce, F. C. & Yildirim. S. (2006). A Web-Based Synchronous Collaborative Review Tool: A Case Study of an On-line Graduate Course Educational Technology & Society, 9 (2), 166-177. A Web-Based Synchronous Collaborative Review Tool: A Case Study of an On-line Graduate Course Fatma Cemile Serce Information Systems, Informatics Institute, Middle East Technical University, 06531 Ankara, Turkey Tel: +90 312 210 3747 Fax: +90 312 210 3745 cemile@metu.edu.tr Soner Yildirim Department of Computer Education and Instructional Technology Middle East Technical University, Faculty of Education, 06531 Ankara, Turkey Tel: +90 312 210 4057 Fax: +90 312 210 1112 soner@metu.edu.tr Abstract On-line collaboration is an instructional method that facilitates collaboration in an on-line learning setting. To promote effective collaboration, it is vital to reveal both the student’s and the instructor’s point of view pertaining to effective on-line collaboration. In this study, the effectiveness of a learning management system in on-line collaboration was first investigated in a graduate course offered through the means of distance learning. In this first phase of this study, the nature of collaboration, the students perceptions of the effectiveness of the tool in on-line collaboration, the factors contributing to effective peer interaction among students and the role of the instructor as perceived by the students in the on-line course were explored. This phase of the study also involved the reviews of other learning management systems, course management systems and groupware systems regarding the tools used to encourage collaboration. Also revealed was a lack of diversity in collaboration tools. Based on these preliminary findings, an on-line document reviewing tool was developed and pilot tested in the second phase of the study. In this phase, a web-based synchronous collaborative review tool called WebSCoRe is proposed to promote online collaboration. WebSCoRe is proposed as an attempt to develop and implement a new platform for on-line document reviewing, to promote effective on-line collaboration among students and the instructor. Keywords Collaborative learning, Distance learning, Learning technologies, On-line review tools Introduction The advances in technology and changes in the organizational infrastructure put an increased emphasis on teamwork within the workforce. Employees need to be able to think creatively, solve problems, and make decisions as a team. Therefore, the development and enhancement of collaborative skills and producing graduates who are flexible and have market-related skills and abilities has been among the primary goals of higher education. On-line education provides collaboration opportunities both to the educational institution and to learners. Several studies have shown that the implementation of collaborative learning strategies result in higher student involvement in the course and more engagement in the learning process and collaborative learning methods are more effective than traditional methods in promoting student learning and achievement (Hiltz, 1998). Gokhale (1995) stated that if the purpose of instruction is to enhance critical- thinking, problem-solving and collaborative skills, then the utilization of collaborative learning techniques will be more beneficial. The learning management systems and groupware systems are the main platforms for distance education (Horton & Horton, 2003). It is essential to understand the factors affecting on-line collaboration in these platforms. In this study, several widely used learning management and groupware systems were first reviewed as tools provided to promote collaboration. Moreover, in order to obtain the users’ point of view on on-line collaboration, an in-depth assessment study was conducted on a learning management system, called NET-Class. NET-Class has been developed at the Informatics Institute at Middle East Technical University. It has been used for delivering on-line graduate programs and some general education courses given at METU. It is also been used as a tool to support face-to-face instruction. As a tool used for promoting computer-mediated communication, the first part of the study examined the effectiveness of NET-Class on on-line collaboration for project based ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 166 learning in a graduate course offered through the means of distance learning. The student’s perceptions of their experiences in on-line collaboration in the course studied were obtained. Moreover, the nature of collaboration in the course was identified from the postings in the forum with respect to the issues of interaction, participation, feedback and the utterances for on-line collaboration. Findings indicated that NET-Class failed in providing diverse collaboration tools and platforms. Thus, a webbased synchronous collaborative review tool named WebSCoRe was developed and implemented. WebScoRe is a real-time collaborative meeting-based document review system that allows all comers to participate regardless of location. An object-oriented approach was used in the design and development of the tool and Java was chosen as the development language to ensure compatibility with the NET-Class Learning Management System. Collaborative Learning Learning is never a passive act. The key to the learning process is the interactions among students themselves, the interactions between the faculty and the students, and the collaboration that results from these interactions. Learning by collaborating is a social process and leads to learning being not only active, but also interactive. The requirements for education in the twenty-first century result in more emphasis on active participation rather than passive approaches presenting learning (Hiltz, 1998) Collaboration is a synchronous activity of a gathering of parties with diverse skills and backgrounds, contributing those skills and resources in an atmosphere of trust, respect and flexibility, in order to achieve shared goals and objectives. Collaboration is something that human beings have been experiencing from early times and applying throughout their lives (Muronaga & Harada, 1999). Collaborative learning is an instructional method that encourages students to work together in groups toward a common goal. (Bruner, 1991) The students learn or attempt to learn something together and they are responsible for one another’s learning as well as their own. Thus, the success of one student helps other students to be successful. Gokhale(1995) examined the effectiveness of individual learning versus collaborative learning in enhancing drill-and-practice skills and critical-thinking skills. Regarding gaining factual knowledge, both individual learning and collaborative learning were found to be equally effective. The study reveals that collaborative learning fosters the development of critical thinking through discussion, clarification of ideas and evaluation of others’ ideas. Therefore, the study concludes that if the purpose of instruction is to enhance critical-thinking and problem-solving skills, then collaborative learning will be more beneficial. The need for collaborative skills is commonly emphasized in information systems. Layzell, Brereton, and French (2000) argue that software engineering is no longer the preserve of individuals but is essentially a team-based activity involving a wide variety of stakeholders and thus making the need for communication and co-operation an inherent characteristic. Because of the changes in support technology, economic factors and globalization of the software process, the personnel are separated. This results in the need for effective communication and collaboration skills of globally distributed personnel. However, many of the barriers to the successful implementation of team-based collaborative software engineering practices trace directly to the poor understanding of and inadequate training for the interaction skills software professionals need to successfully enact such collaborative activities, such as requirements elicitation, project management and peer review (Schoeder and Brunner, 1996). The course assessed in this study was an information systems project course, which requires software engineering practices. Promoting Collaborative Learning Dillenbourg and Schneider (1995) stated some conditions for effective collaborative learning, such as group composition, task features and communication media. Group composition includes the age and levels of the participants. The size of the groups and the difference between group members are the variables affecting the group composition. Task features stress the relation between the nature of the task and the effectiveness of collaboration. There are some tasks that may not be shared, requiring individual work; some tasks are straightforward without the potential to lead to any disagreement or misunderstanding. The communication media is another essential condition for effective collaborative learning. According to the task in hand and the group members that have been selected, the collaborative process may not work, due to the inadequacy of the medium used for communication. 167 Dillenbourg (1999) argues that collaborative learning is not one single mechanism. If one talks about “learning from collaboration”, one should also talk about “learning from being alone”. Individual cognitive systems do not learn because they are individual, but because they perform some activities (reading, building, predicting), which trigger learning. Similarly, peers do not learn because they are many, but because they perform some activities, which trigger specific learning mechanisms. Additionally, the author argues that the interaction among students generates extra activities (explanation, disagreement, and mutual regulation), triggering knowledge elicitation, and internalization, and reduces cognitive load. The field of collaborative learning is precisely about these activities and mechanism. In this study, collaborative reviewing was chosen as the activity to promote collaboration. Collaborative reviewing involves negotiation of the meaning of facts and demands for consensus for an appropriate solution. Working together on a common document allows collaborators to share and discuss their ideas about further revisions (Kim, 2002). Collaborative reviewing is one of the instructional strategies used to promote collaboration in both face-to-face and on-line settings. Collaborative reviewing provides participants to be engaged in rich interactions, which is the necessity of an effective collaborative learning (Dillenbourg, 1999). Pratt and Pallof (2001) argue that the elements of on-line groups are the individuals, the group, the facilitator (instructor) and the technology. In collaborative learning environments, the role of the instructor and the role of the students are important. The Role of the Instructor Teaching at a distance involves the use of a different set of skills for instructors, compared to those used in a traditional classroom. There is a consensus in the literature that getting people to participate and making learning active at a distance is much more important than presenting the information. According to Moore and Kearsley (1996), the most critical skills that the distance educator must develop are; making the students active participants, inducing inter-learner interaction among the participants, keeping the discussion on track and providing immediate feedback Pallof and Pratt (1999) summarize the role of the on-line instructor in four general areas: Pedagogical, Social, Managerial and Technical. The pedagogical function involves providing guidance and a framework as a “container” for the course and allowing students to explore the course material, as well as related materials. The role of instructor becomes that of an educational facilitator. The social function of the instructor is to facilitate and make room for the personal and social aspects of on-line community. The managerial function consists of objective setting, rule making, decision making, posting a syllabus for the course, including assignments and some other initial guidelines for the group discussions. The technical function is the provision of technical facilitation by the on-line instructor, to the students. This function depends on the instructor being confident concerning the use of technology and making the students comfortable by providing guidance. The Role of the Learner Pallof and Pratt (1999) describe the responsibilities of the learner in distance learning environments. The authors categorized the role of learner into three areas: Knowledge Generation, Collaboration and Process Management. Knowledge Generation involves roles such as active solution seeking, viewing problem from number of perspectives and questioning the assumptions presented by the instructor and others. Collaboration consists of working together to generate deeper levels of understanding and critical evaluation of material under study, sharing resources, giving each other feedback etc. Learners have roles in process management such as being active, participating within minimal guidelines, interacting and engaging with one another and taking the responsibility for formation of on-line learning communities. Collaborative Learning and Technology Technology is accepted as a facilitator in applications of collaboration. The Internet is a natural enabler of collaboration in the context of on-line learning. On-line collaboration can be broadly defined as the cooperation of individuals engaged in a common task, using electronic technologies. 168 In the current literature, many benefits including motivating learners, increasing participation in online courses and empowering learners are directly associated with the use of effective online collaboration strategies. (Kubala, 1998; Allen, 2002). On the other hand, many researchers acknowledge that online collaboration is not an easy task but rather it remains a challenging task for all who teach online. For example Horton (2000) stressed that good design is a must for successful online collaboration, otherwise some learners may not be able to benefit fully. Especially students with high anxiety regarding communication with others may hesitate to participate in online discussions. Another challenge pertaining to online collaboration is that compared with face-to-face interaction; students may demand more interaction with the instructor. As a result, instructor should dedicate more time for discussion sessions. Finally, online communication lacks some of the features of face-toface communication including gestures, tone of voice and body language. On the other hand, most of these drawbacks result from poorly designed or implemented applications, thus they can be overcome by proper design and use of appropriate technologies. Most collaboration tools can be divided into three categories, being asynchronous collaboration, synchronous collaboration and integrated collaboration tools. Many businesses employ some type of asynchronous collaboration technology. These basic collaboration tools include e-mail, bulletin boards and intranets. They allow people to interact in a more friendly way, but disconnections remain. Information sharing cannot occur in real-time. Synchronous solutions represent the next level of collaboration, enabling real time communication. Virtual meetings using text chat tools, audio/video conferencing, and shared whiteboards can provide immediate connections between users. However, security can be a concern. In integrated collaboration, virtual co-location products employ leading edge collaboration technology to create permanent virtual workplaces. These environments allow people to work together in one secure place, wherever they are located, through rich communications tools like text chat, audio/video conferencing presence awareness and document sharing. Edutech (2004) has developed an interactive platform to compare web-based learning environments according to 108 criteria. The results of Edutech (2004) study shows that WebCT, BlackBoard and LearningSpace are among the most powerful widely-used tools. In this study, these three platforms were analyzed in detail by focusing on the collaboration tools. It is found that chat, whiteboard and forum tools are commonly used for providing interaction and collaboration among the on-line participants. Apart from the learning and course management systems, the study also involves the reviews of the groupware systems. In the study, 96 groupware systems were found, the descriptions of each system were collected by means of their web sites and each was reviewed according to the tools that they provide for collaboration and communication. Table 1 summarizes the results of the analysis of groupware systems. Apart from these, various web-based review tools were examined. wbART, a web-based asynchronous review tool, was designed to supporting technical asynchronous review (Yanbaş & Demirörs, 2003). The other review tool examined is iMarkup. This tool also facilitates asynchronous web-based collaboration on web content. Table 1: The results of analysis of groupware systems Tools for Collaboration Forum Textual Chat File Sharing Audio Communication Screen Sharing Integrated E-Mail Instant Messages Polling Group Calendar Video Communication % 49 47 30 29 24 23 22 22 21 21 Tools for Collaboration Whiteboard Workspace Awareness Application Sharing Floor Sharing Version Control Collaborative Browsing Virtual Hand Raising Voice Chat Collaborative Viewing Synchronization of Content % 16 16 15 12 7 6 6 5 3 2 Moreover, MS Word also provides some tools for collaboration and review of the documents. Although asynchronous tools have powerful features for document review, synchronous communication is more appropriate for collaborative activities, because collaboration is synchronous by nature. This study proposes a review tool which promotes collaboration by making use of synchronous communication. 169 An Assessment of On-line Collaboration: A case study of an on-line graduate course To investigate the effectiveness of the learning management system in on-line collaboration, an exploratory case study was conducted based upon descriptive data. Berg (2004) describes case studies as a method which “. . . involves systematically gathering enough information about a particular person, social setting, event, or group to permit the researcher to effectively understand how the subject operates or functions.” (p.251). Further, exploratory case studies can be considered as a preliminary phase of a large study. Thus, for the first phase of this study, the exploratory case study method was employed. The participants of the study included 16 students enrolled in the course offered in a graduate program at the Informatics Institute at Middle East Technical University during spring 2002, as well as the instructor and the teaching assistant. The course was offered through a learning management system, called NET-Class as a case that grounds the whole research process. All the activities involving collaborative tasks in the course were analyzed. Accordingly, the collaborative groups and peers were determined. The collaborative groups were whole class, the discussion groups formed in the forum environment and peers working together to develop the project required in the course. A combination of quantitative and qualitative research methods were employed in the study: a questionnaire survey, observation, interviews and document analysis. The qualitative analysis method was applied to the transcripts of the interviews, the outcomes of the observation reports and the outcomes of the document analysis of messages posted on the forum. The coding schema developed by Curtis and Lawson (2001), was used to describe the utterances in on-line collaboration in discussion forums. The coding scheme consists of 15 behaviors and 5 behavior categories as shown in Table 2. The messages posted in the discussion groups were also analyzed according to 3 important issues, participation, interaction types and feedback. Table 2: Coding Scheme Behavior categories Planning Contributing Seeking Input Reflection / Monitoring Social Interaction Behaviors Group skills Organizing work Initiating activities Providing help Giving feedback. Exchanging resources and information Sharing knowledge Challenging others Explaining or elaborating Seeking help Seeking feedback Advocating effort Monitoring group effort Reflecting on medium Social interaction Findings Although the study is based on a one-semester observation rather than longitudinal observation, it contributes to understanding of the effectiveness of NET-Class on-line collaboration, the factors of effective on-line collaboration and the role of the instructor in an on-line collaborative learning environment. The findings provide information concerning collaboration in class, the factors affecting on-line collaboration, the instructor’s role in on-line collaborative learning environments and the suggested collaborative learning environment. Collaboration in Class The findings indicate that except from the initial phase in which learners start to get to know each other, some peers worked individually throughout most of the phases. They preferred to divide each task into sub-tasks and distribute the sub tasks among themselves. Their work is more cooperative than collaborative work. These peers 170 stressed that the time, work overload and knowledge levels drove them to work individually in most of the phases. The findings regarding both discussion groups and whole class were similar. The on-line interactions in both collaborative groups show that the interactions provide contributions to the participants. According to evidence for collaboration in on-line interactions to the findings, it is found that the Contributing behavior category has the highest proportion among the other behaviors for the whole class. Sharing Knowledge, and Exchanging resources and information are prominent behaviors of this category respectively. On the other hand, the Social Interaction behavior category was not present in the coded behaviors in either of the collaborative groups. This means that the discussions were task focused. According to the quantitative results, it is found that the average feedback provision time was about 2 days and 3 hours. Regarding the type of interactions, 34 % of the messages involved student-student interactions and participation of the students was about 40 % for the whole class. Moreover, disorganized messages and threads in the forum, lack of student-student interaction, the language of the discussions, lack of participation and activities in discussions in the forum were the major problems expressed by the participants of the study. The Factors Affecting On-line Collaboration According to the findings of the study, the factors affecting collaboration in on-line courses are given in Figure 1. Apart from these factors, the participants proclaimed that some strategies used in on-line discussions such as discussion circles were effective, as student-student interaction had the highest value. According to student feedback, it was also found that the provision of a platform in which all students or the group members could meet at a particular time and discuss the projects each week may be an effective strategy to promote effective peer interaction. The discussion is synchronous in nature; however, the forum tool will be used instead of chat tool as in the traditional synchronous communication. This is something similar to 8:30 classes in the traditional learning settings. Figure 1: The factors affecting on-line collaboration Role of the Instructor It was found that the objectives of the course, the content of the course and the nature and purpose of the task were the factors affecting the role of the on-line instructor. Moreover, almost all of the students emphasized giving immediate feedback, keeping discussions active, monitoring group work and motivating learners as the major roles of the on-line instructor in on-line collaborative learning environments. Suggested Collaborative Learning Environment The participants were asked to describe the desired tool with which they thought on-line collaboration could be conducted successfully. Responses indicated that participants consider the forum, instant messaging, common library and review tools as essential to successful online collaboration. Participants also valued inter-group interactions as much as intra-group interactions. Figure 2 depicts the structure and components of an online collaboration session proposed by the students. 171 Figure 2: The structure of on-line class with collaborative groups WebSCoRe: Web-based Synchronous Collaborative Review Tool According to the results, it was found that in the majority of groupware systems, collaboration is provided with some particular tools such as chat, whiteboard, file sharing, audio communication, discussion board, e-mail etc. These tools are important and essential parts of an on-line learning environment. However, it is essential to enrich the learning environment with variety of tools, which support different instructional strategies such as reviews. Because of the lack of diversity of tools, the contribution of collaborative reviewing strategy to gain collaborative skills and the suggestions of the participants of the study, a synchronous web-based tool called WebSCoRe, used for promoting on-line collaborative reviewing, is proposed in this study. WebSCoRe supports distributed and synchronous collaborative document reviewing in on-line learning settings. It is a real-time system that lets everyone participate regardless of location. The users of WebSCoRe are categorized into two groups: Students and Facilitators. A Student can be either the author, who is the creator of the document or a reviewer who gives comments about the document. The facilitator is the manager of the review session, and s/he initiates the review. The facilitator also has the same roles given to the reviewers. The document constitutes the core of the review session. We assume that authors have created the document and sent it to the instructor; therefore reviewers have already read the document and have prepared for commenting. WebSCoRe provides a meeting-based communication platform between reviewers and authors in order to share ideas and comments after the examination of the document created by the authors with an eye to criticism and correction. Discussion is the backbone of the WebSCoRe which promotes collaboration through rich interactions among on-line participants. The facilitator is responsible for facilitating the review session. Figure 3 displays a typical view from the review room screen. This is the main meeting space for on-line review. The facilitator, reviewers and authors examine the document and discuss about the comments in turn through this screen. The review room screen involves three parts. In the first part, reviewers and authors are listed and the content of the document is displayed. The content area is the place where the reviewers and the facilitator select the text from the content and give comments to the selected area. The second part is called the comment area. The comments given by either the facilitator or a reviewer are displayed in this area. Lastly, the third part is the discussion area where all participants of the review session may send messages to each other. The structure of the review room screen is almost the same for each user. However, according to the roles of the users, some of the functions may be disabled in the review room screen. For example, authors are not allowed to give comments about their own document. They may discuss comments given by the reviewers and the facilitator. Therefore, “Add Comment” and “Accept” and “Reject” functions are not displayed in their review room screens. 172 Figure 3: Review room screen Color is one of the most important features of the WebSCoRe. Color is used very simply to differentiate the users from each other. Each user chooses his/her color that will represent the user. It is assumed that this prevents confusion when the number of participants is high. Figure 4 displays an exemplary review session with four users (screenshots 1, 2, 3, and 4): the facilitator, two reviewers (Reviewer A and Reviewer B) and the author. The color of each user is reflected in the review room screens. The selected text area, the name of the users and the messages sent by the users are colored with a differently for each user (screenshot 1). In the example given below, Reviewer A selected the text from the content and gave the comment about the selected content. The color of the selected area is the same color with the color of Reviewer A, who sent the comment (screenshot 2). The other important feature of the WebSCoRe is that each comment is examined and discussed in turn. Therefore, when any of the reviewers or the facilitator provides comments, the functions for providing comments become disabled in the review room screens of the other users who are not providing comments. For example, in the example below, “Add Comment”, “Accept” and “Reject” buttons in the review room screens of other users become disabled when the Reviewer A sends his/her comment (screenshot 3). After discussing the comment, a common decision should be taken to accept or reject the comment. Then, Reviewer A finalizes the examination of his/her comment by either accepting or rejecting the comment. The review session is finalized by the facilitator (screenshot 4). It is assumed that the participants will come to a decision to approve or disapprove the document. The facilitator is responsible for approving or disapproving the document according to the decision taken by the review session members. It becomes clear that WebSCoRe focuses mainly the discussions among the participants. Utilization of WebSCoRe WebSCoRe was applied to review user manuals of a learning management system by the developers of the user manuals. Although the application is an initial experiment of the tool rather than a case study to validate the effectiveness, it contributes to understanding of effectiveness of WebSCoRe in on-line collaboration, and the strengths and weaknesses of the tool. 173 Screenshot 1 Screenshot 2 Screenshot 3 Screenshot 4 Figure 4: View from a review session with four users The participants of the application were the research assistants and graduate students. They were responsible for creating and reviewing the user manuals developed for a learning management system. All participants have the knowledge and experience regarding document reviewing. We constructed each review with at least three users: a facilitator, a reviewer and the author. At the end of the application of WebSCoRe, we had six review sessions. We then interviewed the participants and asked about the effectiveness, strengths and weaknesses of WebSCoRe. The semi-structured interview approach was used. The average time for the interviews was 40 minutes. We analyzed the interviews using qualitative analysis procedures. According to the findings of the analysis of the responses, we found that all of the participants believed that the WebSCoRe could promote both effective on-line collaborative learning and collaborative working. The participants generally compared document reviews in on-line settings with document reviews in face-to-face settings. Few of them believe that document reviews in on-line environment could not be effective as in the faceto-face settings. They argued that in on-line settings, it is necessary to keep sentences as simple and small as possible. However, in face-to-face environments, they argued, even mimics can explain lots of things, which cannot be expressed in another way. On the other hand, all of the participants argued that the distance education and on-line learning settings have become necessary in this era, and therefore, there is a need for variety of tool that might increase the quality of the instruction given through the Internet. They believed that the WebSCoRe could be an example of this kind of tool. In fact, one of the participants said, “… the WebSCoRe was developed for conducting documents reviews in on-line settings and it has the closest design to the document reviews in the face-to-face settings…” 174 The findings also revealed the strengths and weaknesses of WebSCoRe (see Table 3). • • • • • • • Table 3: The Strengths and Weaknesses of WebSCoRe Strengths Weaknesses Immediate feedback • Tracking is difficult when the group size is high Discussion-oriented • Communication is slow Color coding • There is a need for writing short sentences Providing ease of tracking comments and discussions • The details cannot be considered at all Providing high participation • The effectiveness is highly dependent on of the facilitator Increasing motivation Better than e-mail or forum tools The findings indicate that determining the review strategy, providing guidelines for review process, engaging the group in discussion of guidelines, formulating shared understanding, getting people to participate to the discussions, and providing support as needed are found as the major roles of the on-line facilitator in synchronous document review environment. The findings also provide some recommendations for the further improvement of the WebSCoRe such as giving more control to the participants, increasing the use of keyboard and extending the features of the tool by adding sound facilities. The findings state that like color, sound might also help to track the review session. Conclusion “Collaborative learning”, “team work”, “student-centered learning” and “students taking responsibility for their own learning” have become essentials in education in the information age. There is a high demand from staff and employers for students to graduate with good interpersonal skills, knowledge of group dynamics, the flexibility to work in teams, the ability to lean, to solve problems and to communicate effectively. This study indicates that collaboration in on-line learning environments depends on both instructional and technical issues. Although technical aspects are important, the spirit of distance education lies in the instructional strategies applied in the learning settings. Therefore, it is suggested to design and develop new instructional strategies and methods and then to design and develop necessary tools accordingly. Conducting reviews on a document is one of those effective instructional strategies and WebSCoRe, the collaborative review tool proposed, was found to be effective in facilitating on-line document reviews. According to Dillenbourg (2000), interaction can be rich when the participants explain themselves in terms of conceptions and not simply answer, when they argue about the meaning of terms and representations and shift roles. It is found that WebSCoRe encourages participants to build such rich interactions through providing a platform for working together on a common document, which provides for the collaborators to share and discuss their ideas about further revisions. Coomey and Stephenson (2001) argue that getting immediate feedback is one of the essentials of an effective on-line learning environment. Moore and Kearsley (1996) also add two more issues as the essentials such as increasing motivation and participation. The study also confirms that the giving immediate feedback, increasing participation, encouraging student-student interaction and social interaction are the three essential tasks of an effective on-line collaborative learning environment. According to the findings, it can be concluded that WebSCoRe satisfies these essentials of on-line learning environment. The study showed that pedagogical and the managerial functions of the facilitator to be more important and highly emphasized in on-line document reviews. The group size, the quality of facilitation, the speed, and the lack of facial facilities such as mimics, the limited size of discussions were stressed as the important issues affecting the effectiveness of WebSCoRe and given as the weaknesses of the tool. In fact, these issues are the weaknesses of the distance education environment rather than the weaknesses of WebSCoRe. To summarize, collaborative review is one of the instructional methods that promotes rich interactions. It is effective to apply collaborative reviewing strategy to develop collaborative skills and to promote collaborative learning. According to the findings of the study, it can be concluded that the WebSCoRe is a promising tool that might be a positive influence in developing collaborative skills and supporting effective collaborative learning. 175 References Allen, M. (2002). Guide to e-learning: building interactive, fun, and effective learning programs for any company, New Jersey, USA: John Wiley & Sons. Berg, B. L. (2004). Qualitative research methods for the social sciences, Boston, MA, USA: Pearson Education. Bruner, C. (1991). Thinking collaboratively: Ten questions and answers to help policy makers improve children's services, Washington, DC, USA: Education and Human Services Consortium. Coomey M., & Stephenson, J. (2001). On-line Learning: it is all about dialogue, involvement, support and control - according to the research. In Stephenson, J. (Ed.), Teaching and Learning On-line: Pedagogies for New Technologies, Sterling, VA, USA: Kogan Page, 37-52. Curtis, D. D., & Lawson, M. J. (2001). Exploring collaborative on-line learning. Journal of Asynchronous Learning Networks, 5 (1), 21-34. Dillenbourg, P., & Schneider, D. (1995). Collaborative learning and the Internet. Proceedings of International Conference on Computer Assisted Instruction (ICCAI 95), retrieved January, 05 2006 from http://tecfa.unige.ch/tecfa/research/CMC/colla/iccai95_1.html Dillenbourg, P. (1999). Introduction: What do you mean by "collaborative learning"? In Dillenbourg, P. (Ed.), Collaborative learning: Cognitive and computational approaches, Amsterdam: Pergamon, 1-19. Dillenbourg, P. (2000). Virtual learning environments. Paper presented at the Workshop on Virtual Learning Environments of the EUN Conference: Learning in the New Millennium: Building New Education Strategies for Schools, retrieved January, 05 2006, from http://tecfa.unige.ch/tecfa/publicat/dil-papers-2/Dil.7.5.18.pdf. Edutech (2004). Evaluation of Learning M<anagement Systems - 2003, retrieved January 05, 2006 from http://www.edutech.ch/lms/ev2.php. Gokhale, A. (1995). Collaborative learning enhances critical thinking. Journal of Technology Education, 7 (1), 22-30. Hiltz, S. R. (1998). Collaborative Learning in Asynchronous Learning Networks: Building Learning Communities. Invited address at WEB98 Conference, Orlando Florida, November 1998, retrieved January, 05 2006 from http://eies.njit.edu/~hiltz/collaborative_learning_in_asynch.htm. Horton, W., & Horton, K. (2003). E-Learning Tools and Technologies, Indianapolis, IN. USA: Wiley. Horton, W. (2000). Designing Web-Based Training: How to Teach Anyone Anything Anywhere Anytime, New York, USA: John Wiley and Sons. Kim, H. C. (2002). From comments to dialogues: a study of asynchronous dialogue processes as part of collaborative reviewing on the Web. Proceedings of 35th Annual Hawaii International Conference on System Sciences (HICSS-35) (Vol. 4), retrieved January, 05 2006 from http://csdl2.computer.org/comp/proceedings/ hicss/2002/1435/04/14350116.pdf. Kubala, T. (1998). Addressing student needs: Teaching on the internet. Technological Horizons in Education Journal, 25 (8), 71-74. Layzell, P., Brereton, O. P., & French, A. (2000). Supporting collaboration in distributed software engineering teams. Paper presented at the Seventh Asia-Pacific Software Engineering Conference (APSEC'00), December 58, 2000, Singapore. Moore, M. G., & Kearsley, G. (1996). Distance education: A systems view, Belmont, CA, USA: Wadsworth. Muronaga, K., & Harada, V. (1999). The art of collaboration. Teacher Librarian, 27 (1), 9-14. 176 Palloff, R. M., & Pratt, K. (1999). Building Learning Communities in Cyberspace, San Francisco, USA: JosseyBass. Palloff, R. M., & Pratt P (2001). Lessons from the Cyberspace Classroom: The Realities of On-line Teaching, San Francisco: Jossey Bass. Schoeder, U., & Brunner, M. (1996). Opinion - Distributed Discussion Support Based on the Metaplan Method. Technical Report No. PI-R 3/96, Software Engineering Group, Department of Computer Science, Technical University of Darmstadt, retrieved January, 05 2006 from http://www.pi.informatik.tu-darmstadt.de/ publikationen/technische%20Berichte/1996/TR_1996.3/opinion.html. Yanbaş, E., & Demirörs, O. (2003). A Web-based Review System and its Usability in a Small Software Organization. Paper presented at the Applied Informatics 2003 International Conference (IASTED), February 10-13, 2003, Innsbruck, Austria. 177 Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Educational Technology & Society, 9 (2), 178-197. An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students Petri Nokelainen Research Centre for Vocational Education, University of Tampere, FIN-13101 Hämeenlinna, Finland http://www.uta.fi/u/petri.nokelainen Tel: +358 40 557 4994 Fax: +358 3 6145 6111 petri.nokelainen@uta.fi ABSTRACT This paper presents the pedagogical usability criteria for evaluating the digital learning material. Pedagogical aspects of designing or using digital learning material are much less frequently studied than technical ones. Further, there are relatively few inventories measuring subjective end-user satisfaction with the pedagogical aspects of digital learning material and not a single inventory has undergone a rigorous process of empirical psychometric testing. They include the following components: 1. Learner control, 2. Learner activity, 3. Cooperative/Collaborative learning, 4. Goal orientation, 5. Applicability, 6. Added value, 7. Motivation, 8. Valuation of previous knowledge, 9. Flexibility and 10. Feedback. The pedagogical usability criteria have been operationalized into an on-line Likert -scale self-rating Pedagogically Meaningful Learning Questionnaire (PMLQ) that has 56 items. In the PMLQ, separate items have been developed to measure the usability of the learning management system (LMS) and the learning material (LM). When evaluating the usability of a LMS, it is possible in the pedagogical context to evaluate the kind of learning material it enables the users to produce. Evaluation of the usability of a LM is based on a fact that the learning content is based on a certain learning goal or goals. The criteria of pedagogical usability presented here have undergone two-step psychometric testing process using empirical samples of 5th and 6th grade elementary school students (n = 66 and n = 74). Students evaluated one LMS and four LM’s with PMLQ. Results supported the existence of theoretical dimensions of the criteria. The PMLQ was able to capture differences in the pedagogical usability profiles of the learning modules. Generalizability of the pedagogical usability criteria to other domains is limited by the small sample size, the small age range of respondents’ and the small number of learning material evaluated. However, empirical studies that aim at evaluating a more representative set of learning material in different domains targeted for both adolescent and adult learners are currently conducted. Keywords Pedagogical issues, Usability, Evaluation methodologies, Evaluation of CAL systems, Authoring tools and methods Introduction New software has eliminated some previously prerequisites student skills, while simultaneously creating new requirements for the computer systems they are using. The usability analysis of learning material is intended to influence the process by which the correct applications and target groups find one another so that a student can focus his/her energy on the content of the learning material rather than technical issues caused by the software design or user interface. Several sets of recommendations for the evaluation of technical usability have been developed over the last twenty years (e.g., Shneiderman, 1998; Chin, Diehl & Norman, 1988; Nielsen, 1993; 1994; Lin, Choong & Salvendy, 1997; Preece, Rogers & Sharp, 2002; Chalmers, 2003; Tognazzini, 2003). However, pedagogical aspects of designing or using digital learning material are much less frequently studied than technical ones. This paper presents criteria for evaluating the pedagogical usability of digital learning material. The purpose of the criteria is not to brand any learning material as “good” or “bad,” but to help users choose the most suitable alternative for any particular learning situation. When evaluating technical usability, the basic assumption is that it should be easy to learn to use the central functions of the system and the functions are efficient and convenient in use. Another assumption is that error responses to incorrect operation of the software should help teach the user to use the system as intended so that the error will not be repeated. When evaluating pedagogical usability, the assumption is that the designers of the learning platform or learning unit were guided by either a conscious or subconscious idea of how the functions ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 178 of the system facilitate the learning of the material it is delivering. Examples of the range of learning theories that influence design paradigms are objectivism (instructivism, behaviourism), and constructivism (focus on learner, learner’s active role in learning and learning from experience). The criteria presented here have been operationalized into an on-line application (Nokelainen, 2004a) that applies a Likert –scale self-rating questionnaire with 56 items (see Appendix). In the Pedagogically Meaningful Learning Questionnaire (PMLQ), separate items have been developed to measure the usability of the learning platform and the learning unit it is delivering. When evaluating the usability of a specific learning platform (learning management system), we can evaluate how easy it is to use the platform itself (technical usability), or what kind of learning material it enables the users to produce (pedagogical usability). When evaluating the usability of a learning unit or a learning object, we assume that each learning unit has its own interface relating to the content, and learning content based on a certain learning goal. When evaluating the pedagogical usability of a learning unit, we must try to control the effect of the pedagogical solutions of a learning platform. The compilation of criteria for usability of digital learning material presented here is based on the following preparatory analyses: 1. Charting of systems and taxonomies developed for the evaluation of digital learning materials, 2001-2002, 2. Analysis of learning theories suitable as the basis of evaluation, 2001-2002, 3. Compilation of the first draft of criteria for pedagogical usability, 2001-2002, and 4. Testing of the first draft of criteria for pedagogical usability with empirical samples, 2001-2002. Central Concepts Digital learning material means in this research all material that is designed for educational purposes, published in a digital form and intended to be accessed by computer. A Learning object (LO) is, as the name implies, the smallest reasonable unit of learning material. Examples of this are a pronunciation sample from an English language-teaching program or an animation clip that describes how a dangerous procedure is completed safely. Learning material (LM) can include such things as 1) a WWW page introducing the effects of image compression on the quality of an image, 2) a JAVA applet to practice touch-typing, 3)an individual learning module in a learning platform, the topic of which might be, for example, positioning decimal numbers on a continuum. Regardless of the type, each individual learning material has its own user interface, the usability of which can be evaluated, as well as a definable learning goal. A Unit of learning material (ULM) consists of several individual learning materials connected by a common main goal. A ULM is a concept that combines several learning objects (and possible meta, history and relation data) related to the same subject as a single entity. A Virtual learning environment (VLE) is an application that uses learning materials, or units of learning material. There are numerous commercial and non-commercial virtual learning environments available. They can be divided into two groups according to whether they include prepared learning material (e.g., eWSOY’s OPIT) or offer a presentation service for learning material provided by the user (e.g., BlackBoard, Belvedere, Dyn3W, Future Learning Environment, Knowledge Forum, WebCT) Usability can, according to Keinonen (1998, 62) be defined as a characteristic related to 1)the product’s design process, 2) the product itself, 3) use of the product, 4) user experiences of the product or 5) user expectations. In the present research, usability is understood primarily as the product’s usability attributes, which are measured through subjective user experiences with a self-evaluation questionnaire. The operationalisation of the usability attributes in this study is done according to the model presented by Nielsen (1990, 148). Figure 1 shows that the top-level concept is system acceptability. Acceptability is further divided into two parts: 1) Social acceptability and 2) Practical acceptability. The implementation of a system that serves as a presentation platform reveals the views of its developers about both social and practical acceptability. The example given by Nielsen to illustrate social acceptability (ibid. 147-148) is a curriculum that lets only the teacher make changes that increase viewpoints on the studied subject, which could in some societies be seen as inappropriately emphasising the teacher’s authority and restricting the students’ individual knowledge formation. The tone of Nielsen’s later example of social acceptability (1993, 24) puts more emphasis on the ethical and 179 moral choices of the program developers: Is it right that people applying for unemployment benefits would have to use an application form that attempts to use their input (questions) to reveal those who are applying for benefits for invalid reasons? According to Nielsen (1990, 148), practical acceptability means, among other things, cost, compatibility, reliability and usefulness of the system. Nielsen states (1990, 148) that usefulness can be further divided into utility and usability. Utility refers to the ability of the system to generally provide functions that correspond with the needs of the users and usability refers to how well the users are able to use the functions offered by the system (ibid. 148-149). According to Nielsen’s examples (1990, 151), an entertainment application (e.g., a game) is utilisable for users if they enjoy themselves with it, and learning material is utilisable if students learn with it. Nielsen (ibid. 151) further divides usability into learnability, efficiency, memorability, errors, and satisfaction, calling them usability attributes. Learnability depends on how long beginners’ use a system before they learn the essential skills necessary to perform their tasks. Efficiency refers to how well experienced users can operate an application after they have mastered it. Memorability means the ability of an occasional user who has previously used the system to remember its operational principles. Errors are divided into two groups: less serious errors that disturb the work of the user and serious errors that, for example, endanger the preservability of the users’ outputs. Satisfaction is a subjective judgment by the user. Figure 1. Conceptual mapping of the technical and pedagogical usability to the Nielsen’s usability model (Adapted from Nielsen 1990, 148) Pedagogical usability, which is defined in this research according to Nielsen’s classification (1990, 148), is a sub-concept of utility, and technical usability is a sub-concept of usability (Figure 1). Thus, in addition to the dialogue between a user and a system, the pedagogical usability of a system and/or learning material is also dependent on the goals set for a learning situation by the student and teacher. Evaluation in this research is a subjective judgment made by the user of digital learning material. The evaluation criteria are applied using a self-evaluation questionnaire that employs a Likert scale 1 (strongly disagree) – 5 (strongly agree), 6 (not applicable), 7 (don’t know). The teacher and learners/students each have their own versions of the evaluative statements. Pedagogical Usability Learning is a largely unobservable and uncontrollable process that happens all the time. Attributes attached to learning such as ‘effectiveness’ or ‘informality’ are both difficult to understand and measure. The term education, on the other hand, is easier to analyse because it is more often bound to observable artefacts such as books or computer programs that are related to a curriculum. Some researchers term education that is tied to a set 180 curriculum as ‘formal,’ and education that is more conversation based in nature as ‘informal’ (Livingstone, 2000). Although we will go no further with this topic, one may ask what portion of measurement error of the pedagogical aspects of usability is related to the difference between the formal and informal aspects of education. Education can be provided in groups or according to individual needs as teacher directed, cooperative, or individual practice. When computers and digital learning material are used in a learning situation, it is expected that this is done to introduce identifiable added value to the learning in comparison to, for example, printed material, and material produced by the teacher or the students themselves. The expectation of “added value,” not merely “reaching the same level,” is demonstrated by the fact that each group of students is under the charge of an educational expert who has primary responsibility for the provision of education that agrees with the general goals, and further supported by computers and software that require not only capital investment but also continuous maintenance. In the case of distance education, the equation is different and so it may be that reaching the same level as in contact teaching is considered acceptable. In this context, the level of educational quality desired is at least that of the average performance in a good quality contact teaching context – it is unlikely that one would want to match digital material with poor learning materials and teaching practices! In the following the problematic concepts of the quality of education and learning are discussed in more detail. Many of the “innovations” that have been executed in computer environments have their equivalents in the world of traditional education. For example, instead of a “mind map” type of computer application that collects arguments and counter-arguments, cooperative knowledge construction can be achieved with large pieces of paper on the walls of a classroom, on which various points and opinions that have been raised during a week can be recorded. The added value of computers in this case is technical, because it gives the students a chance to work simultaneously on several different things (for example in different subjects) and save each phase of their work on their own workspace. In a classroom, the activity is more limited and inflexible because the space on the walls of the classroom is limited. Then again, if the unfinished work and problems were visible at all times, the students could work on them in the background for the whole time that they spend in the classroom. A counterargument to this is the Sartrean thought about comprehensive thinking that follows the learner from school to home. In this case, a computer-based application allows the learners to continue their work at home, in a library or in a coffee shop with the help of a computer network. Developing the New Criteria for Pedagogical Usability We analysed the criteria that have been developed for the factors in learning materials that promote learning (Reeves, 1994; Squires & Preece, 1996; Quinn, 1996; Albion, 1999; Squires & Preece, 1999; Horila, Nokelainen, Syvänen & Överlund, 2002). The summary of different criteria is presented in Table 1. The small number of existing models shows that pedagogical aspects of designing or using digital learning material are much less frequently studied than technical ones. The table shows that one criteria is a theoretical model (Squires & Preece, 1996), four are theoretical models with heuristic checklists (Reeves, 1994; Quinn, 1996; Albion, 1999; Squires & Preece, 1999) and only one is inventory measuring subjective end-user satisfaction with the pedagogical aspects of digital learning material (Horila et al., 2002). Further, not a single heuristic checklist or inventory had undergone a process of empirical psychometric testing. The most common pedagogical usability features of the criteria presented in Table 1 are “learner control”, “possibility for cooperative or collaborative learning activities”, “explicit learning goals”, “authenticity of learning material” and “learner support (scaffolding)”. In this study, we created new criteria for the assessment of the pedagogical usability of digital learning materials, as the earlier research work had not addressed all the relevant issues of the topic neither on a theoretical nor practical level. The existing criteria neglect partially the role of learner’s activity, added value of digital learning material, learning motivation and feedback related to user input. We found that none of the existing criteria included such concepts as valuation of previous knowledge and role of pre-testing and diagnostics. Our model includes ten dimensions as follows: 1. Learner control, 2. Learner activity, 3. Cooperative/Collaborative learning, 4. Goal orientation, 5. Applicability, 6. Added value, 7. Motivation, 8. Valuation of previous knowledge, 9. Flexibility and 10. Feedback. 181 Next we proceed to describe the theoretical structure of each dimension. The model presented here contains 51 sub dimensions. To get the picture of the overall structure of the criteria, the reader is encouraged to see the pedagogical usability sub dimensions in the Appendix 2. 1 Learner Control When learning a new topic, the learner’s memory should be burdened to an optimal level (Miller, 1956; Shneiderman, 1998, 355). It is difficult to define a universally optimal level (generally people can have 5 to 9 items in their short term work memory), but it is certainly helpful to break down the material to be learned into meaningful units (Wilson & Myers, 2000). In so-called structured learning materials, the learned material has already been broken down into meaningful units from the point of view of the students (albeit usually by the teacher). Such a “one-size-fits-all” approach has been criticized for the fact that the students are required to adjust their learning to fit the teacher’s conception of the best way of learning certain material (Jonassen, Myers & McKillop, 1996). The factor structure of this dimension is presented in Table 2. 2 Learner Activity A teacher’s “didactic role” in a learning situation may strongly scaffold the learners’ own activity, and, correspondingly, the learners’ independent activity may be increased when the teacher stays in the background, as a “facilitator” (Reeves, 1994). Learners’ activity is determined in large measure but the characteristics of the learners themselves, but the learning material can affect it through assignments that support student activity by being interesting and based on real life. An alternative approach to structured material is problem-based learning (e.g., Savery & Duffy, 1995). In this case, the teacher gives the students a certain amount of source material from which the students (individually or in groups) construct their own conception of the topic to be learned. However, even in this approach all available information (i.e. “source material”) has usually already been structured by someone else! Problem-based learning or accessory software also increases student activity (Jonassen, Peck & Wilson, 1999). Items 8, 9 and 10 reflect more collaborative nature of learning (Forman & Cazden, 1985; Palloff & Pratt, 2005; Tudge & Rogoff, 1989). Study of collaborative activity in computer-based settings, often cited as computer supported collaborative learning (CSCL), can enhance our understanding of the processes of productive interaction (e.g., Littleton & Häkkinen, 1999). The factor structure of this dimension is presented in Table 3. 3 Cooperative/Collaborative Learning Cooperative and collaborative learning means studying with other learners to reach a common learning goal. According to Barab and Duffy (2000, 27-28), learners are moving away from acquisition metaphor (i.e., acquiring knowledge that is constituted of symbolic mental representations) to participation metaphor (i.e., knowledge is considered fundamentally situated in practice). Instead of acquiring personal knowledge, learners construct knowledge as members of communities in practice (Lave & Wenger, 1991). To be more specific, cooperative learning is more structured than collaborative learning, as the teacher has the control. Learning takes place in groups in which the members gather and structure information, in which case the system or learning material should offer the learner tools that can be used to communicate and negotiate different approaches to a learning problem (Jonassen, 1995). Through the use of computer-supported learning material it is possible to practice cooperative learning so that all students are connected to each other over a distance, for example through discussion groups and chat forums (Quinn, 1996; Reeves 1994). In a system or learning material supporting cooperative knowledge construction (KB, knowledge building or KC, knowledge construction) may include, for example, a visual tool that the learners can use to fashion simultaneous mind maps of the topic. Often such a system also includes tools for social navigation (Mayes & Fowler, 1999; Kurhila, Miettinen, Nokelainen & Tirri, 2002) with which a learner gains information about what the other learners have done (asynchronic social navigation) or are presently doing (synchronic social navigation). 4 Goal Orientation As learning is a goal-oriented activity, goals and objectives should be clear to the learner (Quinn, 1996). The best results are attained when the goals of the learning material, teacher and student are closely aligned. The goals 182 may vary from concrete (for example the basic techniques of first aid) to abstract (learning material aimed to develop the appreciation of modern arts) (Reeves, 1994). If the learners themselves do not set the goals, their meaningfulness should be justified from the point of view of motivation. The students should have a chance to pursue their own interests in relation to the learning goals. According to instructivist learning theory, learners should be introduced to only with a few, clearly specified goals at any one time. According to constructivist learning theory, the goals should be clearly defined, but they have to originate, as much as possible, with the learners themselves. (Wilson & Myers, 2000.) If the goals do not originate with the students, their meaning should be explained to them. The students should have a chance to make choices with respect to the course of their studies in relation to the learning goals. For example, in some learning materials, the assignments are graded and granted points and thus the student can access the next assignment after garnering enough points from the previous assignments. The downside of the point system is the fact that the students are less able to complete the material in the order of their own preference, or bypass areas that they already know. Table 1. Summary of the pedagogical usability criteria research Reeves (1994) “Pedagogical dimensions” b 1. Learner control 2. Pedagogical philosophy 3. Underlying psychology 4. Goal orientation Quinn (1996) Squires & Preece “Educational (1996) “JIGSAW design heuristics” b model” a 1. Clear goals and objectives 2. Context meaningful to domain and learner 3. Content clearly and multiply represented and multiply navigable 4. Activities scaffolded Albion (1999) “Content heuristics” b 1. Establishment of context 2. Relevance to professional practice 3. Application operation tasks 3. Representation of 3. Match between professional responses designer and learner to issues models 3. Technical requirements 4. General system operation tasks 4. Relevance of reference materials 4. Intuitive efficiency 5. Elicit learner understandings 5. Presentation of video resources 6. Teacher role 6. Formative evaluation 6. Assistance is supportive rather than prescriptive 7. Program flexibility 7. Performance should be ‘criteriareferenced’ 7. Materials are engaging 8. Value of errors 8. Support for transference and acquiring 'selflearning' skills 9. Support for collaborative learning 8. Presentation of resources 10. Motivation 11. Epistemology 9. Overall effectiveness of materials 1. Appropriate levels of learner control 2. Navigational fidelity Horila, Nokelainen, Syvänen & Överlund (2002) “Pedagogical usability of digital learning environments” c 1. Specific learning tasks 2. General learning tasks 5. Experiential value (Authenticity) 9. Cooperative learning Squires & Preece (1999) “Learning with software heuristics” b 4. Prevention of peripheral cognitive errors 5. Understandable and meaningful symbolic representation 1. Learnability 2. Graphics and layout 5. Suitability for different learners and different situations 6. Support personally 6. Ease of use: significant approaches Technical and to learning pedagogical approach 7. Strategies for the 7. Interactivity cognitive error recognition, diagnosis and recovery 8. Match with 8. Objectiveness curriculum 9. Sociality 10. Motivation 11. Added value for teaching 12. User activity 13. Accommodation of individual differences (Scaffolding) 14. Cultural sensitivity a = Theoretical model. b = Theoretical model and heuristic checklist. c = Theoretical model and subjective enduser inventory. 183 Table 2. Factor structure of the first pedagogical usability dimension (1. Learner control) Target group Child Factors 1.1 Minimum memory load Items LMS 1.2 Meaningful encoding 1.3 Take responsibility for one’s own learning 1.4 User control 2 Teacher Items LM 47 Items LMS Adolescent/adult Items LM 47 Items LMS Items LM 47 48, 49 48, 49 48, 49 1 1 1 2 2 2 2 2 8.2 Valuation of Previous Knowledge 40 40 40 -> Elaboration Note. LMS = Learning management system. LM = Learning module/material. See Appendix for the item labels. Italicised items show relationship to the other usability dimensions. 5 Applicability The approach taken in learning material should correspond to the skills that the learner will later need in everyday and working life (Jonassen, Peck & Wilson, 1999; Quinn, 1996). The skills or learned knowledge should be transferable to other contexts (Quinn, 1996; Reeves, 1994). Learning something new is most effectively accomplished through so-called learning by doing methods that involve practical tasks. Learning by doing has been found to be an effective learning method for both practical (changing some part of a paper pulp machinery) and in abstract (integration in mathematics) issues. Learning material should always be at an appropriate level from the point of view of a learner’s learning process (Wilson & Myers, 2000). For example, students in the early grades of comprehensive school have limited ability to adopt abstract concepts, but it becomes far easier for fifth and sixth graders. It is possible to adapt learning material to meet the needs of a student for, example, by periodically asking for feedback about the experienced difficulty of the material. In a dynamic system, it is possible to follow a student’s activity, such as the time it takes to finish assignments and the number of mistakes, and adapt future tasks according to this information. Learning material should be planned and executed in cooperation with both groups of end-users (students and teachers); systems produced by experts rarely meet their everyday needs without modifications. In the planning of learning material special attention can be given to those issues that are most likely to cause problems for the learners (for example the adoption of a new concept) and build support structures that the students can use if needed. The system can also observe the learner’s activity and proactively offer hints (prompting and fading, Hannafin & Peck, 1988, 47). When students are in a so-called ZPD (zone of proximal development, Vygotsky, 1978, 84-91), they are in the process of rising into a new level of skills and knowledge and progress depends on the understanding of some detail. When they acquire a hint to solve this problem (scaffold), they experience a break through and can progress to the next step in their studies (Chalmers, 2003). 6 Added Value A formal learning situation, planned by a teacher or a mentor, can be carried out in many ways, for example through cooperative or individual learning approaches, directed by a teacher or as group work or individual practice. When computers and digital learning material are used in a learning situation, it is expected to introduce definite added value to the learning in comparison to, for example, printed material, and material produced by teacher or the students themselves. The added value is usually in the form of creative use of the possibilities that the computer offers, for example voice, image and video files: the learners can choose a media that best fits their preferences. Jansen, van den Hooven, Jägers and Steenbakkers (2002) point out that especially young students are familiar with computers and multimedia programs (for example so-called video games) and so similar components in learning material suit their life styles and future work. Jansen et al. (2002) have devised a list of aspects of computer-assisted learning that offer added value: (1) adaptability to individual needs, (2) number of flexible options, (3) learning is controlled by the learner, initiated by the learner and is in the form that the learner desires, (4) interesting contents, (5) development of communication, and (6) active participation of the students. In practice, the realization of all the items on this list requires that the developers of the learning 184 material have multidisciplinary skills and knowledge, experience or teaching and time to develop the learning material. The system or learning material should offer the students with tools that are suited to control the contents of the learning material and that make the use of the material more effective and economic. The student should have a feeling that the topic is best learned through the use of a computer. Table 3. Factor structure of the second pedagogical usability dimension (2. Learner activity) Target group Child Factors 2.1 Reflective thinking 2.2 Problem-based learning (PBL) Items LMS Teacher Items LM 3, 4r Items LMS Adolescent/adult Items LM Items LMS Items LM 3 5, 9 5, 9 5, 9 2.3 Use of primary data sources 6 6 6 2.4 Immersion 7 7 7 2.5 Ownership 8, 10 8, 10 8, 10 Note. LMS = Learning management system. LM = Learning module/material. r = A reversed item, i.e. values in 1 to 5 Likert scale change as follows: 1 -> 5, 2 -> 4, etc. See Appendix for the item labels. 7 Motivation Motivation (Latin: Movere, to move) affects all learning and makes people behave the way they do. Behaviourists explain the motivation to do things by reference to instincts, desires and reinforcement; cognitive theorists rely on models of cognitive processes and analysis (Wilson & Myers, 2000, 65). Motivations, which can be either consciously or subconsciously goal-oriented, support the direction of an individual’s general behaviour (Ruohotie, 1996). Key concepts of motivation include incentives, self-regulation, expectations, attributions of failure and success, performance or learning goals, as well as intrinsic or extrinsic goal orientation (Reeves, 1994; Ruohotie & Nokelainen, 2003). Someone with intrinsic goal orientation strives to reach learning goals for his or her own purposes, because the material is interesting in itself. Someone with an extrinsic goal orientation strives to achieve better results than others (highest grades in the class), to achieve an extrinsic reward (pay raise, grant) or to avoid punishment (for example repeating a course). Contextual motivation, relating, for example, to the interest of the studied topic, varies dynamically. General level motivation is more static, because it can change according to a stage of life. Attitude and motivation can be distinguished, because attitude affects the quality of one’s work, while motivation affects one’s alertness and vigour (Ruohotie, 1996). 8 Valuation of Previous Knowledge Learning material that presumes previous knowledge from the learner may expect the learner to already possess some skills or knowledge that have been presented, for example, in some earlier learning materials. Learning material that respects the learner’s previous knowledge takes into account individual differences in skills and knowledge and encourages them to take advantage of it during studies. This approach favours learner’s elaboration, contemplation or new issues and the analysis of their relationship with learner’s earlier knowledge constructs (Wilson & Myers, 2000). Computer-assisted learning material can include various predefined “paths” that demonstrate the use of the learning material depending on the previous knowledge of the learner. The learning material may review the central concepts from earlier studies that are important for understanding the present material. In this case, the importance of the learner’s previous knowledge is diminished, but the importance of the previous material and the cumulative nature of knowledge become clear to the learner. 9 Flexibility Flexible learning material takes into account learners’ individual differences. For example, a test given at the beginning of the studies can provide information about previous knowledge, interest towards learning the topic, and expectations of what the learner seeks to gain from the studies. Information gained in pre-testing can be used 185 to provide the learner with optional or alternate routes in the studies (Hannafin & Peck, 1988, 48; Wilson & Myers, 2000). The learner should be given a chance to navigate freely through the learning material. Flexibility in the contents of the learning material means that the material contains diverse assignments (Quinn, 1996). The more adaptable and broadly defined the assignments are (content description with meta coding), the easier it is to combine them to fit the student’s individual needs (Leflore, 2000). Collis and Moonen (2002, 10) address the need for social organization of learning (face-to-face, group, individual), language to be used during the learning situations, modality and origin of the learning resources (teacher, student, library, Internet) and instructional organization of learning (assignments, monitoring). They further suggest that students should share the responsibility of identifying appropriate additional learning resources and even contribute to the learning resources (ibid, 13). 10 Feedback The system or learning material should provide the student with encouraging (Albion, 1999; Quinn, 1996) and immediate feedback. Encouraging feedback increases learning motivation; immediate feedback helps the student to understand the problematic parts in his or her learning. Immediate feedback is particularly important in behaviourist (stimulus – reaction) learning materials (Wilson & Myers, 2000). In interaction between a student and a computer, the benefit is that feedback can be given immediately after the student’s action. On the other hand, computer provided feedback is rarely so valuable and well timed that it can support learning by itself. If the other side of the dialogue is a human (for example, a peer learner on a discussion forum for the learning material), the feedback is more likely to support reflection, depending, of course, on the quality of interaction inside the peer group (Kolodner & Guzdial, 2000). In so-called faultless learning, the learning is based on the fact that mistakes are corrected immediately and one cannot progress to the next part in the material before this is done. On the other hand, in a constructivist learning situation, in which the problem is solved in cooperation, the learners may sometimes come up with a faulty solution to an ill-structured problem (Jonassen, Peck & Wilson, 1999, 196). In this case, the teacher or mentor can react to the mistake after a delay and give the students hints at the correct solution or links to additional material. Empirical Evaluation of the Criteria A PMLQ (Pedagogically Meaningful Learning Questionnaire) research instrument was developed on the basis of the aforementioned technical and pedagogical usability criteria. The first version of the instrument contained 92 multiple-choice items. The five-point scale ranged from 1 (totally disagree) to 5 (totally agree). The sixth response option was “Not applicable”. The PMLQ has three parts. The first part concerns technical and pedagogical usability of the learning platform (or system) containing 43 items; the second is about technical usability of the learning material (24 items), and the third part measures pedagogical usability of the learning material (25 items). The propositions are clearly marked when measuring issues about system or contents. Empirical measurements were carried out in October 2003 with 5th and 6th grade elementary school students (n = 66) and their teachers (n = 4). Twenty-four of the children were boys and 42 were girls. Three teachers were male and one was female. Participants evaluated the OPIT learning system and four learning modules embedded into the system. Two of the modules were about mathematical topics (decimal numbers and fractions) and the remaining two modules were about the third foreign language of Finnish school children, English (singular vs. plural and knowledge test about British Isles). The design of this empirical test contained three stages: First, each participant was profiled using a previously developed ACALQ (Abilities for Computer Assisted Learning Questionnaire) instrument that characterizes respondents by their self-rated motivational level, metacognitive preparedness and social abilities (Nokelainen & Ruohotie, 2004). This information was intended for future purposes in order to control individual differences in the evaluation of learning materials. Second, participants filled out a PMLQ questionnaire for the OPIT platform. Shortly after using each of the four modules they filled out a similar usability questionnaire for each module. Third, for each module, individual score and turnaround time was recorded. 186 Results of this first version of the PMLQ instrument showed that full scale of 1 to 5 was in use. Dependencies between the variables were investigated with Bayesian dependency modeling (Myllymäki, Silander, Tirri & Uronen, 2002; Nokelainen & Tirri, 2004) due to discrete measurement scale and small sample size. The results supported the chosen dimensionality (Nokelainen, 2004b). The next step in the development process was to revise the old items because interviews with the children and teachers revealed some deficiencies in wording. The second version of the PMLQ’s pedagogical usability section contained 56 items after five new items were included (see Appendix). Empirical measurements were carried out in February 2004 with 5th and 6th grade elementary school students (n = 74) with age median of 12 years. Participants evaluated the OPIT learning system and two learning modules embedded into the system. First module was about decimal numbers, and the second one was about English language singulars and plurals. Both modules were chosen by the teacher and research team. The structure of math module was non-linear, allowing the learner to choose from several equal paths to complete the sequence of numbers tasks. The English module represented typical fill-in test that proceeds in a linear fashion with minimal user control. Analysis showed that although the distribution of most items was skewed, the full five-point Likert scale was in use. Table 4 contains the Cronbach’s alpha reliability scores for the first part of the study measuring usability of the OPIT learning management system. Fifty-two out of 74 students evaluated the system with the PMLQ. Only a few items measure the pedagogical usability of a learning management system, as our main focus of interest is in the contents (i.e., learning material). As the alpha coefficient tends to grow as the number of items increases, it is not surprising that the third dimension, Cooperative/Collaborative learning, has higher alpha value (.86) than the fourth dimension, Goal orientation (.72). Table 5 contains the reliability coefficients for the dimensions measuring the pedagogical usability of the two learning modules. Thirty-four students used and evaluated the “Decimal numbers in a continuum” math learning material. Reliability coefficients range from .75 (7. Motivation) to .87 (5. Applicability). Although high alpha value indicates unidimensionality, the inner structures of the dimensions described in this paper are in most cases quite complex, as presented earlier in tables 2 and 3. The second part of the Table 5 shows reliability coefficients for the English “singular or plural” learning material. The alphas are satisfactory ranging from .80 (9. Flexibility) to .92 (5. Applicability). When we compare the pedagogical usability profiles of the two modules (presented in Table 5), we notice that learner control, activity and flexibility dimensions have higher mean scores for the math module (M = 3.7, SD= .94; M = 3.6, SD= .79; M = 3.4, SD= .66) than for the English module (M = 3.4, SD= .87; M = 3.3, SD= .94; M = 3.2, SD= .68). Although the mean differences are quite small, the measurement instrument is able to capture differences in the pedagogical usability profiles. This leads us to cautiously conclude that the finding supports our earlier hypotheses regarding the different roles of the two learning modules: The math module is designed to allow a learner possibility to complete exercises of a different kind and to decide where and when to proceed, the English module presents more inflexible and linear way of practising singulars and plurals with only one type of exercise. Description of the Utilization of the Criteria A typical user of the criteria for pedagogical usability of digital learning material is a teacher, application programmer, or student. The following describes the use of the criteria on a general level: 1. The user logs into the eValuator system. S/he submits background and profile information by completing self-assessment questionnaire scales on, for example, motivation, learning strategies and social skills. 2. The user chooses to either search earlier evaluations or to assess new learning material. 3. The user provides criteria for the search or the evaluation (e.g., target group, topic, purpose of the material), defines the target (e.g., LM, ULM or LMS) and, if needed, the usage context of the material (e.g., autonomous studying at school, autonomous studying at home, teacher-directed learning) and the type of material (e.g., supporting/additional assignments, learning a new topic). 4. On the basis of the above criteria, the application generates an evaluation form, which contains statements drawn from the criteria. Each target group has a different set of statements (e.g., children in the 5th or 6th grade who evaluate learning material from the point of view of their own learning, teachers who read evaluations from other teachers or evaluate material that they themselves use, adolescents in polytechnics or 187 5. 6. 7. universities who choose material for their own use or evaluate material that they use, and adults in working life). The user provides his or her subjective evaluation of the usability of the material by responding to statements. At the end of search or evaluation, the user is presented with an overview of how the material corresponds with the different parts of the technical and pedagogical usability criteria, as evaluated by the user. The system serves also as a search tool. When the evaluation database contains more evaluations, user is able to conduct personalised searches in order to find the most appropriate learning material. The personal profile (collected with ACALQ and various background questions) and pedagogical usability scores (collected with PMLQ) act as filters. For example, user is able to ask for 5th grade elementary school math LM’s that have something to do with decimal numbers and are suitable for collaborative working. A second example might be to ask for such university level statistical LM’s that are rated high by registered users whose personal profile is close to the users’ profile. Table 4. Reliability coefficients of the dimensions measuring usability of a learning management system Sample 1a Pedagogical Usability Dimensions Items M SD α 1. Learner control — 2. Learner activity — 3. Cooperative/Collaborative learning 11, 12, 13, 14, 15 3.4 1.22 .86 4. Goal orientation 23, 24 3.7 1.11 .72 5. Applicability — 6. Added value 35 3.6 1.36 — 7. Motivation — 8. Valuation of previous knowledge — 9. Flexibility — 10. Feedback 52 3.7 1.30 — a Elementary school students, n = 52, age range 10 – 13 years, age median 12 years. Note. See Appendix for the item labels. Conclusion A rapidly growing quantity of digital learning material is available to fill various learning needs via online (e.g., Internet, mobile networks) and offline (e.g., DVD, CD-ROM) media. A broad array of learning material is offered to learners all over the world. However, whether the application selected achieves the desired ends depend on various factors; for example, whether the learner needs to learn new things or just refresh old knowledge, is working alone or collaboratively with other learners, or is willing to use commercial or noncommercial material. This paper presents criteria for evaluating the pedagogical usability of digital learning material. In practice the role of the criteria is to give the learner a chance to choose the most suitable learning material possible for any learning situation. Previously constructed criteria for evaluating pedagogical usability have been introduced, and differences between them have been considered. The various phases of creating criteria in this study have been described and the empirical research setting has been evaluated. The 56 –item multiple-choice inventory (PMLQ) that operationalises the pedagogical usability criteria was presented. Results of a two-stage empirical evaluation of the criteria with 5th and 6th grade elementary school children (2003 n = 66, 2004 n = 74) was reported. Results supported all theoretical dimensions of the pedagogical usability criteria. The PMLQ was able to capture differences in the pedagogical usability profiles of the learning modules. Table 5. Reliability coefficients of the dimensions measuring usability of two learning modules (math and English) Math English Sample 2a Sample 3b Pedagogical Usability Dimensions Items M SD M SD α α 1. Learner control 1, 2, 47, 48, 49, 40 3.7 .94 .86 3.4 .87 .90 2. Learner activity 3, 4, 5, 6, 7, 8, 9, 10 3.6 .79 .86 3.3 .94 .91 3. Cooperative/Collaborative learning 11, 12, 13, 14, 15 3.4 .90 .76 3.4 1.05 .88 188 4. Goal orientation 5. Applicability 20, 21, 22, 23, 24 3.5 .91 .78 3.3 .97 .85 25, 26, 27, 28r, 29, 30, 31, 54, 3.5 .75 .87 3.4 .85 .92 55, 56, 40, 48, 49 6. Added value 32, 33, 34, 35, 45 3.1 1.05 .82 3.2 1.10 .88 7. Motivation 13, 36, 37, 38 3.6 .90 .75 3.4 1.03 .83 8. Valuation of previous knowledge 39, 40, 41 3.4 .96 .79 3.3 .97 .85 9. Flexibility 42, 44, 46 3.4 .66 .77 3.2 .68 .80 10. Feedback 50, 51, 52, 53 3.7 .76 .79 3.6 .97 .89 a Elementary school students, n = 34, age range 10 – 13 years, age median 12 years. b Elementary school students, n = 74, age range 10 – 13 years, age median 12 years. Note. r = A reversed item, i.e. values in 1 to 5 Likert scale change as follows: 1 -> 5, 2 -> 4, etc. See Appendix for the item labels. Discussion The first drawback of the pedagogical usability criteria presented here is their generalisability to other domains. In this study, generalisability is limited by various factors, for example, the small sample size, the narrow age range of respondents’ and the limited number of learning material evaluated. To address these issues, we are currently conducting empirical studies that aim at evaluating more representative set of learning material in different domains targeted for both adolescent and adult learners. The criteria of pedagogical usability will in the future be complemented by a segment evaluating “mobile usability” (Syvänen, Nokelainen, Ahonen & Turunen, 2003). The second drawback, although in a more debatable sense, is the methodology we have applied to measure usability, that is, a self-report questionnaire aiming to measure subjective end-user satisfaction. According to researchers specialising in the measurement of usability (Rubin, 1994; Nielsen, 1993; Kirakowski, 2003), the use of questionnaires is normally justified in usability research as part of a test arrangement where a user answers a pre-test questionnaire, uses the application that is to be evaluated and then answers a post-test questionnaire evaluating the usability. However, Kirakowski (2003) sees self-evaluation questionnaires as particularly appropriate for measuring subjective matters (e.g., the enjoyability of use) in usability research. We all agree, that various other test arrangements (Rubin, 1994; Nielsen, 1993, 192-200) are better for gathering factual data (e.g., time spent in doing a particular task). On the other hand, the criteria developed here can also be used as the basis for heuristic evaluation (Nielsen & Molich, 1990). The third drawback is that the criteria do not directly address the issue of cultural sensitivity (Reeves, 1994). Authority (e.g., who is ‘permitted’ to ask questions or ‘obligated’ to lead group work on the basis of his/her social status) and collaboration (e.g., are questions regarding the peer-learners presentation or evaluation of other learners’ work allowed) are examples of cultural issues that are strongly connected to pedagogics. The problem of describing cultural sensitivity via pedagogical criteria is two-fold. First, it is to some extent connected with the metadata (the definition of creator, ideal target group, religion, language and country) and technical usability (graphical user interface colours and symbols) issues. Second, cultural sensitivity is very strongly latent variable by nature, and hence it is not trivially operationalised into questions that end-user is able to answer. Finally, if we try to verbalise what the pedagogical usability criteria aim to address, we may ask: “Does the system, and/or learning material it contains, make it possible for the student and the teacher to achieve their goals?” The important thing to acknowledge is that the dimensions of technical and pedagogical usability (operationalised dimensions) might correlate and thus describe (i.e., in factor analysis language “load on”) both usability and utility (latent dimensions). From the perspective of the user/learner, the number of things to be memorised and the feedback required are examples of these kinds of common components between the user and the system, and between the user and the contents of the learning material. From the viewpoint of usability, the system works well in user terms when the user does not have to resort to any help and is able to control the program, and not the other way around. The number of things to be memorised is, in terms of utility, related to the functioning of the learning material because this will be minimized if the studied subject matter is presented in manageable sections that make sense to the users. There may be a certain degree of tension between some of the criteria. For example, usability is enhanced if the number of things to be remembered and errors are kept to a minimum, but for the utility of the learning material it is desirable for the user to remember the core set of functions so that they can be used readily without prompting. Another example is authenticity and usability. Using real representations from the outside world in the user interface may not be as efficient as a metaphoric 189 design. In the design of learning materials, the aim is often to correspond closely with the real world so that it will be easier for users to link new knowledge to earlier knowledge structures. On the other hand, learning materials intended to inculcate a certain skill by repetition can be weak in their real-world correspondence but still produce good learning results, particularly if the skill being learned is not derived directly from the real world, either. The criteria described here are not the first attempt to provide an analysis tool for the evaluation of the usability of digital learning material; both its structure and content are to great extend based on previous research. However, the criteria and research that led to its creation, have the following original features that bring added value for research on the pedagogical usability of digital learning material: 1) a multidisciplinary approach involving the fields of both computer technology and education, 2) a review of previous research, 3) the development of the criteria on the basis of systematic empirical research, performed with the help of real users; and 4) a multiple choice inventory that was developed on the basis of the criteria and can be used independently or as part of the eValuator computer application. A preliminary version of the eValuator online software is available for educational and research purposes at http://evaluator.hamk.fi. Acknowledgements This study is part of Digital Learning research project (http://dll.hamk.fi/dl2/en/) financed by National Technology Agency in Finland (http://www.tekes.fi/eng/), Research Centre for Vocational Education, University of Tampere, Finland (http://rcve.uta.fi/), and Hame Polytechnic, Finland (http://www.hamk.fi/in-english/). Author wishes to thank Mikko Horila, Miikka Miettinen, Dr. Jorma Saarinen, Antti Syvänen, Tuomo Tammi and Jan Överlund for their contribution to this project. Dr. Bruce Beairsto’s comments helped to improve the quality of this manuscript. References Albion, P. (1999). Heuristic evaluation of educational multimedia: From theory to practice. Paper presented at the 16th Annual conference of the Australasian Society for Computers in Learning in Tertiary Education, retrieved November 24, 2005 from http://www.usq.edu.au/users/albion/papers/ascilite99.html. Barab, S. A., & Duffy, T. M. (2000). From Practice Fields to Communities of Practice. In Jonassen, D. H. & Land, S. M. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum Associates, 25-56. Chalmers, P. (2003). The role of cognitive theory in human -computer interface. Computers in Human Behavior, 19, 593-607. Chin, J., Diehl, V., & Norman, K. (1988). Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. Proceedings of ACM CHI'88 Conference on Human Factors in Computing Systems, New York: ACM Press, 213-218. Collis, B., & Moonen, J. (2002). Flexible Learning in a Digital World. Experiences and expectations, London: Kogan Page. Forman, E. A., & Cazden, C. B. (1985). Exploring Vygotskian perspectives in education: The cognitive value of peer interaction. In Wertsch, J. (Ed.), Culture, communication, and cognition: Vygotskian perspectives, Cambridge: Cambridge University Press, 323-347. Hannafin, M., & Peck, K. (1988). The Design, Development and Evaluation of Instructional Software, New York: Macmillan. Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Criteria for the pedagogical usability, version 1.0, Hämeenlinna, Finland: Häme Polytechnic and University of Tampere. Jansen, W., van den Hooven, H., Jägers, H., & Steenbakkers, G. (2002). The Added Value of E-learning. Paper presented at the Informing Science & IT Education, June 19-21, 2002, Cork, Ireland. 190 Jonassen, D. H. (1995). Supporting Communities of Learners with Technology: a Vision for Integrating Technology with Learning in Schools. Educational Technology, 35 (4), 60-63. Jonassen, D. H., Myers, J., & McKillop, A. (1996). From Constructivism to Constructionism: Learning with Hypermedia/Multimedia Rather Than from It. In Wilson, B. (Ed.), Constructivist Learning Environments, Englewood Cliffs, NJ, USA: Educational Technology Publishers, 93-106. Jonassen, D. H., Peck, K., & Wilson, B. (1999). Learning with Technology. A Constructivist Perspective, Upper Saddle River, NJ, USA: Merrill. Keinonen, T. (1998). One-dimensional usability – Influence of usability on consumers’ product preference, Saarijärvi, Finland: Gummerus. Kirakowski, J. (2003). Questionnaires in Usability Engineering, retrieved November 24, 2005 from: http://www.ucc.ie/hfrg/resources/qfaq1.html. Kolodner, J., & Guzdial, M. (2000). Theory and Practice of Case-Based Learning Aids. In Jonassen, D. H. & Land, S. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum, 215–242. Kurhila, J., Miettinen, M., Nokelainen, P., & Tirri, H. (2002). Use of Social Navigation Features in Collaborative E-Learning. Paper presented at the E-Learn 2002 Conference, retrieved May 1, 2006 from http://cosco.hiit.fi/ edutech/publications/elearn2002.pdf. Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation, New York, USA: Cambridge University Press. Leflore, D. (2000). Theory Supporting Design Guidelines for Web-Based Instruction. In Abbey, B. (Ed.), Instructional and Cognitive Impacts of Web-Based Education, Hershey: Idea Group Publishing, 102-117. Lin, H., Choong, Y., & Salvendy, G. (1997). Proposed Index of Usability: A Method for Comparing the Relative Usability of Different Software Systems. Behaviour and Information Technology, 16 (4/5), 267-278. Littleton, K., & Häkkinen, P. (1999). Learning Together: Understanding the Processes of Computer-Based Collaborative Learning. In Dillenbourg, P. (Ed.), Collaborative Learning. Cognitive and Computational Approaches, Amsterdam: Pergamon, 20-30. Livingstone, D. W. (2000). Exploring the Icebergs of Adult Learning: Findings of the First Canadian Survey of Informal Learning Practices, NALL Working Paper #10-2000, Ontario, Canada: Ontario Institute for Studies in Education, University of Toronto. Mayes, J., & Fowler, C. (1999). Learning technology and usability: a framework for understanding courseware. Interacting with Computers, 11, 485-497. Miller, G. (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 63, 81-97, retrieved November 24, 2005 from: http://www.well.com/user/smalin/miller.html. Myllymäki, P., Silander, T., Tirri, H., & Uronen, P. (2002). B-Course: A Web-Based Tool for Bayesian and Causal Data Analysis. International Journal on Artificial Intelligence Tools, 11 (3), 369-387. Nielsen, J. (1990). Evaluating Hypertext Usability. In Jonassen, D. H. & Mandl, H. (Eds.), Designing Hypermedia for Learning, Berlin: Springer-Verlag, 147-168. Nielsen, J. (1993). Usability Engineering, Boston: Academic press. Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the ACM CHI'90 Conference, New York, USA: ACM Press, 249-256. 191 Nokelainen, P. (2004a). Designing an Evaluation Tool for Digital Learning Materials. Paper presented at the Society for Information Technology and Teacher Education International Conference, March 1-6, 2004, Atlanta, USA. Nokelainen, P. (2004b). Conceptual Definition of the Technical and Pedagogical Usability Criteria for Digital Learning Material. Paper presented at the EdMedia 2004 Conference, June 21-26, 2004, Lugano, Switzerland. Nokelainen, P., & Ruohotie, P. (2004). Empirical Validation of Abilities for Computer Assisted Learning Questionnaire. In Marsh, H. W., Baumert, J., Richards, G. E. & Trautwein, U. (Eds.), Proceedings of the 3rd International Self-Concept Research Conference, Sydney: University of Western Sydney, Self Research Centre, 676-689. Nokelainen, P., & Tirri, H. (2004). Bayesian Methods that Optimize Cross-cultural Data Analysis. In Campbell, J. R., Tirri, K., Ruohotie, P. & Walberg, H. (Eds.) Cross-cultural Research: Basic Issues, Dilemmas, and Strategies, Hämeenlinna, Finland: RCVE, 141-158. Palloff, R. M., & Pratt, K. (2005). Collaborating Online. Learning Together in Community, San Francisco, CA, USA: Jossey-Bass. Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction Design: Beyond Human-Computer Interaction, New York, NY, USA: John Wiley & Sons. Quinn, C. (1996). Pragmatic Evaluation: Lessons from Usability, retrieved November 24, 2005 from: http://www.ascilite.org.au/conferences/adelaide96/papers/18.html Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In Wild, M. & Kirkpatrick, D. (Eds.) Computer education: New Perspectives, Perth, Australia: MASTEC, 219-246. Rubin, J. (1994). Handbook of Usability Testing: How to plan, design, and conduct effective tests, New York, USA: John Wiley & Sons. Ruohotie, P. (1996). Professional Growth and Development. In Leithwood, K. (Ed.), International Handbook of Educational Leadership and Administration, Dordrecht: Kluwer Academic Publishers, 419-445. Ruohotie, P., & Nokelainen, P. (2003). Practical Considerations of Motivation and Computer-supported Collaborative Learning. In Varis, T., Utsumi, T. & Klemm, W. R. (Eds.), Global Peace Through The Global University System, Hämeenlinna, Finland: RCVE, 226-236. Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35 (5), 31-7. Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer Interaction (3rd Ed.), Menlo Park, CA, USA. Addison Wesley. Squires D., & Preece, J. (1996). Usability and Learning: Evaluating the Potential of Educational Software. Computers & Education, 27 (1), 15-22. Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11, 467-483. Syvänen, A., Nokelainen, P., Ahonen, M., & Turunen, H. (2003). Approaches to Assessing Mobile Learning Components. Paper presented at 10th Biennial Conference of the European Association for Research on Learning and Instruction, August 26-30, 2003, Padova, Italy. Tognazzini, B. (2003). First principles, http://www.asktog.com/basics/firstPrinciples.html. retrieved November 24, 2005 from: 192 Tudge, J., & Rogoff, B. (1989). Peer influences on cognitive development: Piagetian and Vygotskian perspectives. In Bornstein, M. H. & Bruner, J. H. (Eds.), Interaction in human development, Hillsdale, NJ, USA: Lawrence Erlbaum, 17-40. Vygotsky, L. S. (1978). Mind in Society. The Development of Higher Psychological Processes, London, England: Harvard University Press. Wilson, B., & Myers, K. (2000). Situated Cognition in Theoretical and Practical Context. In Jonassen, D. H. & Land, S. (Eds.), Theoretical Foundations of Learning Environments, Mahwah, NJ, USA: Lawrence Erlbaum Associates, 57-88. 193 Appendix 1 The pedagogical usability items for 4 to 6th grade elementary school students 1. When I worked on this assignment, I felt that I, not the program, held the responsibility for my own learning. (Definition: I’m not repeating same kind of steps all the time in my studying, but the assignments make me think and make a different solution for each one.) 2. When I used this learning material, I felt that I controlled what it did and not vice versa. (Definition: The program does not lead me from one step to another, but I can choose by myself in which order I want to finish my tasks.) 3. I have to think and make my own solutions to learn this learning material. (Definition: I have to concentrate on this material, I cannot complete the tasks simply with role learning.) 4. This learning material has been divided into sections, my task is to learn them in pre-defined order (and possibly respond to assignments). 5. This learning material provides learning problems without a pre-defined model for the solution. 6. This material does not have material in itself, but links to various other sources, which I have to use to learn. (Definition: If the topic is ”a dumb yard”, there is no ready-made presentation of the topic. The learning starts, for example, with a short tip by the teacher, which tells what kinds of things are needed in order to build a dumb yard. Some of the information that you need may be in the system, but you have to find most of it from newspapers, books or the Internet in order to make your own presentation.) 7. I get so deep into this learning material that I forget all about what is happening around me and how much time I spend on it. 8. When I work with this learning material, I feel that I know more about some topics than others, I’m ”an expert”. (Definition: The learning material may involve an individual information gathering task, for example, an interview of neighbours or measuring the depth of packed snow in one’s home garden over the period of one month.) 9. When I work in this learning material, I (or us, if a group work) have to find out own solutions without the teacher’s or the program’s model solutions. 10. I am proud of my own solution, or one that I made with others, to the problem presented in the learning material. (Definition: I feel that I, or we together, have made something that is significant.) 11. This learning material lets me talk with my classmates. (Definition: For example, messages in chat or notice board.) 12. I can do group work with my classmates in this learning material. (Definition: If I wanted, I could do assignments together with my classmate so that we both used our own computers.) 13. It is pleasant to use the learning material with another student on the same computer. 14. This learning material lets me know what other users have been doing in the system. (Definition: For example, which learning materials have been read the most or assignment that have been the most popular.) 15. This learning material lets me know what other users are doing when I’m using the system. (Definition: For example, the most read material at the moment or the assignment with which most people are working on.) 16. This learning material offers me simple utility programs (for example, a calculator). 17. This learning material offers me versatile utility programs (for example, Excel sheets, a HTML editor, text processor, etc.). 18. In this learning material the utility programs have a central role. (Definition: I have to, for example, edit an Excel sheet to solve a problem.) 19. I can save my work on this learning material and use or evaluate others’ work. (Definition: I can, for example, explore or evaluate other groups’ group works and use them in my own studies.) 20. This learning material tells me clearly what I’m expected to know (or learn) after I’ve used it. (Definition: The learning goals are clearly set, for example, ”After this assignment, you will know how to divide with decimal fraction” or ”After this assignment you can form interrogative clauses in English”.) 21. This learning material tells me clearly why it is useful for me to learn this material. (Definition: The learning goals are justified, for example, ”This assignment will help you to make interrogative phrases in English.”) 22. The learning material assesses my achievements with scores. (Definition: For example, the system gives a score at the and of an assignment and shows the maximum score.) 23. This learning material tells me how much progress I have made in my studies. (Definition: I know what I have practiced or learned thus far.) 24. This learning material is strictly limited. (Definition: For example, the topic of a math learning material is “Calculating the mean”.) 25. This learning material teaches me skills that I will need. (Definition: I will be able to, for example, convert euros into crowns or marks, or help my parents to choose between different-sized packages according to their prize difference.) 194 26. I feel that I will be able to use the skills and knowledge this learning material has taught me in the future. 27. This learning material is based on the idea that ”one learns the best by doing stuff by oneself”. (Definition: The material offers more assignments than, for example, PowerPoint presentations.) 28. I feel that this learning material will help me to do better in the test. (Definition: I think the assignments in the material are similar to the assignment that we usually have in tests.) 29. This learning material is suitably challenging to me. (Definition: The assignments are not too easy or too hard.) 30. I feel that this learning material has been designed for me. (Definition: The material suits your own needs, and it does not feel that you are considered too smart or too dumb.) 31. This learning material adjusts the difficulty to suit my skills. (Definition: I can practice something that is hard for me until I have learned it and before I move on to the next topic.) 32. The images in this learning material help me to learn. 33. The sounds in this learning material help me to learn. 34. The animations in this learning material help me to learn. 35. It is more useful to me to learn topics with this learning material than with conventional methods in a classroom. (Definition: Think if you would be more willing to do this assignment with a computer or with a normal study book or exercise book.) 36. I try to achieve as high a score as I can in this learning material. 37. I want to learn the topics of this learning material as deeply as I can. 38. I’m interested in the topic of this learning material. 39. This learning material required me to know something that had been taught in some other learning material. (Definition: This material made a reference to some other learning material.) 40. I can use my earlier knowledge when I study with this material. 41. This learning material goes over earlier material before starting to teach a new topic. (Definition: For example, in mathematics, the material first goes over simpler calculations that are needed to learn a more difficult topic.) 42. This learning material offers optional routes for my progress. (Definition: I can choose different assignments each time I use the system.) 43. This learning material does not let me proceed to the next point before I have answered correctly to every question. (Definition: For example, in an English language assignment one has to answer correctly to every question, even with the help of the program, before it lets you proceed to the next topic.) 44. This learning material has many similar, consecutive assignments. (Definition: For example, an English fillin assignment that has many consecutive assignments for am, are, and is sentences.) 45. This learning material makes it quick and easy for me to learn a new topic or recap an earlier topic. 46. If I cannot remember a particular word or concept while using this learning material, I can go back and check its meaning in previous material. 47. When I used this learning material, I felt that I had to remember too many things at the same time. (Definition: I felt at some points that I should have used a paper to write some things down.) 48. This learning material presents information in a format that makes it easy to learn. (Definition: Information is presented in meaningful, interconnected entities, and not in separate pieces that are hard to understand.) 49. This learning material presents new material (or recaps old) in ”portions” suitable for me. (Definition: There are not too many new things presented at once, I have time to learn them before moving onto the next topic.) 50. I can make a certain number of mistakes with this material (for example, wrong answers to calculus tasks), after which the program shows me the correct answer. 51. When I make a wrong solution in an assignment, the program gives me a friendly note. 52. This learning material gives me motivating feedback. (Definition: I am willing to try out the less used functions in the learning material, because I know that the system will give me all the advice that I need.) 53. This learning material provides me with immediate feedback of my activities. (Definition: When I write my response to a calculus task, the system shows me immediately whether the answer is correct or not.) 54. This learning material gives first an example of the correct solution. (Definition: Multiplying with decimal fractions is started with a model performance, after which I will calculate on my own.) 55. In this learning material, I get to carry the responsibility for the solution of an assignment in small portions. (Definition: For example, in a math task, I will be first shown the task and then the result. Next, I see the task but not the result, which I have to solve on my own.) 56. I think I learn more quickly with this material than normally. (Definition: This learning material provides me with the right kind of support when I need it.) 195 Appendix 2 The pedagogical usability subdimensions Note. Related items are presented in parenthesis (see Appendix 1). r = rewersed, i.e., negatively worded, item. 1. Learner control 1.1 Minimum memory load (47) 1.2 Meaningful encoding (48,49) 1.3 Take responsibility for one’s own learning (1) 1.4 User control (2) 8.2 Elaboration (40) 2. Learner activity 2.1 Reflective thinking (3, 4r) 2.2 Problem-based learning (5, 9) 2.3 Use of primary data sources (6) 2.4 Immersion (7) 2.5 Ownership (8, 10) 2.6 Primary data source (for PBL) (Items for teacher only) 2.7 Facilitative teacher (Items for teacher only) 2.8 Didactic teacher (Items for teacher only) 2.9 Individual/distance learning (Items for teacher only) 3. Cooperative/Collaborative learning 3.1 Support for conversation and dialog (11) 3.2 Group work (12,13) 3.3 Asynchronous social navigation (14) 3.4 Synchronous social navigation (15) 3.5 Asynchronous social navigation monitoring (Items for teacher only) 3.6 Synchronous social navigation monitoring (Items for teacher only) 3.7 Tertiary courseware (19) 4. Goal orientation 4.1 Explicit goals (20) 4.2 Usefulness of goals (21) 4.3 Focus on results (22) 4.4 Focused goals (34) 4.5 Monitor one’s own studies (pedagogic feedback) (23) 4.6 Set one’s own goals (Items for teacher only) 5. Applicability 5.1 Authentic material (25,28r) 5.2 Perceived usefulness (26) 5.3 Learning by doing (27) 5.4 Adequate material for the learners needs (human development) (29) 5.5 Pretesting and diagnostics (30,31) 5.6 Prompting (54) 5.7 Fading (55) 5.8 Scaffolding (56) 1.2 Meaningful encoding (40,48,49) 6. Added value 6.1 Overall added value for learning (35) 6.2 Effectiveness for learning (45) 6.3 Added value of pictures (32) 6.4 Added value of sounds (33) 6.5 Added value of animations (34) 196 7. Motivation 7.1 Intrinsic goal orientation (37) 7.2 Extrinsic goal orientation (36) 7.3 Meaningfulness of studies (38) 2.4 Immersion (7) 8. Valuation of previous knowledge 8.1 Prerequisites (39) 8.2 Elaboration (40) 8.3 Examples (41) 9. Flexibility 9.1 Pretesting and diagnostics (42) 9.2 Task decomposition (43r,46) 9.3 Repetitive tasks (44) 10. Feedback 10.1 Encouraging feedback (52,51) 10.2 Accurate feedback (53) 10.3 Errorless learning (50) 197 Zualkernan, I. A. (2006). A framework and a methodology for developing authentic constructivist e-Learning environments. Educational Technology & Society, 9 (2), 198-212. A framework and a methodology for developing authentic constructivist eLearning environments Imran A. Zualkernan School of Engineering, American University of Sharjah, P.O. Box 26666, Sharjah, UAE Tel: +971-06-515-2953 izualkernan@ausharjah.edu ABSTRACT Semantically rich domains require operative knowledge to solve complex problems in real-world settings. These domains provide an ideal environment for developing authentic constructivist e-learning environments. In this paper we present a framework and a methodology for developing authentic learning environments for such domains. The framework is based on an ecological view and characterizes dimensions of a typical constructivist environment in terms of pedagogical design, architecture, the environmental context and what is actually learned. A case-study illustrating the use of the framework to develop a just-in-time game-based learning environment is also presented. Keywords Computer assisted instruction, Expert problem solving, Operative knowledge, Development methodology Introduction This paper focuses on learning that occurs in professional problem solving domains that require a very high level of skill. In some sense, these domains are characterized by knowledge that is “operative” as the professionals in these domains are required to do work in real settings. This paper presents a framework that serves as the foundation for conceptualizing the development of authentic constructivist environments in such domains. The constructivist view of learning has its foundations in Piaget (1975) who believed that learning is not transmitted passively, but attained through well-defined stages by active participation of a learner. Vygotsky (1980) presented similar ideas but focused on the importance of socio-cultural activity in learning in addition to introducing flexible stages of development. More recently, the importance of context and “authenticity” in learning has been emphasized by Brown, Collins and Duguid (1989). According to them, “authentic activity is the ordinary practices of cultures.” (p. 36). Lave and Wenger (1991) further extend this view in their influential work on situated learning to point out that, “…Learning occurs through centripetal participation in the learning curriculum of the ambient community” (p. 100). Where the learning curriculum consists of “…situated opportunities (thus, including exemplars of various sorts often thought of as “goals”)” (p. 97). For the purpose of this paper, constructivist authentic learning environments are defined as those learning environments whose design is consistent with the principles of the more recent constructivist tradition on how people learn as exemplified by the works of Lave and Wenger (1991) and Brown et al. (1989). As Herrington and Oliver (2000) point out, such learning environments typically provide authentic contexts and activities, access to expert performances, and support multiple roles and perspectives. In addition, such environments also support collaborative construction of knowledge and promote reflection and articulation. Finally, such environments may include coaching and scaffolding by the teacher and provide for authentic assessment of learning within tasks. Constructivist learning environments are ideally suited to teaching professional problem solving. However, the complexity and ill-structure of such problem solving activity raises particular issues when it comes to actually building a constructivist e-learning environment. For example, what one means by an “authentic context” or an “authentic activity” in a particular problem solving context is often unclear. Prescriptive guidelines (Herrington & Oliver, 2000) like using “activities which have real-world relevance” or “ill-defined activities” or “a single complex” task to define an “authentic activity” attack such difficult questions at a syntactic level and as such provide only a good starting point. Constructing authentic learning environments in professional problem solving domains requires an analysis at the semantic level based on a deeper understanding of what constitutes learning in these environments. This paper presents a framework and a methodology specifically developed for such complex domains. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 198 Framework The key construct of this framework is the concept of a “learning curriculum.” A learning curriculum consists of situated opportunities (Lave & Wenger, 1991). The principles emerging from this framework will be later used to prescribe the general parameters that govern the construction of authentic e-learning environments. In addition, a methodology based on these principles will also be presented. The primary components of the framework used to describe a learning curriculum are derived from (Johnson, Kochevar & Zualkernan, 1992a) as shown in Figure 1. Briefly, the “Physical Environment” is a description of the objectively observable characteristics (e.g., a disease or defect). The available information part of the physical environment may also consist of artifacts such as books, manuals, databases that exist as well as interaction with peers, experts and teachers. Specific characteristics of the environment require specific actions by the learner (e.g., diagnosis or repair). A task can only be performed by a learner because the information in the task is lawfully related to some physical occurrence (Turvey, Carello & Kim, 1981). The “Task Environment” is the subset of the physical environment that is relevant to a class of agents (e.g., surgeons). The adaptation is the primary construct in this framework and represents what is “learned” under the constraints of the task environment and the constraints from the learner. The constraints on the learner may contain cognitive constraints (e.g., short-term memory, processing capabilities (Anderson, 1990)) and learning styles (e.g., holistic, analytical, field independent vs. field dependent (Rumetshofer & Woss, 2003)) or based on theory of multiple intelligences (see Gardiner (1993), for example) on one side and goals and motivation on the other. Figure 1. A framework for developing authentic constructivist learning environments The first key construct in the framework is the “adaptation” itself. Adaptation is a construct that develops under the constraints of the task environment and the learner (this is similar to Simon’s notion of adaptation as the interface between the outer and inner environment (Simon, 1996)). Adaptation, however, does not exist inside the “head” or “mind” of the learner. It is a construct that represents what evolves as a set of routines (including asking for, and retrieving information, for example) or dynamics that allow the learner to be “fit” for the particular task environment. The second key construct of the framework is the concept of “fit” (this is not the same as the evolutionary biology’s notion of a fit, but it describes a psychological fit). Loosely, the fit describes how well a learner is adapted to the task environment (e.g., how good is the surgeon?). Fit can be classified into two dimensions; semantic and structural. The semantic dimension is a measure of how well the learner’s actions are acceptable in the particular environment (e.g., how well are surgeon’s operations received in the physical world? Or, how many patients actually die under her care?). Hence the semantic fit is primarily related to how well the goals and the intentions of the learner are realized in the actions he/she takes in the physical environment. 199 The structural dimension of fit describes how closely the cognitive constraints and learning styles of the learner “match” to the information present in the environment (e.g., does the surgeon accept a particular type of surgery as suited to her/his skills?). Manifestation of failure of structural fit occurs when, for example, an individual refuses to accept the information provided in the environment as “valid” for their task environment. Authentic Learning Environments Authentic learning environments in the constructivist tradition are situations that allow a learner to create their own personal knowledge in a particular task environment. In simplified manifestation, an authentic learning environment is a surrogate to the actual problem-solving environment (e.g., surgery room as opposed to the surgery simulator). An authentic learning environment can more generally be described as a manifestation of a “learning curriculum” as described by Lave and Wenger (1991). In terms of the framework, the learning curriculum, then, is simply a set of situated opportunities that allow the adaptation to eventually attain a high degree of fit between the task environment and the learner. The design of a good authentic learning environment, therefore, consists of creation of an appropriate set of situated opportunities. Each situated opportunity is described by 4-tuple (I, A, C, G) where I: Information in the environment A: Successful actions in the environment C: Cognitive constraints and learning styles of the learner G: Goals and intentions of the learner A successful authentic environment has to create enough (and the right) situated opportunities to ensure that adaptation that arises for a specific learner has both a high structural as well as a semantic fit. A fundamental problem that arises with using authentic environments is their validity. How does, for example, one ensure that adaptation thus evolved within the authentic environment will in fact transfer to the real environment? The key problem that needs to be solved in developing an authentic constructivist environment, therefore, is to ensure that adaptation that emerges in response to the “fit” requirements (both semantic and structural) of the constructed authentic environment, also establishes a high degree of fit in the real environment. This is exactly what the framework and the resulting methodology helps achieve in a specific situation. Methodology This section presents the various dimensions of a typical authentic learning environment and shows how the framework presented earlier provides conceptual footing for a development methodology based on a) pedagogical design, b) architectures, c) the environmental context and d) what actually gets learned in such environments. These four dimensions culminate in a four step methodology where each step is tied to exploring one dimension. Step 1: Complete pedagogical design The primary objective of this step is to determine what constitutes a “situated opportunity” (i.e., I, A, C and G according to the framework) within a particular pedagogical design paradigm. A pedagogical design imposes broad constraints on what a situated opportunity can be. In doing so, it outlines the possible space of situated opportunities for the learning environment. In addition, based on the properties of the semantic and structural fit in the real environment, pedagogical design also determines constraints on the four components of a typical situated opportunity. Some commonly used pedagogical design paradigms for “authentic” learning environments in the constructivist tradition are given below (Ip & Naidu, 2002; Oliver, 2001) 200 ¾ ¾ ¾ ¾ ¾ ¾ ¾ Problem-based Learning (PBL) - In PBL, a convincing scenario problem (scenario) is created where learners are supported by stories as told by various actors. The primary premise behind these environments is to allow the learners to fail in a “safe” context and to receive feedback as a third-person. Distributed Problem-Based Learning (DPBS) brings the additional element of a group of individuals using the network as a medium to work on and solve a common problem. Inquiry-based learning (IBL) is one variant of PBL that poses ill-structured tasks to the students. Role-Play Simulation and Game-based Learning (RSL) creates situations where learners take on the role-profiles of various characters in contrived educational games. Case Studies-based Learning (CSBL) uses actual events to force students to “practice” on actual data in a safe environment. Critical Incidence-based Learning (CIBL) occurs when learners engage in reflections on critical events from their workplace. Project-based Learning (PRBL) engages students in designing and creating products that meet authentic needs. No matter what the manifestation of the pedagogical design of an authentic environment, each has to pay particular attention to how and why the situated opportunities thus created are authentic. For example, in PBL, a problem consists of the information presented in the environment (I) and the feedback provided by the stories guides the learner on what is successful action (A) through failure. The problems have to be consistent with the goals of the learner (G) as well as the cognitive constraints (C). The DBPS simply adds additional sources of information (I) that learner can access. Critical Incidence-based Learning is particularly interesting in this context in that it is related to low-base rate tasks (Johnson, Grazioli, Jamal & Zualkernan, 1992b). That is, environments where the incidence of situated opportunities is very rare (e.g., earth quakes). In this case, the problem to be solved becomes mostly the generation of an appropriate number of situated opportunities, so that adaptation can attain a high degree of fit. Similarly, in CSBL, the emphasis is not so much on creating the right information (I). The appropriateness of actions (A) is also not a primary issue. The emphasis really has to be on how well the fit can occur between the learner’s cognitive constraints and learning styles as well as the goals. Step 2: Construct architecture for the authentic environment The primary objective of this step is to specify an architecture that provides appropriate support for the situated opportunities outlined in the previous step. The architecture of an authentic environment specifies the various components that must exist in a learning environment or a computer manifestation of it. A general characterization of the constructivist learning environments has been provided by Jonassen, Peck and Wilson (1998). The components needed for such environments are: ¾ ¾ ¾ ¾ ¾ ¾ Problem/project space. The learners are presented with an interesting, relevant and engaging problem. This is simply the creation of a single situated opportunity. Related Cases. When expecting learners to solve problems, they must be provided with a set of related experiences on which the learner’s can draw. These represent a set of situated opportunities similar to the one being presented. Information Sources. Providing learners with information they need help with in a timely manner. This simply stresses the information component (I) of the situated opportunity. Cognitive (knowledge) construction tools. Tools that support the learner’s abilities to solve the tasks at hand. These are a part of the physical environment (e.g., a paper and a pencil, a calculator, a utility program) if the fit determines the learner’s cognitive constraints (C) need to be augmented to achieve an appropriate fit. Conversation (knowledge-negotiation) tools. Tools to support collaboration. These are a part of the physical environment, if accessing information (I) or successful action (A) requires external conversations. Social/contextual support. Physical, organizational, political and cultural aspects of the environment. This can be primarily related to the motivation and goals (G) of the learner. 201 Step 3: Consider context of learning The primary objective of this step is to ensure that the environmental constraints of the learning environment have been considered for the architecture proposed in the previous step. Authentic learning environments exist in a context. The environmental contexts of an intelligent tutoring system as described by Kinshuk, Opperman and Russel (2001) can also be applied to an authentic learning environment. The context is divided into seven categories. ¾ ¾ ¾ ¾ ¾ ¾ ¾ Student (natural abilities, learning styles and motivation) Peers (interaction with fellow students) Social environment (social values, institutional values, evolution of common metaphors) Teacher (teaching styles, personality attributes) Discipline (homogeneity, operational/conceptual, physical/virtual, teaching traditions, levels) Characteristics of knowledge (operational, causal, contextual) Characteristics of medium (hardware, software and communication capabilities). From the perspective of the framework, these categories roughly map as shown in Figure 2. Figure 2. Relationships between framework and environmental context It is interesting to note that the teacher or the teaching style maps mostly to providing the available information (what it provided to the student), successful action (guiding through assessment) and motivation. Most traditional instructional theories such as Gagne’s nine steps (Gagne, 1985), John Keller’s ARCS model (Keller & Suzuki, 1988) or Merrill’s ITT (Merrill, Li & Jones, 1991) can in fact be used to create these parts of situated opportunities as they rely largely on information (I) and successful action (A) and are generally concerned with how to enable a student to do something (as goals are prescribed by the teacher) Step 4: Clearly specify what is learned? The primary objective of this step is to understand and characterize the nature of adaptation that is expected to emerge from the authentic activities being carried out in the learning environment. One of the critical components of the framework is the nature of the adaptation that occurs as a result of interacting with the information, and carrying out actions in the physical environment. The adaptations that develop will be unique to a particular individual learner. While adaptation is an abstract entity, it can be described. In the context of semantically rich problem solving domains, descriptions of these adaptations can take the form of successful arguments generated by a learner. These arguments can be constructed from five basic types of backings (Johnson, Zualkernan & Tucky, 1993) 202 ¾ ¾ ¾ ¾ ¾ Type 1 – This is based on analytic truths; the interesting property is that the all actions preserve global criteria of rationality such as consistency, soundness and completeness. Type 2 – This is based on empirical judgments where the actions are constrained to consist of consensus among groups of individuals. Type 3 – This is based on complementary representations of a problem and actions are constrained by agreement within complementary analytic representations and empirical judgments. Type 4 – This is based on complementary representations of a problem and the actions are based upon resolving conflicts within these. Type 5 – This is based “systems of knowing” that lead the process such that actions are constrained by selfreflection. The adaptation for a highly skilled individual can be described using a complex combination of these arguments across diverse domains including medicine, chess, experimental design, VLSI manufacturing, and frauddetection (Johnson et al., 1993). Bloom’s original description of the types of learning (Cognitive, Affective and Psychomotor) comes close to serving as an appropriate language for a description of such adaptations (Bloom, 1956). For example, the category of “synthesis” as described by Bloom (1956) consists of building a structure or pattern from diverse elements. The keywords that describe this activity are combines, plans, creates etc. The adaptation for a skilled individual engaged in a semantically rich domain such as statistical experimental design (Johnson et al., 1993), however, contains many types of “synthesis”. Figure 3. A description of an “adaptation” tied to a particular goal For example, the individual constructs an initial conceptualization of the client’s problem, subsequently she constructs a refined quasi-statistical representation (and re-representation) of the client’s problem, and then she constructs an appropriate design type and finally a specific design. The goal of carrying out each one of these “synthesis” steps actually are achieved by paying attention to very different types of information and by using various types of qualitatively different backings. For example, conceptualization of a client’s quasi-statistical representation of the problem is based on a Type 3 backing (see Figure 3) while the construction of an appropriate design type is Type 4. Derivation of a particular design uses Type 1 backing. Hence the argument structure described in (Johnson et al., 1993) presents a refined language to describe specific individual Adaptations for semantically rich domains as opposed to the classification based on keywords (e.g., combines, creates etc.) presented by Bloom (1956) to describe what is learnt. In summary, the methodology applied to a particular instance consists of exploring and refining the four dimensions in the context of the framework presented earlier. A case study is presented next to show how the methodology is applied in the context of game-based learning. 203 Case Study – Just-In-Time Game-Based Learning This section shows how framework and the methodology based on four dimensions of pedagogical design, architecture, environmental context and what is learned was applied to the development of a just-in-time gamebased learning environment. Game-Based Learning Game-based learning is a particular form of incidental learning that occurs while the learner is engaged in an activity that may not be directly tied to the task at hand. Schank and Cleary (1995) provide an example of such learning where in order to teach grade students about geography, the students were asked to play a computer game planning a driving trip to the city where their favorite team is playing. Incidental learning, in general (see Marsick and Watkins (2001), for example), has been shown to be a significant mechanism of learning. A detailed analysis of digital games for education is provided in Prensky (2004) that divides such games into the different categories ranging from facts, skills, judgments etc. Attempts at building game-based learning environments typically have fallen between two extremes. On one extreme are games like Virtual-U (Virtual-U, 2005) that simulates working of a university to teach administrators about strategy. In this instance, the operational rules as well as the constitutive rules (Salen & Zimmerman, 2003) are embedded into the game (operational rules of a game are the surface rules while the constitutive rules represent the deep mathematical structure that underlies the game). The end-user, however, has a limited choice in which set of constitutive rules to use by selecting the scenario to be of a particular type of a university (public vs. a private university, for example). On the other extreme are game-engines (see Crookston (2005), for example) that use the cover-story of an existing game format (corresponding to the operational rules of the game) such as the T.V. game show “who wants to be a millionaire?” or more traditional games like a cross-word puzzle where one simply inserts the syntactic elements of the particular domain in the game. For example, a game teaching biology is generated by only using biological terms in a cross-word puzzle. At the level of constitutive rules, there is no deep logical relationship between the game and the domain of learning. For example, in a cross-word puzzle, the two words “bacteria” and “hormone” may appear together just because they have the letter “e” in common and not because of their biological relationship. The case study presented here applied game-based learning in the highly contextual problem solving domain of software engineering. In this domain, complex software designs are constructed and re-constructed at a very rapid pace. One critical problem of learning in these environments is to be kept updated with the continually changing designs. These environments are particularly suited to automated just-in-time learning because of the rapid rate of change and the immense time and cost involved in constructing any learning aids by-hand. In addition, since these are semantically rich problem solving domains, the generic game engines that mostly capture the operative (or surface) rules of a game (e.g., cross-word puzzle) may not be sufficient to capture the richness of contextual learning that needs to take place. The focus of study presented here was on software engineering design processes that typically produce byproducts like the UML diagrams (Jacobson, Booch & Rumbaugh,1999; OMG, 2005). An activity diagram is a UML diagram that represents a flow of activities (see Figure 8, for a partial example). This section presents an application of the methodology to the problem of constructing a constructivist e-learning environment for building and using activity diagrams in a typical software engineering process. Step 1: Complete pedagogical design The first and the most important step of the methodology is pedagogical design. From the perspective of the framework, each instance of a game-play is a “situated opportunity.” The key to designing a game-based elearning environment is to ensure that the adaptation that emerges as a result of playing the game will also lead to successful actions in the software engineering domain. In order to do this, the goals and motivations of the engineers have to be considered (G). Similarly, successful action in this instance consists of actions that will lead to winning a game (A). In addition, the available information or cues provided in the game (I) should be 204 consistent with the learning styles and cognitive constraints (C) of a typical engineer. The pedagogical design is carried out by considering the properties of the semantic and structural fit to determine what represents an appropriate “situated opportunity” (i.e., I, A, C, G) in the game environment. Semantic fit ties successful action to goals and motivation. Semantic fit, in part, determines the nature of adaptation that will occur. In the case of software engineers using activity diagrams in their normal work environment, an important determinant of successful action is their ability to predict the relationships between the various activities. This is, in fact, one primary reason for an activity diagram to exist; knowing which activity happens before which and under what conditions. For the game to lead to an appropriate adaptation (one that would also lead to successful action in the actual environment), this meant that the successful action needed to be tied to the ability of an engineer to predict the order and conditions of precedence of the various activities in an activity diagram. The game, therefore, had to be designed to ensure that more successful the engineers were at predicting relationships between activities, the greater the degree of their semantic fit. Structural fit ties cognitive constraints to available information in the environment. In addition to a specific activity diagram, information in the environment of a software engineer may consist of domain documents, other colleagues and even computer programs. Therefore, the information (I) provided in the game should act as a surrogate for the information in the real environment such that structural fit is high. Prior studies of software engineer’s design activities in their environment suggest that this information is local in nature (Zualkernan, Tsai, Jemie, Wen & Drake, 1992); due to cognitive constraints, engineers are only able to focus on “local” aspects of an activity diagram. In order to be consistent with this assumption on the structural fit, therefore, it was decided that games should only “encode” local information about an activity diagram. The “locality” of information was implemented by providing limited threads of execution through the activity diagram itself. In order to accommodate multiple learning styles of engineers, it was also decided that multiple game formats should be provided. Step 2: Construct architecture for the authentic environment The second step of the methodology is to establish the architectural requirements for the environment. Once the basic parameters of the particular game were established (i.e., what constitutes a “situated opportunity”), the second dimension, architecture was used to further refine the design of this game-based learning environment. The problem/project space describes the process by which a single “situated opportunity” is created. In gamebased learning, a situated opportunity typically consists of a single play of a game. In this context, because of the nature of the problem (rapidly changing software designs), the process by which a situated opportunity is created was considered at two distinct levels; the generation of a game for an arbitrary activity diagram and various repeated plays of this particular game (Zualkernan & Parmar, 2004). Related cases or ways in which a game was played successfully were also required to be a part of the architecture. Similarly, the architecture also required that information (consisting of the actual activity diagram) was also made available to the engineers. For analysis, cognitive tools such as paper or pencil or automated analysis tools for activity diagrams were also required. In addition, user forums and real-time chat facilities that allow software engineers to converse with fellow software engineers were also made a requirement. Finally, the ability to group engineers to provide a social context built around the playing of these games was also mandated. Step 3: Consider context of learning Contextual analysis is used to ensure that the resulting architecture from the last step does not miss any important part of its context of use. After establishing the basic architecture of this game-based learning environment, the contextual analysis was carried out to further fine-tune the design of the environment. It was recognized, for example, that available information (I) is tied to the student, his peers and the characteristics of knowledge. However, the teacher dimension was missed and needed to be supported as well; therefore, the dimension of a attaching a tutor was also added. Similarly, since goal and motivations are tied to peers and social environments, it was decided to add a peer-to-peer challenge component to the game where an engineer could challenge a fellow engineer with a particular game based on an activity diagram. Similarly, since the game also had to run on mobile phones with fairly small screens, in order to be consistent with the cognitive constraints, the number of activity diagrams appearing on the screen was reduced. 205 Step 4: Clearly specify what is learned? The final and most crucial step of the methodology is to ensure that the adaptation that emerges as a result of interaction with the e-learning environment is consistent with what is required for successful action in the real environment. Since activity diagrams are semi-formal, the primary types of arguments constructed by software engineers in a real environment are Type 1. This means that all their actions need to preserve some global notions of consistency and soundness. In other words, if an engineer takes an action that indicates that activity A (say, Pour Coffee) “happens before” B (say, Drink Coffee), the game needs to apply a formal rule to enforce this. This implies that each game needs a model of the activity diagrams to make these determinations; it is necessary for the game to determine what constitutes successful action for the engineer. In a real design context an engineer may look at an activity diagram where activity A “comes before” B and activity B “comes before” activity C and then take the action that activity A “comes before” activity C. The game must also make the determination that in the presence of the information provided (A “comes before” B and B “comes before” C), A “comes before” C constitutes successful action. The game simply needs to preserve the same notion of “rationality” (Type 1, in this case) that connects information with a successful action. Practically this meant that each game needed to maintain an internal model of the activity diagram. Design Based on the analysis presented above, four different forms of common game types were constructed for this game-based e-learning environment. Each is briefly described below. Tetris The classical game of Tetris consists of a sequence of geometric figures falling down at various rates. The learning version of Tetris was created by mapping the geometric constraints to the precedence relationship between activities (Zualkernan & Parmar, 2004). So rather than matching the geometric constraints, the engineers are asked to match the precedence relationships between falling activities. The information provided in this environment, therefore, was a falling block with some precedence constraints. A player who was able to match these precedence constraints achieves a high-score or a high semantic fit. Figure 4. A Tetris-Like game automatically generated for activity diagrams From the framework perspective, each situated opportunity in the Tetris-like game can be summarized as follows: I: Local precedence constraints between activities represented by falling blocks where each block has different types of precedence relations (see arrows in Figure 4) attached to it. A: Successful action consists of matching “local” precedence constraints (who comes before who?) in a very short amount of time by rotating the falling block. C: A small number of options of precedence constraints are provided and is suited to a visual learning style. G: Match as many precedence constraints as possible in order to prevent the blocks from building up. 206 Snake The game of snake consists of a snake that gets progressively longer and faster (and hence difficult to maneuver) as it eats various bits of “food” that appear. The learning version for this game was created by mapping the “eating” activity to the precedence relation and the length of the snake. If the player picks the wrong precedence relationship, the snake gets longer, picking the right now, however, keeps the snake to be the same length. Therefore, a player who is familiar with the precedence relationship will, in fact, achieve a higher score. From the framework perspective, each situated opportunity in this snake-like game can be summarized as follows: I: Local precedence constraints between two activities represented by the head of the snake and the food (see Food and Head in Figure 5). A: Successful action consists of matching two precedence constraints (who comes before who?) in a very short amount of time by eating the food or waiting until the undesired food (violating the precedence constraint) disappears. C: A small number of options of precedence constraints are provided and is suited to a visual learning style. G: Match as many precedence constraints as possible in order to prevent the snake from getting larger. Figure 5. A Snake-like game automatically generated from activity diagrams Hexxagon The Hexxagon game consists of a field of hexagons, where the player is allowed to move his marbles to a region within green or yellow regions. Green regions multiply the original marbles; while a move to yellow regions allow one to “kill” an opponent’s marbles that happen to be in adjacent hexagons. All the hexagons are initially labeled as activities (see Figure 6). A move to a hexagon whose activity follows that activity in the current hexagon multiplies the number of marbles. This rewards the player for knowing the precedence relationship. Figure 6. A Hexxagon-Like game generated from activity diagram From the framework perspective, each situated opportunity in the Hexxagon game can be summarized as follows: 207 I: Local precedence constraints between two activities represented by the adjacency of hexagons (see hexagons in Figure 6). A: Successful action consists of matching two precedence constraints (who comes before who?) and deciding which of the green or yellow move will result in better positioning. C: A small number of options of precedence constraints are provided and is suited to a visual learning style. G: Match as many precedence constraints as possible in order to kill the opponent’s marbles. Space Invaders The classical game of Space Invaders consists of a number of alien ships coming down from the sky. The objective of the game is to shoot them down. The constitutive rule of this game is simply that enough “bullets” will tear away and make an alien ship disappear. A learning version of this game is formulated by mapping the precedence relation to the sequence in which the activities (that now represent alien ships) are targeted. The activities have to be targeted in the right sequence of precedence to achieve a high score. From the framework perspective, each situated opportunity in the Space Invader-like game can be summarized as follows: I: Local precedence constraints between two activities represented by the alien ships (see Figure 7). A: Successful action consists of matching two precedence constraints (who comes before who?) in a very short amount of time by killing only those aliens that represent activities that precede each other. C: A small number of options of precedence constraints are provided and is suited to a visual learning style. G: Match as many precedence constraints as possible in order to prevent the aliens from descending Figure 7. A Space Invader-Like game generated from activity diagram Implementation All the four game types were implemented using Macromedia Flash (Macromedia, 2005; Neave 2005) and were hosted within a modified version of the open-source Claroline Learning Management System (Claroline, 2005). The learning management system provides the functionality of forums, groups, chats and messaging that can be tied to a particular game. In addition, a manager can assign games to various players. Also included are interaction between the engineer, his peers and a tutor. For example, the engineers can post on forums about tips on how to play a particular type of game better. In addition, a tutor can post exercises so that the engineers can learn the peculiarities of an activity diagram better before playing the game. Since the games thus generated are SCORM-complaint (SCORM, 2005), the score of each student is automatically tracked and delivered to the respective manager. Various resources related to the game including the actual activity diagram used to derive the game is also made available as information (I) that can be readily accessed by a learner while playing the game as shown in Figure 8. The player can pause the game, consult the activity diagram and get back to playing the game. 208 Figure 8. Playing a SCORM-compliant game inside Claroline Finally, to better support the peer-to-peer dimension of learning, the games were also implemented to run on mobile phones supporting J2ME (Sun, 2005) and Flash Lite 1.1 (Macromedia, 2005). This allows players to challenge other players on their own activity diagrams. For example, it is possible for a manager to send a game to an engineer from Claroline (integrated with the Ozeki SMS Server (Ozeki, 2005)) or from another mobile phone using the short messaging services (SMS). This score achieved by the challenged engineer is sent back to the challenger via an SMS message. Discussion Game-based learning was specifically chosen as a case study for presenting this framework because it represents an extreme case. Of all the pedagogical design paradigms of an authentic constructivist environment (i.e., problem-based learning, case-based learning etc.), game-based learning seems the most unobvious. For example, software engineers do not really play games of the type presented in this paper. Therefore, one could argue that since these games are not a part of the “cultural practice” of software engineers, learning environments built around such games are not “authentic.” This view is essentially a simulation stance towards “authentic learning” that assumes that an authentic learning environment ought to be a model or a surrogate of the actual environment and therefore necessarily contain the complexity of the environment. Constructing a complex environment, however, is not necessary for guaranteeing authenticity. For example, citing work habits of apprentice tailors from Lave (1988), Brown et al. (1989) who point out: “This is not to say that authentic activity can only be pursued by experts. Apprentice tailors …, for instance, begin by ironing finished garments (which tacitly teaches them a lot about cutting and sewing). Ironing is simple, valuable, and absolutely authentic.” (p. 36) The framework presented here shows that not only can specific slices of authentic activity (like ironing, for example) be isolated, but the “surface form” of the activity can be modified without sacrificing authenticity. This is essentially what the case study in game-based learning shows. The key point is that the learning environment needs to give rise to an appropriate adaptation under the mutual constraints of information, action, cognitive constraints and goals. And even though the surface activity of game-playing seems like simple repetition, when analyzed via the framework, this activity is authentic because the situated opportunities are semantically similar to what one expects in a real environment of software engineers. Semantically similar because the software engineer has the same goals (determining precedence relationships and higher motivation because game-playing is fun), similar cognitive constraints (locality, in this case) and varying learning styles (catered for by different game types) and finally, successful action in the game (higher score in the game) corresponds to getting an appropriate answer in the real environment. Another observation from the case study is that technology is required to implement much of the social interactions (e.g., peer to peer interaction, sharing etc.) is already readily available in pre-packaged forms and once the methodology is applied, the implementation is relatively straight forward. 209 Finally, the methodology is neutral in that it only supplies broad guidelines. For example, Bloom’s taxonomy (Bloom, 1956) can be used to extend the low-level pedagogical design of the game-based learning environment presented in the case study. The environment currently only supports learning of precedence relationships mostly at “analysis” and “comprehension” level. However, the environment can easily be extended to higher levels like “application”, “analysis” and “synthesis” by simply constructing a different characterization of the situated opportunities. In summary, the case study shows that the framework and the methodology enable a designer to build slices of authentic activities at the semantic level. This methodology, therefore, does not dictate that a massive simulation or reproduction of the real environment be constructed for all authentic learning environments (although, it does not preclude these). This enables a designer to build authentic environments incrementally. In addition, the methodology frees up the designer to employ novel forms (like game-based learning) while retaining authenticity. Conclusion This paper presented a framework and a methodology for analyzing the development of authentic constructivist e-learning environments for semantically rich problem solving domains. The development of an e-learning constructivist environment has been conceptualized in terms of the creation of “situated opportunities.” These opportunities have further been specified in terms of information available, successful action, cognitive constraints/learning styles, and goals and motivation. A methodology based on pedagogical design, architecture, context of learning and an analysis of what was learned when developing such environments was also presented together with a successful application of this methodology to create a rich just-in-time game-based learning environment. Although the framework has currently only been applied in the context of game-based learning, this framework is being used in building other types of constructivist environments such as problem-based learning. One interesting outcome of this exercise has been that most technology required for such environments is readily available as seen by the various types of open-source software used in the game-based e-learning environment. What is required and needed are methodologies and frameworks to guide the development in novel instructional paradigms such as game-based learning. References Anderson, J. (1990). The adaptive structure of thought, Hillsdale, NJ, USA: Erlbaum. Bloom, B. S. (1956). Taxonomy of educational objectives (Handbook 1: The cognitive domain), New York, USA: Addison-Wesley. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32-42. Claroline (2005). Claroline : Open Source e-Learning, retrieved January 09, 2006 from http://www.claroline.net. Crookston (2005). Flash Learning Games, http://itc.umcrookston.edu/index.asp?section=flashgames. retrieved January 09, 2006, from Gagne, R. M. (1985). The conditions of learning and the theory of instruction, Chicago, USA: Harcourt Brace College Publishers. Gardiner, H. E. (1993). Multiple Intelligences: The theory in practice, New York, USA: Basic Books. Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48 (3), 23-48. Ip, A., & Naidu, S. (2002). Experience-based pedagogical designs for e-learning. Education Technology, XLI (5), 53-58. 210 Jacobson, I., Booch, G., & Rumbaugh, J. (1999). The unified software development process, Addison-Wesley. Johnson, P. E., Kochevar, L. K., & Zualkernan, I. A. (1992a). Expertise and fit: Aspects of cognition. In Pick, H. L. Jr., Broek, B. V. & Knill, D. C. (Eds.), Cognition: Conceptual and methodological issues, American Psychological Association, Washington, D.C., USA, 305-331. Johnson, P. E., Grazioli, S., Jamal, K., & Zualkernan, I. A. (1992b). Success and failure in expert reasoning. Organizational Behavior and Human Decision Processes, 53 (2), 173-203. Johnson, P. E., Zualkernan, I. A., & Tucky, D. (1993). Types of Expertise: an invariant in problems solving. International Journal of Man-Machine Studies, 39, 641-665. Jonassen, D. E., Peck, K. L., & Wilson, B. G. (1998). Learning with technology: A constructivist perspective, Prentice-Hall. Keller, J. M., & Suzuki, M. K. (1988). Use of ARCS motivation model in courseware design. In Jonassen, D. E. (Eds.), Instructional design for microcomputer courseware, Hillsdale, NJ, USA:Lawrence-Erlbaum, 34-40. Kinshuk, Patel, A., Oppermann, R., & Russell, D. (2001). Role of human teacher in web-based intelligent tutoring systems. Journal of Distance Learning, 6 (1), 26-35. Lave, J. (1988). The culture of acquisition and the practice of understanding. IRL report 88-00087, Palo Alto, CA: Institute for Research on Learning. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation, Cambridge, England: Cambridge University Press. Macromedia (2005). Adobe Website, retrieved January 09. 2006 from http://www.macromedia.com. Marsick, V. J., & Watkins, K. E. (2001). Informal and incidental learning. In New directions for adult and continuing education, 89 (Spring 2001), Jossey-Bass Publishers, 25-34. Merrill, M. D., Li, Z., & Jones, M. K. (1991). Instructional transaction theory: An introduction. Educational Technology, 31 (6), 7-12. Neave (2005). Neave Games, retrieved October 17, 2005 from http://www.neave.com/games. Oliver, R. (2001). Developing e-learning environments that support knowledge construction in higher education. In Stoney, S. & Burns, J. (Eds.), Working for excellence in the e-economy, Churchlands, Australia: We-B Center, 407-416. OMG (2005). OMG Unified Modeling Language Specification, Version 2.0, retrieved January 09, 2006 from http://www.omg.org/technology/documents/formal/uml. Ozeki (2005). Ozeki Website, retrieved January 09, 2006 from http://www.ozeki.hu. Piaget, J. (1975). The construction of reality in the child, Ballantine Books. Prensky, M. (2004). Digital game-based learning, McGraw-Hill. Rumetshofer, H., & Woss, W. (2003). XML-based adaptation framework for psychological-driven e-learning systems. Education Technology and Society, 6 (4), 18-29. Salen, K., & Zimmerman, E. (2003). Rules of play: Game design fundamentals, MIT Press, Cambridge, Mass, USA. Schank, R., & Cleary, C. (1995). Engines for education, Hillsdale, NJ, USA Lawrence Erlbaum. SCORM (2005). IMS Content Packaging Best Practice Guide Version 1.1.1 Update Specification, retrieved January 9, 2006 from http://www.imsglobal.org/content/packaging/imscp_bestv1p1p1.html. 211 Simon, H. A. (1996). The sciences of the artificial, Cambridge, MA, USA: MIT Press. Sun (2005). Java Platform, Micro Edition (Java ME), retrieved January 9, 2006 from http://java.sun.com/j2me. Turvey, M. T., Carello, C., & Kim, N (1981). Cognition: The view from ecological realism. Cognition, 10 (1-3), 313-321. Virtual-U (2005). Virtual-U Website, retrieved January 9, 2006 from http://www.virtual-u.org. Vygotsky, L. S. (1980). Mind in society: The development of higher psychological processes, Cambridge, MA, USA: Harvard University Press. Zualkernan, I. A., Tsai, W. T., Jemie, I, Wen, I. C., & Drake, J. M. (1992). Object-oriented analysis as design: A case study. International Journal of Software Engineering and Knowledge Engineering, 2 (4), 489-521. Zualkernan, I. A., & Parmar, F. (2004). AGORA - An Engine for Automatic Generation of Game-Based Incidental e-Learning Environments from UML Activity Diagrams. Paper presented at the 2004 International Research Conference on Innovation in Information Technology, October 4-6, 2004, Dubai, UAE. 212 Zayim, N., Yildirim, S. & Saka, O. (2006). Technology Adoption of Medical Faculty in Teaching: Differentiating Factors in Adopter Categories. Educational Technology & Society, 9 (2), 213-222. Technology Adoption of Medical Faculty in Teaching: Differentiating Factors in Adopter Categories Nese Zayim Department of Biostatistics & Medical Informatics, Akdeniz University, Medical School, 07059 Antalya, Turkey Tel: +90 (242) 227 43 43 nzayim@akdeniz.edu.tr Soner Yildirim Department of Computer Education & Instructional Technology, Middle East Technical University Faculty of Education, 06531 Ankara, Turkey Tel: +90 (312) 210 4057 Fax: +90 (312) 210 1112 soner@metu.edu.tr Osman Saka Department of Biostatistics & Medical Informatics, Akdeniz University, Medical School, 07059 Antalya, Turkey Tel: +90 (242) 227 43 43 saka@akdeniz.edu.tr ABSTRACT Despite large investments by higher education institutions in technology for faculty and student use, instructional technology is not being integrated into instruction in higher education institutions including medical education institutions. While the diffusion of instructional technologies has reached a saturation point among early adopters of technology, it has remained limited among the mainstream faculty. This investigation explores instructional technology usage patterns and the characteristics of medical school faculty as well as contributing factors to IT adoption. The focus of the study was to explore the differences between faculty members who have adopted new technology and those reluctant or resistant to IT adoption, and to determine whether faculty characteristics contribute to the prediction of faculty adopter categories. Faculties from the disciplines of basic and clinical science at a state university Faculty of Medicine were surveyed to gather data concerning faculty characteristics, adoption patterns, perceptions of computer-use self efficacy, the value of IT, barriers and incentives to adoption and preferences related to help and support in technology adoption. The data analysis was based on Rogers’ theories of diffusion and adopter categories. Significant differences were found between early adopters and the mainstream faculty in terms of individual characteristics, adoption patterns, perceptions of barriers and technology learning preferences The results indicated that computer use self efficacy and rank significantly contribute to the prediction of faculty adopter group. Keywords Technology adoption, Diffusion of innovation, Adopter categories, Medical faculty technology use Introduction In the past few years, higher education institutions have invested heavily in infrastructure to support the diffusion and adoption of technology (Green, 1999; Jacobsen, 2000). However, despite large investments by higher education institutions in technology for faculty and student use, instructional technology is not being integrated into instruction in higher education institutions including medical education institutions (Geoghegan, 1994; Spotts, 1999; Surry, 1997; Albright, 1996; Carlile & Sefton, 1998). There are many reasons both technical and societal, explaining why innovative technologies have not been widely adopted, however, the major reason for this lack of utilization is that most university-level technology strategies ignore the central role that the faculty plays in the process of change (Surry & Land, 2000). The Association for Educational Communications and Technology (AECT) has defined instructional technology (IT) as a complex, integrated process involving people, procedures, ideas, devices and organizations, for analyzing problems and devising, implementing, evaluating and managing solutions to those problems involved in all aspects of human learning (Seels & Richey, 1994). Despite the AECT definition of IT, in which the emphasis is on IT rather than its’ products, many of the debates regarding the use of technology in education continues to focus on products: computers, software, networks and instructional resources (Green, 2000). ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 213 Certainly, the use of an adequate technology infrastructure is a prerequisite of IT integration, but the major challenge is to encourage the faculty to adopt these technologies once they are made available. Goeghegan (1994) expresses this challenge as follows: One of the most basic reasons underlying the limited use of instructional technology is our failure to recognize and deal with the social and psychological dimension of technological innovation and diffusion: the constellation of academic and professional goals, interest, and needs, technology interest, patterns of work, sources of support, social networks, etc., that play a determining role in faculty willingness to adopt and utilize technology in the classroom. Adoption of or hesitation to adopt new instructional technologies by the faculty involves a complex system involving multiple variables. As stated by Spotts (1999), “…, the reality of instructional technology use is in the relationship between the new instructional technologies and the faculty members’ individual and organizational context and their personal histories” (p. 93-94). Diffusion of Innovations Studies of diffusion and adoption help to explain the what, where, and why of technology acceptance or rejection in education (Holloway, 1996). Therefore, Rogers’ (1995) theory of diffusion of innovations provides a theoretical framework for analyzing faculty technology adoption patterns. Rogers (1995) defines an innovation as an idea, practice or object that is perceived to be “new” by the individual, and diffusion as the process through which an innovation is communicated through certain channels over time among the members of a social system. The innovation in the present investigation is represented by instructional technology, including computer-based tools and processes, and diffusion is represented by the extent to which the medical school faculty at a state university has adopted instructional technology in teaching and learning. According to Rogers (1995), individuals in a social system do not adopt an innovation at the same time, a certain percentage of individuals are relatively earlier or later in adopting a new idea. Based on the innovativeness criterion that the degree to which an individual is relatively earlier in adopting new ideas than other members of a social system, the distribution of various adopter categories forms a normal, bell-shaped curve that illustrates Innovator (2.5%), Early Adopter (13.5%), Early Majority (34%), Late Majority (34%), and Laggards (16%) (Figure 1). Figure 1. Adopter Categorization on the Basis of Innovativeness (Rogers, 1995) Individuals who adopt an innovation at different points in diffusion process differ from one other in terms of social and psychological characteristics (Rogers, 1995). Those characteristics determine the individuals’ willingness to adopt an innovation and their leadership functions. Some of the differences that have been cited separating early adopters from the mainstream include: ¾ ¾ ¾ ¾ Early Adopters Favor revolutionary change Visionary Strong technology focus Risk takers ¾ ¾ ¾ ¾ Mainstream Favor evolutionary change Conservative Problem oriented Risk Averters 214 ¾ ¾ ¾ Experimenters Generally self-sufficient Horizontally connected ¾ ¾ ¾ Want proven applications May need significant support Vertically connected The differences between people who fall into Rogers’ Early Adopter and Early Majority categories create gaps in motivation, expectations and needs. The literature on individual characteristics of the faculty indicated that early adopters of instructional technology share common characteristics such as higher perceptions of efficacy and expertise (Anderson, Varnhagen & Campell, 1999; Jacobsen, 1998; Lichty, 2000; Oates, 2001), risk taking and experimentation (Oates, 2001), positive attitude toward technology ( Spott, 1999) and personal interest in technology (Oates, 2001). As stated by Bates (2000), “because of the central role that faculty members play in the work of the universities and colleges, any change, especially in core activities such as teaching and research, is completely dependent on their support” (p.95). Therefore, in order for large-scale technology adoption and diffusion to happen, it is critical to understand and bridge differentiated needs and expectations of faculty members (Garofoli & Woodell, 2003). Thus, the purpose of the present study is to explore the differences between faculty members who are open to, and those who are reluctant or resistant to IT adoption, and to determine if faculty characteristics contribute to the prediction of faculty adopter categories. Rogers’ (1995) theories of diffusion of innovations were used to analyze the differences in the IT adoption of the medical school faculty. Method The present investigation surveyed faculty members from basic and clinical science disciplines at a faculty of medicine at a state university in Turkey. Information was gathered about technology use patterns, computer experience and use of technology for teaching, perceived computer use self efficacy, perceived value of IT, perceived incentives, and barriers. Survey items were adopted or selected from previous investigations of faculty adoption patterns (Anderson, Varnhagen, & Campbell, 1999; Jacobsen, 1998) and Microcomputer Utilization in Teaching Self-Efficacy Beliefs Scale (Enochs, Riggs, & Ellis, 1993). The survey was distributed to 305 faculty members and complete data was obtained from 155 (50.3%) participants (72.7% male and 27.3% female), holding various academic ranks (i.e. 32.7% professors, 19.6% Assoc. Professors, 22.1% Asst. Professor and 25.6% others) having an average of 10 years of teaching experience. While the average age was 41 years, the largest group (≈55%) was in the 31-40 age groups. The majority of the respondents (76.8%) were instructors of clinical science. Adopter Groups In order to classify respondents into adopter categories (early adopters (EA), mainstream faculty (MF)), the individual innovativeness scoring procedure developed by Anderson, Varnhagen and Campbell, 1999 was used. The scoring procedure was developed on the basis of the assumption that EAs have come to use these technologies earlier and have gained more expertise relatively to majority faculty (Anderson, Varnhagen, and Campbell, 1999). A composite score was calculated for innovativeness of faculty by adding the self_rated expertise level of each individual faculty member (i.e., 1 for Extensive, 2 for Good, 3 for Fair, 4 for Novice, 5 for None) indicated for each of the 11 types of computer software and tools The total possible range for cumulative score for innovativeness is 11-55; the sample scores ranged from 13 to 51. Consistent with Rogers’ assertion that adoption of an innovation will be normally distributed, cumulative frequency of the scores on this scale approach an S-shaped curve which lends confidence to assumption of normalcy. By using Rogers’ (1995) adopter categories and innovativeness scores, 16% (n=25) of the respondents were assigned to the Early Adopter group (EA) ( 2.5% Innovators+13.5% Early Adopters =16% EA), and 84% (n=130) of the respondents were assigned to the Mainstream Faculty (MF) group( 34% Early Majority+34% Late Majority+ 16% Laggards=84%MF) (Figure 2). 215 Figure 2. Distribution of Faculty Innovativeness Scores Results The EA group is more likely to consist of junior assistant professors (χ2(1, 155)=5.59, p=0.019), in the 20-40 age interval (χ2(1, 25)=4.840, p=0.028). Although there are no significant differences between adopter groups in terms of computer ownership and internet access at home and in the office, the EA group uses computers more often(χ2(1, 146)=11.34, p=0.001) . Technologies Used in Teaching Participants were asked to indicate which of the 12 instructional technologies they use in the teaching-learning process. Early adopters significantly have used more technologies than Mainstream Faculty group (t(151)=2.841, p<0.05 Ms 5.58 vs. 4.38), and it is likely that they have used course web pages (Pearson χ2(1, 153)=8.306, p=0,009), web resources (χ2(1, 153)=7.018, p=0.018) and commercial educational software (χ2(1, 153)=22.077, p=0.000) more than the Mainstream faculty. The proportion of technologies used by the adopter group is presented in Table 1. These findings indicate that relatively new instructional technologies have diffused into the early adopter group more than the mainstream faculty. Table 1. Adopter Groups’ Technology Use Early Adopters % Blackboard 62.5 Overhead 75.0 Slide Projector 75.0 Computer + Projection 100.0 Video 25.0 Sound 8.3 Special Laboratory 29.2 Course web sites 33.3 Web resources as a part of content 37.5 Commercial educational software 16.7 Word processors for course materials 45.8 Presentation software 50.0 Mainstream Faculty % 67.4 69.8 76.7 94.6 19.4 3.1 20.9 10.9 14.7 0.0 31.0 29.5 Computer Use Self-Efficacy Computer use self-efficacy is defined as an individual’s belief regarding their ability to use computer competently (Compeau & Higgins, 1995). Previous research provides evidence that perceived efficacy regarding 216 computer use is a significant factor in an individual’s adoption decision (Lichty, 2000; Marcinkiewicz, 1994; Compeau & Higgins, 1995). In order to assess faculty self-efficacy in computer use in teaching, the 10-item “Microcomputer Utilization in Teaching Efficacy Belief Scale” developed by Enochs, Riggs and Ellis (1993) and used in previous research (Lichty, 2000) was used. Respondents were presented with a five-point scale (i.e. 5 for Strongly Agree, 4 for Agree, 3 for Neutral, 2 for Disagree, 1 for Strongly Disagree) to indicate their level of agreement with each statement about computer use. Internal consistency of this subscale yielded a coefficient alpha of .91. As expected, EAs had significantly higher level competency of computer use than the MF group (t(144)=6.263, p<0.01 Ms 44.68 vs. 37.38) Perceived Value of IT “The individual’s attitudes or beliefs about the innovation have much to say about his or her passage in the innovation-knowledge process” (Rogers, 1995, p.167). To adopt an innovation, the individual must define the innovation as relevant and useful in a specific situation. In order to gather data about faculty perception of the value of IT in medical education, participants were asked to indicate their level of agreement on 10 items regarding value of IT using five-point scale (i.e. 1 for Strongly Agree, 2 for Agree, 3 for Neutral, 4 for Disagree, 5 for Strongly Disagree). Although IT use in medical education is perceived to be valuable by the majority of the faculty (Mean=11.75, SD = 3.4), EAs valued IT more than MFs significantly ( t(151)= -2 .681, p<0.01, Ms 10.12 vs. 12.07). The EA group had statistically stronger beliefs than the MF group that technology enables instructors addressing different learning styles, using time effectively and increasing their productivity (Table 2). Table 2. Significant Differences between Adopter Groups on the Perceived Value of IT Items t df p Means Technology enables me to address the different -3.055 153 0.003 1.24 vs. 1.64 learning styles of students. Using technology enables me to use lecture time -2.357 153 0.020 1.28 vs. 1.61 efficiently Using technology increases my productivity as -2.357 153 0.020 1.28 vs. 1.61 an instructor Perceived Barriers In explaining the limited adoption of instructional technology among faculty members, Jacobsen (2000) states that “the explanation for limited adoption may be found in the many barriers that still constrain use by enthusiastic beginners” (p.2). Table 3. Percentages of Agreement on Barrier Items Scale of Adopter Groups Early Mainstream Barrier Items Adopters Faculty % % Lack of computers for faculty members. 66.7 54.3*** Lack of printers and/or other peripherals needed to effectively 72.0 60.0*** use computers for teaching and learning. Lack of computers for students 68.0 68.0*** Lack of support for supervising student computer use. 80.0 66.1*** No reward structure that recognizes faculty members for using 66.7 67.2*** technology in teaching and learning. Inadequate financial support for the integration of instructional 54.2* 37.3 technology. Lack or inadequacy of training opportunities for faculty 44.0 51.9** members to acquire new computer knowledge and skills. *** rated agree or strongly agree by over 50% of the both adopter groups *rated agree or strongly agree by over 50% of EA ** rated agree or strongly agree by over 50% of MF 217 Participants were presented with a five-point scale (i.e., 1 for Strongly Agree, 2 for Agree, 3 for Neutral, 4 for Disagree, 5 for Strongly Disagree) to indicate the level of their agreement with 20 barrier related items. An estimate of the internal consistency of this subscale yielded a coefficient alpha of .86, which indicates that the faculty responded consistently across the items. Over 50% of both adopter groups agreed or strongly agreed that the lack of computers for individual faculty and students, the lack of computer peripherals, the reward structure and the lack of support service for students were barriers to the integration of technology into medical education. In addition to these barriers, 54.2 % of the EA group agreed or strongly agreed that inadequate financial support was a barrier and 51.9% of the MF group agreed or strongly agreed that the lack of training opportunities was an impediment to technology integration (Table 3). Although the adopter groups gave similar responses, the EA and MF groups had exhibited significant differences in their perceptions about barriers to adoption (Table 4). Because the EA group has a relatively higher level of technological knowledge and skills, and interest in technology use, as expected, the EA group rated knowledge and skills barrier lower than the MF group. The EAs also disagreed highly than the MF group about the statements that software and IT are unsuitable for their instructional needs. The EA group expressed a higher level of agreement than the MF group regarding faculty members’ interest in using technology for instruction. Table 4. Significant Differences Between Adopter Groups’ Perceptions About Barriers Items t df p Means I have lack of necessary knowledge and skills for 6,74 152 p < .001 4,64 vs.3,28 using technology effectively. Available software doesn’t fit to my instructional 2,92 149 p < .05 4,24 vs. 3,75 needs. Instructional technologies do not fit the course or 3,55 152 p < .05 4,48 vs.3,95 curriculum that I teach. Faculty members are not interested in using -2,43 151 p < .05 3,00 vs. 3,52 technology for instruction. Perceived Incentives The participants were presented with a five-point scale (i.e., 1 for Very Important, 2 for Important, 3 for Neutral, 4 for Not Important, 5 for Not Important At All) to rate the level of importance of various statements on incentives for adoption. Over 90% of both adopter groups rated that among university policy and plans for diffusion of IT, investments in infrastructure, training and support and financial support are very important incentives for IT adoption. Reward structures and the reduction of teaching loads also were rated as important incentives by over 60% of both groups. Preferred Methods of Learning Concerning Technology and Support In many research studies, resources for training and support were identified as important incentives to faculty adoption of technology for teaching (Anderson, Varnhagen & Campbell, 1999; Green, 1999). Jacobsen (1998) suggest that “Individuals tend to have preferred methods for learning more about technology (p.96). In terms of methods for acquiring knowledge and skills about technology and support respondents rated their level of preferences on 12 items using a five -point scale (i.e., 1 for Strongly Prefer, 2 for Prefer, 3 for Neutral, 4 for Don’t Prefer, 5 for Don’t Prefer at All). Over 90% of the both adopter groups rated online resources and printed materials as strongly preferred methods. Because EAs have a higher level of expertise in technological resources, not surprisingly, EAs preferred online resources more strongly (t(153)=-2.726, p<0.01 (Ms 1.24 vs. 1.64)) than MFs. While the EA group preferred or strongly preferred workshops and presentations (84%), structured in service training (79.2%) and experimenting alone (68%), MFs preferred structured in service training (88.3%), workshops and presentation (82.5%) and experimenting alone (58.4%) in descending order. Participants’ preferences of help and support are given in Table 5. Consistent with the Rogers’(1995) assertion that early adopters’ interpersonal networks are more likely to be outside, EAs preferred colleagues at another 218 institution (t(146)=-2.807, p<0.01 (Ms 2.04 vs. 2.69)) and outside professionals (t(148)=-2.851, p<0.01 (Ms 2.08 vs. 2.71)) significantly stronger than MFs. Table 5. Preferred Methods of Support By Adopter Groups Early Adopters Mainstream Faculty Items Experienced graduate students Colleagues on campus Colleagues at another institution Outside professionals Support service Hot-line, or telephone assistance One on one help % % 95.8 96.0 72.0 80.0 87.5 50.0 79.2 84.6 90.6 50.4 48.0 90.4 58.1 89.5 Relationships between Faculty Characteristics and the Adopter Categories Logistic Regression analysis was conducted to determine whether a set of faculty characteristics (rank, sex, age, discipline, teaching experience, self-efficacy, perceived value of IT) contribute significantly to prediction of faculty members’ adopter categories. More than 30% of the variation in the Adopter Group variable explained by the Logistic Regression model (Cox&Snell R2=0.307, Nagelkerke R2=0.520). Results of the stepwise logistic regression analysis indicated that among the most powerful predictors of faculty adopter groups are rank and computer use self-efficacy (Table 6). This model shows that faculty members whose ranks are lower than professor and, faculty members whose self-efficacy beliefs are higher are more likely to be an Early Adopter. Table 6. Results of Stepwise Logistic Regression Beta S.E. z.stat Wald Variable .930 2.57 6.609 -2.390* Rank -.375 .080 -4.68 21.769 Computer-use Self Efficacy 21.57 Constant Model Chi-square=78.216, df=2, p<0.001 Cox&Snell R2=0.285, Nagelkerke R2=0.483 Note:*Significant at the .05 level. a Odds ratio associated with one unit increase in rank b Odds ratio associated with one unit increase in computer use self-efficacy p 0.010 0.000 Exp(B) 0.092a 0.687b The model explained more than 28% of the variation in the Adopter Group variable and was able correctly to classify 47.8% of early adopters and 96.6% of mainstream faculty, for an overall success rate of 88.7%. Discussion The findings of this study provided additional evidence that early adopter and mainstream faculty have different characteristics and different needs in technology integration. As suggested by Jacobsen (1998), understanding the differences between early adopter faculty and mainstream faculty will help us build programs and encourage faculty members to pursue the adoption of instructional technology. Identifying the differences between early adopters and the mainstream faculty leads to the understanding that different approaches are needed to bridge the gap between EA and MF groups in the diffusion of instructional technology and to encourage EAs for their efforts. It is obvious that faculty support programs designed with a “one-fit-all” approach fail to succeed. The results of this study indicate that the early adopter faculty and the mainstream faculty in this study have different needs in training and support. It seems that formal training programs do not appeal to early adopter faculty members who have a level of expertise in technology use. In designing training programs, institutions might consider gearing early adopters towards advanced topics and focusing on the specific needs of these faculty members. By providing incentives such as release time for training, providing funds for developing 219 instructional materials, supporting symposia and conference participation, EAs must be motivated to continue their focus on innovation, the re-invention of technology and to share their expertise with the mainstream. Making innovative faculty members aware of each other and encouraging them to share resources and expertise could be an effective way of increasing motivation and sustaining growth (Surry & Land, 2000). The literature and the results of this study suggest that computer use self-efficacy belief of individuals is a significant factor in their utilization of technology. From the standpoint of the self-efficacy theory, the ideal method for developing teachers' self-efficacy for computer use would be to provide them with training and support to work successfully with technology (Albion, 1999). In designing training and support programs, institutions might consider a number of strategies that address self-efficacy perceptions of the mainstream faculty members. Besides providing training opportunities, building comprehensive and systematic technology support systems is essential to increasing the faculty member’s confidence with the use of technology in teaching and learning. Faculty development seemed to work best when the institution had a culture pervaded by the use of technology and supported by a wide range of strategies (Bates, 2000). One of these strategies is an extensive investment in infrastructure. Although ownership of computers for professional/home use is almost completely diffused among the faculty in this study, most faculty members were dissatisfied with the current investment in technology and distribution of available resources among departments. Therefore, to encourage the adoption and diffusion of technology, the institutions’ investment in technology should be based on a long range technology plan driven by the institutions’ overall vision and strategy for its teaching. Similar to most higher education institutions, research publications are the predominant or even the only criterion for appointment, tenure and promotion in medical institutions. Medical school faculty members, especially young faculty members who have carrier concerns are unwilling to spend time and effort to develop technology-based applications for teaching, although they are more familiar with technology than their senior professors. The results from this study also indicate that the lack of a reward system that recognizes faculty efforts in technology use for teaching is an impediment for wide-spread diffusion of instructional technologies. Therefore, to motivate faculty members in technology use, intuitions should develop reward and incentive systems that are linked to technology use in teaching and learning such as release time, funding to support the development of technology materials and considering instructional activities in the promotion process. Recommendations for Higher Education Institutions Higher education institutions today are confronted with instructional technology innovation, which is transforming the way in which faculty and students interact and the roles they take. If the goal of the higher education institution is the integration of technology for a transformative change, then rather than the acquisition of technology itself, there must be a clear focus on the faculty members who use technology. For large-scale technology integration to occur in teaching, it is essential to understand and address differentiating needs of faculty in faculty development and support systems. The following suggestions are offered to higher education institutions to improve their faculty members’ IT adoption for teaching and diffusion of instructional technology in medical education: 1. Develop a long-range technology plan driven by the institutions’ overall vision and strategy for its teaching. 2. Establish a promotion system that places a high value on teaching and the use of innovative teaching methods. 3. Design faculty development programs considering the needs of different faculty member profiles. 4. Provide training programs not only on the technical aspects of technology, but also about the integration of technology for teaching and learning. 5. Establish an instructional technology center in which faculty members can get help from and work together with IT related professionals. 6. Provide systematic technical and professional support. References Albion, P. R. (1999). Self-Efficacy Beliefs as An Indicator of Teachers' Preparedness For Teaching With Technology. In Price, J. D., Willis, J., Willis, D. A., Jost, M., & Boger-Mehall, S. (Eds.), Technology and Teacher Education Annual 1999, Charlottesville, VA, USA: AACE, 1602-1608. 220 Albright, M. J. (1996). Instructional Technology and Higher Education: Rewards, Rights, and Responsibilities. Keynote Address at the Southern Regional Faculty and Instructional Development Consortium, Baton Rouge, LA, USA, February 5 1996, ERIC, ED392412. Anderson, T., Varnhagen, S., & Campbell, K. (1999). Faculty Adoption of Teaching and Learning Technologies: Contrasting Earlier Adopters and Mainstream Faculty. The Canadian Journal of Higher Education, 28 (2-3), 7198. Bates, A. W. (2000). Managing Technological Change: Strategies for College and University Leaders, San Francisco: Jossey-Bass. Carlile, S., & Sefton, J. (1998). Healthcare and Information Age: Implications for Medical Education. Medical Journal of Australia, 168 (7), 340-343. Compeau, D., & Higgins, C. A. (1995). Social Cognitive Theory and Individual Reactions to Computing Technology: A Longitudinal Study. MIS Quarterly, 23 (2), 145-158. Enochs, L. G., Riggs, I. M., & Ellis, J. D. (1993).The Development and Partial Validation of Microcomputer Utilization in Teaching Efficacy Beliefs Instrument in Science Setting. School Science and Mathematics, 93, 257-263. Garofoli, E., & Woodell, J. (2003). The Faculty Development and the Diffusion of Innovations. Syllabus, 16 (6), 15-17. Geoghegan, W. H. (1994).Whatever Happened to Instructional Technology. Paper presented at the 22nd Annual Conference of the International Business and Schools Computing Association, July 17-20, 1994, Baltimore, Maryland, USA. Green, K. C. (1999). The 1999 National Survey of Information Technology in Higher Education. The Campus Computing Project, retrieved November 29, 2005 from http://www.campuscomputing.net/summaries/1999. Green, K. C. (2000). The Real IT Challenge: People, Not Products. Converge, January, retrieved November 29, 2005 from http://www.centerdigitaled.com/converge/?pg=mag&issue=1:2000. Holloway, R. E. (1996). Diffusion and Adoption of Educational Technology: A Critique of Research Design. In. Jonassen, D. H (Ed.), Handbook of Research For Educational Communications and Technology, New York, USA: Simon& Schuster Macmillan, 1107-1133. Jacobsen, D. M. (1998). Adoption Patterns and Characteristics of Faculty Who Integrate Computer Technology for Teaching and Learning in Higher Education. Doctoral Dissertation. Educational Psychology, University of Calgary, retrieved November 29, 2005 from http://www.acs.ucalgary.ca/~dmjacobs//phd/diss/ Jacobsen, D. M. (2000). Examining Technology Adoption Patterns by Faculty in Higher Education. Paper presented at the ACEC2000: Learning Technologies, Teaching and the Future of Schools, July 6-9, 2000, Melbourne, Australia. Lichty, M. (2000). The Innovation-Decision Process and Factors That Influence Computer Implementation by Medical School Faculty. Doctoral Dissertation, Instructional Technology, Wayne State University, UMI ProQuest Digital Dissertations, 9966157. Marcinkiewicz, H. R. (1994). Computers and Teachers: Factors Influencing Computer Use in the Classroom. Journal of the Research on Computing in Education, 26 (2), 221-237. Oates, K. D. (2001). University Faculty Who Use Computer Technology. Doctoral Dissertation, Instructional Technology, Georgia State University, USA. UMI ProQuest Digital Dissertations, 3008105. Rogers, E. M. (1995). Diffusion of Innovations (4th Ed.), New York, USA: Free Press. Seels, B. B., & Richey, R. C. (1994). Instructional Technology: The Definitions and the Domains of the Field, Washington, DC, USA: Association for Educational Communications and Technology. 221 Spotts, T. H. (1999). Discriminating Factors in Faculty Use of Instructional Technology in Higher Education. Educational Technology and Society, 2 (4), 92-99. Surry, D. W. (1997). Diffusion Theory and Instructional Technology. Paper presented at the Annual Conference of the Association for Educational Communications and Technology (AECT), February 12-15, 1997, Albuquerque, New Mexico, USA. Surry, D. W., & Land, S. M. (2000). Strategies for Motivating Higher Education Faculty to Use Technology. Innovations in Education and Training International, 37 (2), 145-15. 222 Nykänen, O. (2006). Inducing Fuzzy Models for Student Classification. Educational Technology & Society, 9 (2), 223-234. Inducing Fuzzy Models for Student Classification Ossi Nykänen Senior Researcher, Tampere University of Technology, Digital Media Institute Hypermedia Laboratory, P.O.Box 553, FI-33101 Tampere, Finland Tel: +358 3 3115 3544 ossi.nykanen@tut.fi ABSTRACT We report an approach for implementing predictive fuzzy systems that manage capturing both the imprecision of the empirically induced classifications and the imprecision of the intuitive linguistic expressions via the extensive use of fuzzy sets. From end-users' point of view, the approach enables encapsulating the technical details of the underlying information system in terms of an intuitive linguistic interface. We describe a novel technical syntax of fuzzy descriptions and expressions, and outline the related systems of fuzzy linguistic queries and rules. To illustrate the method, we describe it in terms of a concrete educational user modelling application. We report experiments with two data sets, describing the records of the students attending to a university mathematics course in 2003 and 2004. In brief, we aim identifying the failing students of the year 2004, and develop a procedure for empirically inducing and assigning each student a fuzzy property "poor", which helps capturing the students needing extra assistance. In the educational context, the approach enables the construction of applications exploiting simple and intuitive student models, that to certain extent are self-evident. Keywords Fuzzy systems, Student modelling, Linguistic interfaces, Fuzzy queries and rules Introduction Modern information management systems enable the recording and the management of data using sophisticated data models and a rich set of management tools. In the context of educational systems, the information typically includes details about learning material, the tasks and the objectives, the course information, the contact information, the teacher and the student profiles, and the information related to student assignments, the tests, the grades, and other records. When perceived in the technical terms of the underlying information system, it soon becomes very difficult to manage, integrate, and access different kinds of information. In this article, we seek means to model the imprecision of information and simplify the access to information systems, in terms of fuzzy modelling. In brief, computer-based learning management systems seek to be equivalent, if not superior, to the traditional learning systems (Darbhamulla & Lawhead, 2004). However, an increased technological potential does not automatically mean better applications. For instance, while the standardisation of the application interfaces (see, e.g., (Dodds, 2001)) certainly improves the development of learning systems, and it is possible to semantically integrate different sources of information, say, by using standard general-purpose query languages (Boag et al. 2005; Prud'hommeaux & Seaborne, 2005), the engagement with the technical details may itself become an obstacle. A data format and a technical vocabulary that suits the needs of technical storage, does not necessarily suit the needs of the developers of the more abstract learning applications, not to mention the conceptualisations of the teachers and the students. In particularly, while it is relatively easy to store information about students' progress technically and to classify assignments as "easy", "intermediate", or "difficult" by hand, it is surprisingly difficult to automate the process of classifying students with respect to these semantic labels in terms of crisp computing. Assuming a good teacher and only a handful of students, this problem is largely irrelevant. However, when the number of the students increases or they are out of reach, seeking computersupported means becomes interesting. In addition, the ability to capture the key domain objects using intuitive expressions not only supports the activities of the teacher, but also makes it easier to implement and integrate learning applications based on domain-specific design. It is instructional to consider this problem from the perspective of domain, task, and user modelling (see, e.g., (Brusilovsky & Cooper, 2002)). In general, computer-aided teaching, learning, and collaboration systems (as information systems) enable the construction of various kinds of decision-support systems that help organising courses and adapting educational content to suit the needs of different kinds of students. At the heart of such systems lies a student (user) model, that records and explains the progress of the students, to be exploited by the learning environment. The basic requirement of student modelling is the ability to assign individuals and user groups meaningful labels related to their characteristics and activities, to be exploited on the level of the reacting ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 223 application. Historically, the development of modelling has evolved from crisp to fuzzy models, eventually taking context explicitly into account (Jameson 1995; Jameson 2001). Nevertheless, a good user model is intuitive and simple, allows sharing domain knowledge, and matches the requirements of the modelling task. The introduction of fuzziness typically aims managing imprecision (or vagueness) in applications. In short, the existing fuzzy systems fall into two categories. While fuzzy logic or fuzzy expert systems consider fuzziness in terms of fuzzy implication and a generalisation of crisp (two or finitely-valued) logic, fuzzy control systems aim reproducing the behaviour of intuitive control rule groups by exploiting fuzzy models, i.e., computing with fuzzy sets (Turunen, 1999; Jang & Sun, 1997; Hüllermeier, 2005; Cox, 2005). Further, the fact that fuzzy membership functions appear often overly precise in applications has motivated the development and application of, e.g., interval-valued and type-2 fuzzy sets (Vaucheret, 2002; Mendel, 2001). In the context of educational applications, the reported soft computing experiments include applications of Bayesian and Neuro-Fuzzy methods (Jameson, 1995; Xenos, 2004; Stathacopoulou et al., 2005). These methods have been widely applied, taking many aspects of education into account. The reported applications notably demonstrate analysing and evaluating both students based on individual assessments, students' views, education quality, grades of journals, and the performance of entire academic departments (Ma & Zhou, 2000; Liu & Maes, 2004; Suarez, 2003; Nokelainen et al., 2001; Arriaga et al., 2005; Ibrahim, 2001; Turban et al., 2000; Lopes & Lanzer, 2002). The claimed advantages of fuzzy modelling range from faithfully evaluating students' learning and cognitive abilities to moving towards personalised education (Stathacopoulou et al. 1999; Weon & Kim, 2001; Kavcic et al., 2003). In this article, we report an approach of implementing a predictive fuzzy system that manages capturing both the imprecision of the empirically induced classifications and the imprecision of the intuitive linguistic expressions via the extensive use of fuzzy sets. We describe a novel technical syntax of fuzzy descriptions and expressions, and outline the related systems of fuzzy linguistic queries and rules. In the educational context, the approach provides a concrete way of inducing intuitive semantic labels from the existing data archives, suitable for capturing the domain objects (users, user groups, processes, tasks, and artefacts) using fuzzy linguistic expressions. The basic idea is to describe the domain objects and the associated linguistic expressions in terms of neutral fuzzy sets, subject of fuzzy queries and rules. For the benefit of a concrete discussion, we describe implementing a novel predictive fuzzy system that demonstrates fuzzy modelling in the context of a realistic application. We report experiments working with two data sets, describing the records of the students attending to a particular university course of mathematics in 2003 and 2004 at the Tampere University of Technology, Finland. In short, we aim predicting the poor students based on the archives, and semantically integrate this information with fuzzy linguistic rules and queries. The main contribution of this article is threefold: First, we introduce a design of a realistic predictive fuzzy system that provides a basis for semantically integrating and encapsulating various kinds of information systems. Second, we develop the syntax of fuzzy expressions by introducing the notions of fuzzy linguistic constructors and pure linguistic concepts, and describe the related rule and query systems. Third, we provide a concrete bridge between the two complementary aspects of fuzziness in educational design, by syntactically integrating the imprecision due to lack of information (prediction accuracy) with the imprecision due to use of informal language (fuzzy linguistic expressions). The rest of this article is organised as follows: Section 2 describes the empirical data sets and Section 3 demonstrates the potential of predicting the failing students. Section 4 presents the design of predictive fuzzy systems and Section 5 outlines the definitions related to fuzzy queries and rules. Section 6 briefly considers the applicability of model-free methods, and finally, Section 7 concludes the article with few notes. Empirical Data The success user modelling applications is determined by the quality of the user data. In the context of learning management applications, the backbone of modelling may be economically compiled from the formally recorded achievements. Our experiments were conducted using the accumulated data sets from the well-established basic course Engineering Mathematics I, during two successive years 2003 and 2004. The data describes the performance of the individual students in terms of 22 attributes: points from 13 assignments (0-10, 'x' denotes a missing value), four mid-term exams (two attributes of 0-6 points each, 'x' denotes a missing value), and the final grade (0-5). The final grades were compiled using the attributes and scaled according to the final statistics. Compiled into chronological order, this approach provides a vector of 22 224 attributes for each student ([a1, a2, a3, e1, e2, a4, a5, a6, a7, e3, e4, a8, a9, a10, e5, e6, a11, a12, a13, e7, e8, g ]). Considering ongoing courses, the attribute vectors become available gradually. The data includes 358 student instances considering the year 2003, and 311 instances considering the year 2004. This provides two data sets D2003 = [αij], i = 1, 2, ,..., 358, j = 1, 2, ..., 21, and D2004 = [βij], i = 1, 2, ,..., 311, j = 1, 2, ..., 21, accompanied with the final grades [gi], i = 1, 2, ,..., 358, and [hi], i = 1, 2, ,..., 311, respectively. We can use this information to define a training data set for the concept failing student, [g'i], where g'i = 1 when gi = 0 and g'i = 0 otherwise (define [h'i] accordingly). The training data sets [αij|g'i] and [βij|h'i] may now be used for recognising the failing students. In particularly, the construction of the data sets enable using the first data set D2003 as training data for predicting the second data set D2004. 100 120 90 100 80 80 60 Students Students 70 50 40 60 40 30 20 20 10 0 0 0 1 2 3 4 5 0 1 2 3 4 5 Grades Grades (a) (b) Figure 1. The grade histograms of 2003 and 2004 Figures 1a and 1b depict the grade frequencies of the years 2003 and 2004. Of course, the data from the year 2003 does not completely characterise the year 2004, even if we assume that the course format remained unchanged. In principle, any change in the course agenda makes it hard to use the data from the previous year for the benefit of the next year. Changes might notably include (and partly did include) change of topics, providing extra tutoring for the poorly performing students, extra tests and questionnaires, and simply showing statistics like Figure 2 to the students (please see below). Using Training Data in Applications: Making Predictions Considering applications that rely on user modelling in the educational domain, perhaps the most important use case is classifying the students based on their observed and expected performance. This yields categories of fuzzy user groups providing decision-support information for various applications. For instance, consider the expression "student is very poor" in the context of the following informal rule as a part of the adapting application: If a student is very poor and lives far from the campus, 1) then the need for extra attention is high. In order to establish rules, the related fuzzy terms must be defined. In this article, we assign students (as identifiable domain objects) fuzzy descriptions which are to be associated with fuzzy linguistic constructors (as a sort of hedges) in rules and queries. Let us first briefly consider the predictive component of the process of inducing fuzzy descriptions. In our application, it is possible to establish a straightforward procedure for recognising the failing students using a simple domain-specific design, simply by assuming average performance: A student fails the course if she does not accomplish at least 40% of the assignments or gets less than 24 points from the mid-term exams (the point limit may decrease based on the final statistics). Additional requirements include attending to at least four computer exercises and completing a preliminary test. The additional requirements were not used in the predictions in this study. Accompanied by the fact that the grades were scaled based on the final statistics, this actually makes the prediction problem somewhat harder. 225 1,2 Accuracy 1 0,8 2003 2004 0,6 0,4 0,2 21 19 17 15 13 11 9 7 5 3 1 0 Attributes Figure 2. Prediction accuracy of the years 2003 and 2004 Figure 2 depicts a demonstration of using the series of the first 1, 2, ..., 21 attributes of D03 and D04 for predicting the concept failing student by simply assuming average performance. In the experiment, the data was divided into k sets, according to the number of the attributes used in the prediction. The experiment shows that a naïve classifier, relying upon domain-specific information and simple design insight, manages to predict, or recognise, the failing students with a relatively high accuracy. Informally speaking, Figure 2 provides a clear message for the students struggling to pass courses. When grades are considered, failing performance can be reasonably well recognised after mid-course. Of course, this data alone can not tell what to do in order to improve the situation, expect for the "work harder" signal. In addition, when separated from the more decisive contextual characteristics, the recorded points and the mid-term exams do not necessarily faithfully reflect the students' skills nor understanding. Considering fuzzy systems, however, the experiments seem to support the rationale of implementing predictive fuzzy models, since the trend of the prediction accuracy is clearly positive. Implementing a Predictive Fuzzy System Assume we wish to assert rules and perform queries on top of a database, using fuzzy linguistic expressions. For instance, a teacher might want to establish heuristic rules like (1) and search for the students needing extra attention. Further, assume all the information is not readily available and we have to rely on generalisations learned from the historical archives. For instance, the points from assignments become available only during courses. In practice, we face this task during the internal design of an adapting educational system. Trend of the Prediction Accuracy Accuracy IF s.a is rather A AND s.b is very B THEN s.c is very C ... Attributes (k) Training Data α1,1 α1,2 α1,3 ... α2,1 α2,2 α2,3 ... ... g1 g2 Fuzzy Rules Predictive Model notAtAll notVery moderately rather very Definitions of the Fuzzy Linguistic Constructors Fuzzy Descriptions Fuzzy Knowledge Base Observed Instance Data s1: β1,1 β1,2 ... β1,k ? ? s2: β2,1 β2,2 ... β2,k ? ? ... ? ? si : poor Application Default Knowledge Base Figure 3. The basic components of a predictive fuzzy system Figure 3 depicts the basic components of a predictive fuzzy system. At the heart of the systems lies the predictive model which assigns domain objects (e.g. students si) fuzzy descriptions, based on the available 226 training data and the observed instance data from the application. The fuzzy descriptions add to the default knowledge base (e.g. a default thesaurus defining the basic concept taxonomies and the terminological synonyms), providing a basis for interpreting the fuzzy rules and queries. The linguistic component is defined by identifying the linguistic concepts to which the descriptions apply (e.g. poor), and establishing the fuzzy linguistic constructors (e.g. rather), to be used in the rules and queries. In brief, applications use the predictive fuzzy system for performing fuzzy inference and making fuzzy linguistic queries. This might also include, e.g., interpreting fuzzy SQL queries (see, e.g., (Cox, 2005)). More complex applications might in addition explicitly control the default knowledge base and the definitions of the fuzzy linguistic constructors, in terms of bootstrapping the predictive fuzzy system with application-specific defaults (e.g., modifying the fuzzy linguistic constructors upon context). Let us next consider the task of establishing the fuzzy linguistic expressions for fuzzy linguistic rules and queries. For technical purposes, let us equate, in the context of students attending a course, the concepts failing and very poor. Effectively, this provides a fuzzified notion that enables capturing the students that are "close to" failing. Turned around, a student that passes courses with (nearly) maximal grades is not at all poor. Technically speaking, we would thus like to assert, e.g., the following kinds of expressions concerning students: T = { not at all poor, not very poor, moderately poor, rather poor, very poor }. In a typical fuzzy application, these labels would be modelled in terms of a linguistic variable, using the elements of T as the term set (see, e.g., (Jang & Sun, 1997; Mendel, 2001; Cox, 2005)). Alternatively, however, the expressions include internal structure: a constructor part, and a concept part. The constructor resembles the notion of a hedge or a modifier while the concept part identifies the (pure) linguistic concept that defines the type of the particular expression. For instance, instead of asserting s is RatherPoor, (2) we may assert s isRather Poor. (3) The subtle syntactical difference becomes noteworthy when considering the constructors in the context of crisp classes. Two practical consequences are particularly significant: First, when compared to (2), separating the constructor and the concept part in (3) reduces the number of the semantic labels required in applications since the constructors may be associated with several concepts. Second, since the shape of the associated fuzzy set does not have to follow the limitations of simple functional hedges (e.g., concentration for "very"), the constructors may be defined empirically. In particularly, this includes the case of taking carefully into account the actual use of particular semantic labels in applications (about survey-based terms, see Mendel (2001)). notAtAll notVery moderately rather very (1, 1) Coupling (0, 0) Degree of Association Figure 4. A fuzzy linguistic constructor sequence Figure 4 depicts a fuzzy linguistic constructor sequence that models the constructor terms. In short, a constructor may be perceived as a predicate that assigns a relationship between an subject and an object (see (3)), in terms of a (type-2) fuzzy set. This provides a proper generalisation of the is-a relationship or a crisp membership function and is suitable for describing the relationship between any domain objects and the selected linguistic concepts (e.g., students and the concept poor). Using fuzzy sets in modelling thus allows capturing imprecise descriptions which establishes an extension to description logic (see (Baader et al., 2002)). A singleton fuzzy set coincides with the plain (scalar) degree of fuzzy membership while fuzzy sets of other shapes enable modelling, e.g., the imprecision of the fuzzy definition itself. It is worth emphasising that while in this article we use fuzzy linguistic expressions in the syntax of the rules and the queries, they also enable recording imprecise assertions as well227 established statements. However, the considerations related to the assertion context are not trivial and are not discussed here. Let us then address the question of adequately modelling the semantic descriptions of the domain objects in the context of the course application. In brief, we wish to assign students a fuzzy property poor, using a predictive model. We do this by assigning each student identifier a fuzzy description that associates the identifier with the linguistic concept poor in terms of a type-1 fuzzy set (see Figure 5 below). In queries and rules, we may then compute with these properties using the related term poor (establishing the type of the expression) and a linguistic constructor, e.g., "moderately" in expressions (establishing the meaning of the constructor term as a fuzzy set). (1, 1) c ~ Centroid of Association Coupling p ~ Imprecision c (0, 0) p Degree of Association Figure 5. A fuzzy model for a student being poor We say that the fuzzy set depicted in Figure 5 is neutral. This is due to the fact that the domain of the membership function is normalised to [0, 1] and is not assigned any particular datatype information. Datatype information, however, might be required for establishing the definition via bindings to explicit quantitative data (e.g., predicted points of a course, distance in meters, etc.). Neutral fuzzy sets provide an intuitive explanation of the concepts imprecision (or expectancy, modelled by p), association (centroid of which is denoted by c), coupling (the max height of the membership function), precise, and crisp. In particularly, a type-1 fuzzy set is called precise if it is defined by a singleton membership function (p=0). Precise fuzzy sets for which c = 0 or c = 1 are called crisp. Note that the definitions of association and coupling are not traditionally introduced in terms of type-1 fuzzy sets; they become meaningful only when interpreted in the context of the following rule and query semantics (or in the context of type-2 fuzzy sets). 2 1,8 1,6 1,4 1,2 1 0,8 0,6 0,4 0,2 0 21 19 17 15 13 11 9 7 5 3 2003 2004 1 Average Error The experiment yielding Figure 2 provides a basis for modelling students being poor in terms of two observations. First, the predicted or the projected performance enables the definition of an appropriate semantic label in terms of a recognition task. Second, the precision of the assigned semantic labels is not a constant: it changes according to the available information or according to the estimated accuracy of the predictions. As a consequence, we may capture students throughout courses with a fixed set of semantic labels that get more precise as more information becomes available. In other words, the fuzziness of the application to certain extent decreases in time. Attributes Figure 6. The average error of predicting the exact grades 228 It seems obvious that we may try to categorise students directly by predicting the grades. As suspected, assuming average performance provides a concrete method for doing this. Figure 6 demonstrates an example of trying to predict the exact grades. The average errors from using the data of 2003 and 2004 follow a similar pattern. It seems that the difference can be explained by the statistical corrections of the grades. After the second mid-term exam, the grades can be predicted with the average error of 0.8. Indeed, the experiments (see Figures 2 and 6) seem to demonstrate that the naïve prediction algorithm provides systematic results and that we may use the archives for estimating the prediction errors reliably. Considering the application, we may describe the students of the year 2004 using the fuzzy property poor, based on the above definitions. Thus, as the course proceeds, we receive more information about the students (in terms of attributes) and may classify the students more accurately. In brief, the application involves two kinds of imprecision. The first flavour of imprecision is due lack of information. The second is due choice. Roughly speaking, a student may change her projected trajectory by studying harder (or less harder). However, when considering getting a grade via the mid-term exams, the window of influence obviously gets narrower as the course proceeds. We will next describe a simple algorithm that can be used to associate each student a fuzzy description. It might be helpful to imagine that we assign each student s a triangular neutral fuzzy set Ãs of the type poor. The shape of the related membership function μs can be described in terms of two parameters, centroid of the degree of the association, s.c, and imprecision, s.p: μs (x) = μs (x; s.c, s.p). By letting u = s.c - s.p - ε, v = s.c, and z = s.c + s.p + ε, 0 < ε << 1, we get the conventional definition (Jang & Sun, 1997): μs (x) = triangle(x; u, v, z) = max ( min ( (x-u)(v-u)-1, (z-x)(z-v)-1 ), 0), x ∈ [0, 1]. (4) Let s.x ∈ X = { 0, 1, ..., 54 } denote the number of points student s will earn from the mid-term exams and the exercises. By doing sufficiently exercises, students may earn max 54 - 48 = 6 bonus points to be added to the results of the mid-term exams. In practice, s.x = s.x(k) where k is the number of the attributes available during the course. Thus, s.x is a projected value, based on observing the performance of the past and assuming an average performance in the future. Define a function s.c: X → [0, 1], for modelling the (centroid of the) degree of association, associating the student s with the concept poor: s.c(s.x) = max( 0, min( 1, (24 - (s.x-24) ) / 24 ) ). (5) Let Error(k) denote the error of the prediction of the grades during the previous year(s), using k attributes (see Figure 6). Define a function s.p: K → E ⊂ ℜ, where k ∈ K = { 1, 2, ..., 21 } is the number of the attributes available, and E denotes the set of errors, for modelling the imprecision of the association: s.p(k) = Error(k) / 5. (6) We can now assign each student s a pair (c, p)s which models her being poor. Figure 6 depicts the development of the imprecision as more information becomes available. Intuitively, the pair (c, p)s may be interpreted as a neutral type-1 fuzzy set of certain shape (for triangular functions, consider (4) and Figure 5). The shape, however, must be decided by the application designer. The definitions (5) and (6) establish the induced fuzzy properties. For instance, the fuzzy set depicted in Figure 5 suffices modelling a student s when s.c = 0.8 and s.p = 0.2. Considering the binding, this happens when s.x = 28.8 and Error(k) = 1.0. Fuzzy descriptions provide intuitive labels of data in terms of the fuzzy property poor. As an example, Figure 7 depicts classification trajectories describing six particular students that took the year 2004 course with the grades 0, 1, 2, 3, 4, and 5. In brief, the trajectories denote the association of the students being poor, suitable for fuzzy modelling. The imprecision associated with the fuzzy models may be read from the Figure 6, using the errors of the year 2003. In other words, the classifications get rather precise around mid-course. Intuitively, the analysis reveals that the student s3 seems to improve her performance during the course while the student s1 is doing worse and worse as the course proceeds. Of course, more information is needed for explaining why the behaviour occurs. Other fuzzy properties may be modelled and studied in a similar fashion. For instance, a fuzzy property passive can be induced from the attendance statistics. Examples of completely crisp properties include the number of credit units, sex, age, etc. Finally, students might also be explicitly asked further attributes for various properties 229 that can help categorising them. For instance, learning style, interest towards the subject, estimated effort, time to travel to the campus, studying at home, and being active or passive in the course. As the user models become more refined, it is possible to model and interpret students' behaviour more faithfully. Note however, that these extra sources of information inevitably introduce new sources of fuzziness as well, in particularly the imprecision related to communication. Nevertheless, using neutral fuzzy sets for modelling these properties provides a well-established reference system for various kinds of applications. The chief benefit is the ability to model a wide range of fuzzy properties in terms of uniform descriptions. From the perspective of fuzzy systems and applications, this helps integrating information from various sources. Associated with the idea of linguistic constructors, the approach provides a concrete way to describe domain objects using a relatively small set of intuitive linguistic concepts and expressions. Association of Being Poor 1,2 1 s0 s1 s2 s3 s4 s5 0,8 0,6 0,4 0,2 21 19 17 15 13 11 9 7 5 3 1 0 Attributes Figure 7. The classification trajectories of five particular students Notes About Implementing Fuzzy Linguistic Queries and Rules In the previous section, we demonstrated the construction of a predictive fuzzy model yielding a simple knowledge base of fuzzy descriptions. However, the model and the rationale behind the associated technical definitions can not be appreciated unless the related query and the rule applications are outlined as well. In order to illustrate the technical use cases, let us next very briefly reconsider fuzzy properties in the context of linguistic expressions, namely fuzzy linguistic queries and rules (see (1)). A typical use case of educational decision-support systems might be retrieving a list of students that are likely to fail the course, i.e. a list of poor students as a fuzzy user group. Consider the following informal fuzzy linguistic query: Find students that are at least rather poor and very passive. (7) The expression (7) can be formalised in terms of conjuncted match patterns where match patterns may include an associated modifier part (at least, around, at most). Formally speaking, the expression (7) establishes a sequence of two match patterns <Aj> = <atLeast(rather), very> (where, in general, j = 1, 2, .., m) now associated with the fuzzy properties <pj> = <poor, passive>. Assuming each student s = 1, 2, ..., n is appropriately associated with the related fuzzy properties <vsj>, j = 1, 2, ..., m, we may compute the match for each student and rank the students according to the matches. Perhaps the most straightforward method for computing fuzzy matches is due to aggregating the firing strengths computed from the matching sequences of the fuzzy sets. Let Aj and vj denote two neutral fuzzy sets, associated with the fuzzy membership functions μAj: [0, 1] → [0, 1] and μvj: [0, 1] → [0, 1]. Define wj = maxx μAj(x) ∧ μvj(x). (8) This definition demonstrates linearity which is useful in implementations. We say that wj denotes the firing strength of <Aj, vj> and define the fuzzy match of <Aj, vj> as: 230 fMatch( <Aj, vj> ) = minj <wj>. (9) In other words, the fuzzy match is a value between 0 and 1, suitable to be used to order the students based on the query. There exists other strategies for computing fuzzy matches as well, e.g., aiming to preserve the aspect of imprecision of the matches. For brevity, we ignore the other definitions. Fuzzy linguistic queries are useful but do not provide good syntactic means for using the information further. Sometimes it is useful to assign students new fuzzy properties based on the existing ones. For instance, one might wish to assert the following simple fuzzy linguistic rule: IF a student is at least rather poor AND very passive, THEN she is problematic. (10) It is easy to see that fuzzy queries establish a special case of (simple linguistic) fuzzy rules. For instance, query (7) might be equivalently modelled in terms of assigning each student a new fuzzy property, problematic. The key difference, however, is that the new property problematic can be utilised in further queries and rules, to be aggregated with other properties of the same type. With certain restrictions, simple fuzzy linguistic queries provide the semantics of simple fuzzy linguistic rules. This is due to the fact that it is possible to interpret the fuzzy match as a fuzzy set; in our simple example, a fuzzy singleton would do. In general, more complex fuzzy rules may be implemented in terms of (control) rule groups. With certain assumptions, rule groups may be modelled using (slightly modified) Mamdani fuzzy systems (see (Jang & Sun, 1997; Cox, 2005)). The key differences between simple fuzzy linguistic rules and rule groups lie in descriptional complexity and coupling. Rule groups enable the modelling of more complex relationships than simple linguistic rules, but by default, the coupling (max height) of the implied fuzzy properties might decrease during the inference (outputting non-normal fuzzy sets). This causes certain difficulties related to interpretation in applications. For brevity, we will not consider rule groups further in this article. From Domain-Specific Design to Model-Free Methods There are several ways to implement predictive fuzzy models. In particularly, even if applying domain-specific design (e.g., assuming average performance) may provide the best results, it is not always feasible nor necessary. In principle, there exists a wide range of machine learning, data mining, and exploration methods suitable for the task (Russel & Norvig, 1995; Mitchell, 1997; Cox, 2005). We describe elsewhere a study and a method for inducing membership functions of fuzzy sets empirically upon a training set of positive and negative examples (Nykänen, 2004). In short, the study provides a method that generalises concept learning to include learning fuzzy concepts. The basic idea is to train a redundant Decision Tree (DT) to classify data according to a goal attribute that denotes the related crisp concept, eventually representing the positive end of the bipolar fuzzy concept. The membership grades of the instances are then computed from the topology of the tree, based on the depth of the tree and the number of attribute tests required for recognising a negative instance. Assuming the crisp concept can be learned well, the method allows assigning objects fuzzy membership degrees with an information-theoretic explanation. Let us next briefly describe an approach for implementing a model-free fuzzy system. A decision tree was trained for each j = 1, 2, ..., k data sets, using [αij | g'i], i = 1, 2, ..., 180 as the training data for predicting [h'i], i = 1, 2, ,..., 311, based on the instance vector [βij], i = 1, 2, ,..., 311. The available training data was simply split into half, in order to minimise the negative effect of overfitting. Figure 8 depicts an example of learning the concept failing student for the year 2004 with a rudimentary ID3 decision tree learning algorithm (for details, please see (Mitchell, 1997)). The experiment shows that even if blind concept learning is indeed possible (using a transparent DT learner), the naïve classification method clearly outperforms it. Thus, while the performance of recognising the failing students is better than chance, the prediction accuracy does not seem sufficient for inducing good descriptions. The situation might be improved by encoding the data differently, strengthening the algorithm, and introducing appropriate domain heuristics (Murthy, 1997). Of course, the other more powerful concept learning algorithms, such as neural networks or other equally expressive statistical methods, might perform better. It is worth noticing, however, that by definition, no statistical learning algorithm is able to demonstrate positive performance in every domain (Schaffer, 1994). 231 1,2 Accuracy 1 0,8 Decision Tree Naive Classification 0,6 0,4 0,2 21 19 17 15 13 9 11 7 5 3 1 0 Attributes Figure 8. A comparison of the prediction accuracy Conclusions We have reported a design and a demonstration of a predictive fuzzy system that enables the description of the domain objects with fuzzy descriptions, suitable for fuzzy linguistic rules and queries. We have discussed the method in the context of an educational application, providing a way to capture students with fuzzy descriptions based on accumulative attribute sequences. We have developed the notion of neutral fuzzy sets, and introduced a system of fuzzy linguistic constructors, to model the structure of fuzzy linguistic expressions in rule and query expressions and logical statements. In brief, this construction relies upon the notion of pure linguistic concepts, providing a fuzzy extension to the relationship. The design of our system demonstrates two pivotal aspects of fuzziness in applications: While we welcome fuzziness in linguistic expressions because it enables, e.g., writing useful queries using intuitive expressions, we would nevertheless appreciate our knowledge base to be completely precise. For instance, we would appreciate an exact, a priori characterisation of the poor students – which is of course impossible because such data is simply not available. Motivation for our work is due rather practical considerations. As more information becomes available in applications, it gets difficult to set up user models and formulate expressions solely using the crisp design properties and the technical terms. Developing means to capture domain objects with fuzzy descriptions provides a concrete alternative that helps managing the built-in complexity in applications. Using intuitive semantic labels and expressions in modelling should also help making more transparent systems, thus supporting the activities of the end-users more openly. This process may be perceived as an activity of constructing (fuzzy) linguistic (enduser) interfaces. From the end-users' point of view, the main contribution thus lies in the potential of using intuitive expressions, without the need of adopting expert user interfaces, or assuming a strict terminology and in-depth understanding about the encapsulated data model. Note that in order to simplify the discussion we have ignored, e.g., the issues related to contextual interpretation (e.g., bootstrapping of the constructors) and heuristic modelling (selection of the labels) in this treatment. The quality of our fuzzy system is determined by the (induced) fuzzy model and the rule heuristics. For instance, the early classifications are rather imprecise. However, since this imprecision is transparent, the approach manages to escape several problems related to crisp modelling. In particularly, the system does not force crisp nor overly precise fuzzy classifications, and the "minor" errors in the classifications are not severe, due to fuzzy rules and queries. The approach is quite general and suits the needs of various application domains that require inducing fuzzy classifications and decorating archives with intuitive labels. In fact, we have developed the notion of our predictive fuzzy systems in the context of general-purpose Semantic Web technologies (Nykänen, 2005). Finally, we wish to emphasise that it is not clear whether the introduced semantic categories should be made publicly available, even if it might support, say, the creation of certain kinds of student groups. Indeed, being "accurate" or not, people might find certain semantic labels uncomfortable or even stigmatising. In turn, this highlights the subtle issues related to user modelling in the fundamentally social context of humans. 232 Acknowledgements The author wishes to thank the teachers of the course Engineering Mathematics I, in particular Dr. Lasse Vehmanen, for the opportunity to use the anonymous records of the years 2003 and 2004 in this study. References Arriaga, F. de, Alami, M. El., & Arriaga, A. (2005). Evaluation of Fuzzy Intelligent Learning Systems. In Méndez-Vilas, A. & González, J. A. M. (Eds.), Proceedings of m-ICTE2005, June 7-10, Spain, retrieved November 24, 2005, from http://www.formatex.org/micte2005/200.pdf. Baader, F., Calvanese, D., McGuiness, D. L., Nardi, D., & Patel-Schneider, P. F. (2002). Description Logic Handbook, Cambridge University Press. Boag, S., Chamberlin, D., Fernández, M. F., Florescu, D., Robie, J., & Siméon, J. (2005) (Eds.) XQuery 1.0: An XML Query Language, W3C Working Draft 04 April 2005, retrieved November 24, 2005 from http://www.w3.org/TR/2005/WD-xquery-20050404/. Brusilovsky, P., & Cooper, D. W. (2002). Domain, task, and user models for an adaptive hypermedia performance support system. In Proceedings of the 7th International Conference on Intelligent user interfaces, San Francisco, California, USA: ACM Press, 23-30. Cox, E. (2005). Fuzzy Modeling and Genetic Algorithms for Data Mining and Exploration, USA: Elsevier. Darbhamulla, R., & Lawhead, P. (2004). Paving the way towards an efficient Learning Management System. In Seong-Moo Yoo, S-M. & Etzkorn, L. H. (Eds.), Proceedings of the 42nd annual Southeast regional conference, Huntsville, Alabama, USA: ACM Press, 428-433. Dodds, P. (2001). Sharable Content Object Reference Model (SCORMTM) Version 1.2: The SCORM Overview, Advanced Distributed Learning, retrieved November 24, 2005 from http://www.adlnet.org. Hüllermeier, E. (2005). Fuzzy methods in machine learning and data mining: Status and prospects. Fuzzy Sets and Systems, 156 (3), 387-406. Ibrahim, A. M. (2001). Assessment of Distance Education Quality Using Fuzzy Sets Model. In Proceedings of the SEFI Annual Conference, Copenhagen, 12-14 September, 2001 (in CD). Jameson, A. (1995). Numerical Uncertainty Management in User and Student Modeling: An Overview of Systems and Issues, User Modeling and User-Adapted Interaction, 5 (3-4), 193-251. Jameson, A. (2001). Modelling both the Context and the User, Personal and Ubiquitous Computing, 5 (1), 2933. Jang, J-S. R., & Sun, E. M. (1997). Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence, USA: Prentice Hall. Kavcic, A., Pedraza-Jimenez, R., Molina-Bulla, H., Valverde-Albacete, F. J., Cid-Sueiro, J., & Navia-Vazquez, A. (2003). Student Modelling Based on Fuzzy Inference Mechanisms. In Zajc, B. & Tkalcic, M. (Eds.), Proceedings of the EUROCON 2003, Ljubljana, Slovenia, 379-383. Liu, H., & Maes, P. (2004). What would they think? a computational model of attitudes. In Vanderdonckt, J., Nunes, N. J. & Rich, C. (Eds.), Proceedings of the 9th International Conference on Intelligent User Interfaces, Funchal, Madeira, Portugal, USA: ACM Press, 38-45. Lopes, A. L. M., & Lanzer, E. A. (2002). Data envelopment analysis - DEA and fuzzy sets to assess the performance of academic departments: a case study at Federal University of Santa Catarina - UFSC, Pesquisa Operacional, 22 (2), 217-230. 233 Ma. J., & Zhou, D. (2000). Fuzzy Set Approach to the Assessment of Student-Centered Learning, IEEE Transactions on Education, 43 (2), May 2000, 237-241. Mendel, J. M. (2001). Uncertain Rule-Based Logic Systems: Introduction and New Directions, USA: Prentice Hall. Mitchell, T. (1997). Machine Learning, USA: McGraw-Hill. Murthy, S. K. (1997). On Growing Better Decision Trees from Data. Doctoral dissertation, University of Maryland, retrieved June 21, 2005 from http://www.cbcb.umd.edu/~salzberg/docs/murthy_thesis/thesis.ps.gz. Nokelainen, P., Silander, T., Tirri, H., Nevgi, A., & Tirri, K. (2001). Modeling Students' Views on the Advantages of Web-Based Learning with Bayesian Networks. In Ruokamo, H., Nykänen, O., Pohjolainen, S. & Hietala, P. (Eds.), Proceedings of the 10th International PEG Conference, June 23-26, 2001, Tampere, Finland, 101-108. Nykänen, O. (2004). Fuzziness as a Recognition Problem: Using Decision Tree Learning Algorithms for Inducing Fuzzy Membership Functions. In Zanasi, A., Ebecken, N. F. F. & Brebbia, C. A. (Eds.), Data Mining V, UK: WIT Press, 143-153. Nykänen, O. (2005). Implementing Semantic Web Agents that Learn Upon Experience: A Machine Learning Module for the CWM Rule System. In Guimaraes, N. & Isaias, P. (Eds.), Proceedings of the IADIS International Conference on Applied Computing, 22-25 February, 2005 Algarve, Portugal, 409-416. Prud'hommeaux, E., & Seaborne, A. (2005). SPARQL Query Language for RDF, W3C Working Draft 19 April 2005, retrieved November 24, 2005 from http://www.w3.org/TR/2005/WD-rdf-sparql-query-20050419/. Russel, S., & Norvig, P. (1995). Artificial Intelligence: A Modern Approach, USA: Prentice-Hall. Schaffer, C. A. (1994). Conservation Law for Generalization Performance. In Proceedings of the International Conference on Machine Learning, San Mateo, Ca., USA: Morgan Kaufmann, 259-265. Stathacopoulou R., Magoulas G. D., & Grigoriadou, M. (1999). Neural network-based fuzzy modeling of the student in intelligent tutoring systems. In Proceedings of the INNS-IEEE International Joint Conference on Neural Networks, Washington, USA, July 1999, retrieved December 5, 2005 from http://www.dcs.bbk.ac.uk/~gmagoulas/JCNN0679.PDF. Stathacopoulou, R., Magoulas, G. D., Grigoriadou, M., & Samarakou, M. (2005). Neuro-fuzzy knowledge processing in intelligent learning environments for improved student diagnosis. Information Sciences, 170 (2-4), 273-307. Suarez, J. (2003). Student evaluation trough membership functions in CAT systems. Revista Mexicana De Fisica, 49 (4), 371-378. Turban, E., Zhou, D., & Ma. J. (2000). A Methodology for Evaluating Grades of Journals: A Fuzzy Set-based Group Decision Support System. Paper presented at the 33rd Hawaii International Conference on System Sciences, January 4-7, 2000, Maui, Hawaii. Turunen, E. (1999). Mathematics Behind Fuzzy Logic, Heidelberg, Germany: Physica-Verlag. Vaucheret, C., Guadarrama, S., & Muñoz. S. (2002). Fuzzy Prolog: A Simple General Implementation using CLP(R). Paper presented at the 9th International Conference on Logic for Programming Artificial Intelligence and Reasoning, October 14-18, 2002, Tbilisi, Georgia. Weon, S., & Kim. J. (2001). Learning Achievement Evaluation Strategy Using Fuzzy Membership Function. Paper presented at the 31st ASEE/IEEE Frontiers in Education Conference, October 10-13, 2002, Reno, NV, USA. Xenos, M. (2004). Prediction and Assessment of Student Behaviour in Open and Distance education in Computers Using Bayesian Networks. Computers & Education, 43 (4), 345-359. 234 Suhonen, J. (2006). Book review: Using Technology in Teaching (William Clyde and Andrew Delohery). Educational Technology & Society, 9 (2), 235-237. Using Technology in Teaching (Book Review) Reviewer: Jarkko Suhonen Senior Lecturer Department of Computer Science University of Joensuu, Finland jsuhon@cs.joensuu.fi Textbook Details: Using Technology in Teaching William Clyde and Andrew Delohery http://yalepress.yale.edu/yupbooks/book.asp?isbn=0300103948 Yale University Press, New Haven and London ISBN: 0-300-10394-8 2005, 256 pages + includes CD-ROM Overview The book “Using technology in teaching” is about using common technology to support everyday teaching activities. The authors of the book are William Clyde, dean of academic technology (finance teacher), and Andrew Delohery, director of the Learning Center (English composition teacher), both at Quinnipiac Univerisity, Hamden, Connecticut, United States. The aim of the book is to give practical ideas to instructors or teachers for enhancing their teaching with technology. The authors have divided the book into nine chapters organized around instructional goals and commonly encountered tasks and problems in teaching. Each chapter presents a different instructional activity common in many educational contexts, such as distribution of the course materials to the students, collaboration as a way to improve learning and learning through experiences. The chapters introduce several technological solutions to support the presented activities. The solutions are mainly based on technological tools that are widely used and easily available, such as emails, word processors, and Course Management Systems (CMS). Chapter Summary Each activity (e.g. chapter) is presented through several cases or practical scenarios which are common in everyday teaching. The following example is a scenario where students are contacting the instructor between classes (related to the communicating with students activity): “You have a big exam coming up in your class. You expect several lastminute questions as your students prepare for the exam. Unfortunately, you must be at a conference the two days before the exam and will miss office hours both days (p.1).”. The authors then present how to use email, threaded discussion and chat in this scenario. All the chapters in the book are organized in a similar way. Each scenario within a chapter is presented separately (or in some cases two scenarios are presented together). First, the authors present a non-technological solution to the scenario’s challenges. Secondly, usually several technology-based solutions with relation to the traditional solution are presented. Finally, potential pitfalls of the technological solutions are discussed. Chapter 1 deals with the need to communicate with students during a course. The authors stress, for instance, the value of between lectures communication. Scenarios within the chapter include issues such as making changes to lectures and students submitting course work. Technological solutions based on common email programs, threaded discussions (within a CMS tool), chat and web postings are presented. Chapter 2 covers the activity of distributing course materials to students during a course. Scenarios within the chapter consist of cases such as distributing the course syllabus, providing access to extra materials and providing graphics, videos and audio materials to students. The authors present, for instance, how to send a course syllabus written with a word processor to students via email. The authors also show how to use hyperlinks to provide supplementary material. Chapter 3 includes instructions on how to support collaboration among the students. The scenarios in the chapter deal with issues such as facilitating group work and fostering peer feedback. The authors present, for instance, ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 235 how to use discussion forums to organize peer learning activities. Other solutions presented in this chapter are based on chat, annotation and change tracking features in word processors to give precise feedback. Chapter Four of the book explores the possibilities to support learning through experience. At the beginning of the chapter, the authors give a short introduction on how a teacher can find technology tools for supporting “virtual” experiential activities. The authors present a set of tools that they have found from MERLOT (Multimedia Educational Resource for Learning and Online Teaching) database. The following tools, for instance, are presented to give an idea of what virtual environments can offer: A.D.A.M virtual learning system for learning human anatomy, Sniffy, The Virtual Rat for learning the idea of conditioning and Chemland for exploring the field of chemistry. Some of the presented tools are freely available for everyone. In Chapter 5, the authors show how to use hypertext to build and present a course with the Strategic Planning model framework. Hypertext is used, for instance, to link course activities and assessment to course objectives. The authors also show how to use hypertext to provide detailed information about different aspects of the course to the students. In Chapter 6, different solutions are discussed to improve students writing activities. The authors present, for instance, how to use commenting and highlighting functions in Word to give feedback to students. Technological solutions to detect plagiarism are also discussed. The theme in Chapter 7 is to improve students research skills. The authors acknowledge that there is a crowing need for students to be able to find and evaluate information on their own. They explain, for instance, how to use search engines, online databases and virtual libraries to find relevant literature. Chapter 8 deals with technical solutions to use assessment and feedback to enhance students learning. The authors present, for instance, how teachers can use technology to provide targeted feedback and follow-up instructions to the students with word processors and CMS tools. In the final chapter, the authors show how to use hypermedia to gather course materials in the course. The authors also discuss how students can use technology (mainly web searches) to find relevant information for a project work. At the end of the book, there is an appendix that includes a short introduction to copyright laws in U.S.A. The appendix also includes a list of plagiarism detection services available in the net. The final part of the book is a glossary including all the technical terms used in the book. Evaluation The authors take an interesting approach by presenting technological solutions through practical cases and scenarios. This gives a contextual nature to the theme of the book; how to apply technology to support teaching activities. The authors have very positive attitude throughout the book. One can sense that they really appreciate the possibilities of technological solutions. The book includes lots of practical examples and solution alternatives to show that technology can be applied by teachers who master the basic technology. The technology does not exists on its own right, but it is applied to overcome real problems and challenges in teaching. The real need and meaningful use of technology is emphasized throughout the book. All the concepts and terms are explained carefully. All new terms are underlined like links in webpages and there is a separate explanation box for the term in the same page. Furthermore, the authors have gathered a comprehensive glossary of relevant technological terms at the end of the book. The text was also easy to read and follow. The authors use nontechnical style avoiding confusing ICT jargon. Various snapshots taken from different applications and tools clarify the authors message. The structure of the book is clear because all the chapters follow the similar structure as explained earlier. However, because same technology is applied in several scenarios there is some repetition. In my opinion, Chapter 9 does not include any new perspectives. The authors could have distributed the content of the chapter to other chapters. The main shortcoming of the book (to a person working in educational technology with computer science background) is that it does not progress from commonly used tools, such as word processors, emails and CMSs, to more advanced technologies. This is, however, acknowledged by the authors already in the introduction part of the book. “This book is not for the technology innovators, nor it is for the skeptics. It is for the 75 to 80 percent in the middle: the mainstream faculty who are using some technology..”, authors point out in the very first page of the book. For me, the most interesting part of the book was Chapter 4. It included several examples of more advanced technologies, such as tools for experimental learning (e.g. virtual reality), role playing via the net, and problem solving (for instance Mike's Bikes – Advanced; simulator for experiencing the creation of business strategies). The chapter also includes a short introduction to the content of MERLOT database for searching meaningful tools within different disciplines. I also found some minor problems and improvement areas in the book. First, the amount of figures is even too high. Although the figures support the text nicely, the authors could have decreased the amount of snapshots at some parts of the book. This would have given the opportunity to deepen the discussion to more advanced use of 236 the technology. I also found that the authors explained some basic functions of the tools too carefully, such as saving a picture from a website. Some figures are also blurred (e.g. Figure 4.11, Figure 4.2a-b), it was difficult to see the content of the figure clearly. Furthermore, in some figures the authors use annotation to clarify the meaning of some functions. The black text inside the annotations is placed in a dark background making it hard to read (e.g. Figure 6.2, Figure 6.3). Although every technological solution includes a short discussion on potential pitfalls, the authors could have deepen the discussion in this section. In many cases, they mention that the main problem is that students are not able use or do not know how to use the presented technology. For instance, in Chapter 8 the authors explain that the main pitfall with online surveys is that students do not know how to access or use the available survey tool (p.192). I would have expected more discussion about technology per se. For instance, in my experience the discussion forums in many CMS tools are often limited. They do not support, for example, graph-like presentation of the discussion threads. In some cases, it is also difficult to give precise feedback using the discussion forum. We have found, for instance, that commenting a piece of programming code with a common discussion forum can be a problem. It would have been useful to know, for instance, how the authors would develop these tools to make them even more useful or what needs cannot be fully met with available technology. The supplementary CD-ROM was aimed to improve the understanding of technology presented in the book. It included video snapshots of tools discussed with audio explanation. In the CD-ROM, the authors present, for instance, how to set up discussion threads in the Blackboard CMS tool. In my opinion, the CD-ROM does not provide much extra value to the content of the book. I would recommend this book to a teacher who has not been working with the technology, but would like to enhance his/her teaching through the technology. Furthermore, the book might be used in teacher training to show how to apply technology in everyday teaching context The aim of the book is to give readers ideas of possible technological solutions, and to encourage them to look for appropriate tools in one’s own discipline. In my opinion, the book succeeds in this goal. If you already master the basic technology or work with educational technology, this book is not for you. At least for me, the book did not provide much new ideas or knowledge. The authors also take a contact teaching perspective, there is no discussion about possible problems and challenges in distance learning (although many of the presented solutions can be easily applied in the distance learning context). The book also serves a good reference in basic educational technology courses as an example of utilizing common technology in everyday teaching. 237 Malinski, R. (2006). Book review: Unlock the genius within; neurobiological trauma, teaching, and transformative learning (Daniel S. Janik). Educational Technology & Society, 9 (2), 238-240. Unlock the genius within; neurobiological trauma, teaching, and transformative learning (Book Review) Reviewer: Richard Malinski Ryerson University, Canada richard@ryerson.ca Textbook Details: Unlock the Genius Within: Neurobiological Trauma, Teaching, and Transformative Learning Daniel S. Janik, MD, PhD Rowman & Littlefield Education Paper 1-57886-291-4 September 2005 230pp http://www.rowmaneducation.com/ISBN/1578862914 Introduction There has been a great expansion over the last decade of neurobiology through the use of progressively more sophisticated imaging technology. With these imaging techniques clearer links are being drawn between our physical, rational, and emotional functioning and specific and/or multiple brain locales. This book surveys some of this recent research, draws together insights from the author’s medical and educational experience, and outlines a foundational learning theory. Because of this breadth of scope, the book is both exciting and frustrating to read, not to mention review. The author sets out a daunting set of goals. Specifically he states that this book is; ¾ an exploration of the principles underlying exceptional, effective, traumatic learning; ¾ a sketch of neurobiological theory that makes sense of it all; ¾ a description of hypothetical deductions which with attendant research should refine the theory and method; and ¾ the development of a framework within which basic principles of effective, non-traumatic learning may be successfully applied. (Janik, 10) This tantalizing set of goals should attract educators interested in theoretical discussions, language teachers because of the English as a second language (ESL) context, and teachers searching for process and technique to help their classes. Content ‘Unlock the genius within’ has, after the preparatory sections, ten chapters with a number of binding threads which run them. The chapters are devisable into three major groupings; supporting material, theory, and promotion. The supporting material is in the first seven chapters. This covers discussions of the need for ‘Yet another teaching theory’ (a chapter title), the author’s idea that ‘It all begins with traumatic learning,’ the significance of ‘What the body tells us,’ the importance of ‘What machines are saying about us,’ the integrative approach of ‘The neurobiological way,’ and the value of a neurobiological theory basis for learning that is non-traumatic, non-violational, and is direct (scientific) and founded on physical (neurobiological). The author weaves an integrated philosophical and physical net to contain his neurobiological theory of learning. This reviewer found these chapters full of intriguing details of brain functioning, insights into trauma cases from the author’s medical experience, and contrarian views of traditional education. The technical jargon and the lack of clear definitions may be a hurdle for some readers. While much of the material is not new, the comments on education and the suppositions on the physicality of learning are intriguing and fascinating. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org. 238 It is in this supporting group of chapters that four main threads or themes become evident. First, right from the beginning the author pleads for a return to the Socratic method of teaching or the use of a mentorship approach in learning. This method contrasts with the Platonic approach which is ideational and currently the mainstay of our schools. The author pleads that this is unfortunate for our students because while it is effective it is traumatic! The author’s disdain for traditional teachers and teaching seems overly harsh and not likely to engender a second look. Second, the author draws numerous links between the physical workings of the brain viewed through imaging technology and learning. Much of this is reported in the literature but some seem like leaps of faith. For instance, literature links myelination and the thickness of myelin with intelligence but not specifically with learning. What direct learning techniques would be most beneficial during the periodic process of myelination and/or the types of myelin present? Do these periods commence at the same time for everyone? Practical examples and some definitions would have been useful here. Another instance concerns birth. One might agree that the birth event is a traumatic experience but what is actually learned by the infant and how do we know, as the author suggests, that the thalamic gateway is active in learning at that moment? In a later chapter the author states that ‘…learning after birth is driven first and foremost by the raw power and form of the traumatic birth event’ (p 141) but doesn’t elaborate what this power and form are. Some of the difficulty here may be a result of semantics. The author does not define learning or thalamic learning or explain how one might evaluate or understand what is learned or what is occurring. As a result the reader might be at a loss as to what is actually being suggested. Third, the main premise of the author is that understanding trauma and its significance to learning is essential if we are to break with our traditional traumatic teaching approach. The author makes it clear that traumatic events produce learning that results in triggers and associations that can be destructive to the individual. The strength with which the author calls for a change to this traditional traumatic teaching seems overdone and counterproductive. The author suggests that teachers are not sufficiently learned in the workings of the brain to make this trauma-teaching association but if they were they would cease and desist from their approach. Fourth, the author talks about mentorship as a viable alternative to today’s teaching environment. His view might be encapsulated in the desire to move from a sage-on-the-stage approach in which teachers simply dump the structured information into their students (the empty vessels they are!) to a guide-on-the-side in which the mentor lets them find their own way but is there to provide assistance when asked. The first seven chapters can be seen as an attempt to provide background information in traumatic learning and, to promote imaging technology as a useful tool in understanding the development of the brain with its particular responses to learning situations, and to foster the author’s practical, physical view of effective learning. The second unit in this book consists of the eighth chapter, ‘Neurobiological learning’ and contains the kernel of the author’s view. The author defines two key terms and sets out seven tenets or laws. First, neurobiological learning (NL) is at the core of the author’s thesis and ‘…occurs in discrete steps, phases or levels, beginning with object-data and proceeding through association, symbolism, interpretation, and cognition to metacognition and possibly beyond.’(p141). Second, transformative learning (TL) is learning that is ‘volitional, curiosity-based, discovery-driven, and mentor-assisted’ (p144). The author maintains that ‘The physical evidence speaks strongly in favor of the existence of a single unifying theory of neurobiological learning that in its application follows two pathways – traumatic and transformative learning’ (p 146). We are to avoid traumatic learning with all is negative triggers and associations, and strive for TL. These are not particularly new ideas. In essence, we need to understand what learning is with NL being another tool helping us approach learning. The author then provides a set of seven tenets which together provide the ‘natural laws’ of neurobiological theory and method and transformative learning. Many of these would be recognized and understood by good teachers anywhere. First, we must assist learning by demonstrating curiosity, discovery, and self-discovery and not just explaining how to learn. Second, using TL techniques creates self-sustaining success and interest in learning within the student. Third, effective learning is not localized in classrooms as schooling at home or distance learning can show and so we need to think in broader terms about learning. Fourth, effective learning results when students noticing something of interest bring it into central focus where they can discovery and reflect on it. The mentor’s role is to provide a resource rich environment and the occasional guidance for the student as he or she progresses through the levels of NL as (noted above). Fifth, building upon such tools as listening, language, and visualization the use of rhythm (music) and rhythmic patterns (stages in TL and/or cycles of brain development) can promote structure and success in learning. Sixth, mentors are at the core of the 239 tenets. This tenet links closely with the others in emphasizing the resources richness of the learning environment and the laissez-faire approach of TL. At the same time there is a need for mentors (teachers don’t seem to have this ability!) to be aware of and assist the learners to sense their feelings, be aware of their cyclic rhythms and internal time consciousness. Seventh, this last states that ‘neurobiological method underlies all teaching and learning methodologies’ and ‘reasons, rules, meanings, sense, and truth are defined by the learner.’ Here again is a call to understand brain development cycles and merge this with TL techniques to allow, as constructivism suggests, for the individual to build his or her knowledge. This chapter is the summation of the exploring, sketching, and describing as well as the development of a framework within which to work. The two definitions and the seven tenets integrate many concepts and processes with which some traditional teachers are already familiar but the integration of neurobiological components with learning should prove new and insightful to many more. The third and last group of chapters, chapter nine and ten, consists of the author’s assessment of the success of his approach in teaching ESL at his college and an invitation to further discussion. The chapters might be called promotional because of the laudatory nature of the comments about the college and the enthusiasm being built for NL and TL training and certification through the ESL college. There is some description of the use of the author’s method at the college but insufficient for any real understanding of techniques and how they might apply to other situations. Concluding comment The author alludes to the need for further work in solidifying the links between learning and brain imaging and in firming up his tenets. These refinements are essential to help explain how NL and the TL techniques introduced can be used by others. How might one determine the rhythmic patterns of students and once this was done determine what strategies, tactics, and data-objects might be appropriate? How this might be done for many students as opposed to those one or two mentored learners would be most useful. Resource richness is always a concern of teachers but in the case of TL mentors what resources need be in place? These are issues that the author needs to spend a great deal of time detailing. He has the odd example but much more needs doing. The author’s next book which deals with the application of NL/TL to distance education needs to focus more on tactics and concrete illustrations. The overall result of all this material in these few pages is a very choppy writing style with the focus jumping back and forth over the major themes all the while loading on more and more information some as factual material and other as supposition. There are many terms that need definition and/or concrete illustrations. The index is overly complex and not without error. There is some summary and direction at the ends of chapters but the reader would have benefited from a variety of graphics and/or summarizing lists as guides. While the information is rich the visual component is not! There is a sense that the natural laws of NL/TL, as drawn from the author’s experience of trauma cases and ESL and his outlining the links between brain activities and learning language, are universal and can be applied broadly. Is this approach tenable? The extrapolation from trauma cases with their negative learning triggers and associations through brain imaging with its views of learning and from traditional education to the application of NL/TL in ESL are fascinating and worth discussion. However, can we rely, as the author does, on these as essential links and applicable to learning in general? Is trauma in our education system as negative and pervasive as the author details? Is the work on ESL applicable to other types of learning? Are the tenets proposed within NL/TL sufficiently robust to be the basis of a learning theory that makes sense of it all? 240