Research Visit Presentation

Institute of Information Systems and New Media, WUW Lunch-Talk, OUNL, Heerlen, 13.11.2006 Fridolin Wild Vienna University of Economics and Business Administration My Plan: One Nervous Breakdown Here in this Room ƒ Technology-Enhanced Learning at my University: learn@WU ƒ Technology-Enhanced Learning in Austria and (mainly) New Europe ƒ Research at the Institute for IS & NM ▪ Short iCamp Introduction ƒ Two Hot Topics we are Researching in ▪ TEL Interoperability ▪ Automated Competence Assessment (ACA) ▪ Two Examples for ACA <2> TECHNOLOGY-ENHANCED LEARNING AT THE WUW: LEARN@WU <3> Key Facts Learn@WU ƒ Initial Project: ▪ Start: Autumn 2001, 2 Years, Budget: 3.4 M € ▪ 36 Full Time Content Developer (2 per Course) ▪ 2 people didactic support, 2 people technical support (incl. help desk) ▪ Content (not platform) Project ƒ From Project to Infrastructure: ▪ ▪ ▪ ▪ 2002: Deployment of First Version based on OpenACS 2003: TEL became a Strategic Goal of the University 2004: Relaunch based on dotLRN (+ own Components) Since 2005: • TEL is part of Trainee programs • Development of an In-House e-Learning Academy • Currently 48 People employed • More than 250 Content Developers <4> Current Status learn@WU ƒ More than 37.000 Learning Resources developed ƒ Broad Acceptance ▪ ▪ ▪ ▪ More than 2.000 courses More than 29.000 registered members (mostly students) Students solve up to 380.000 interactive exercises per day More than 120.000 exams through mark-reader ƒ Technical Figures ▪ Up to 4,3 Mio requests (hits) per day from registered users ▪ Average response time less 0.4 sec ▪ Up to 41 GB/Day traffic ƒ Current annual growth rate: 10-20% ƒ One of the most intensively used TEL platforms world-wide „Without Learn@WU, the operations of our university would not have been possible.“ – Christoph Badelt, WUW President <5> Collaborative Learning and Teaching Environment ƒ Community Framework ▪ University as a „community of communities“ ▪ Communities composed of • Groups of students, classes, courses, programs, alumni, ... • Members and administrators (decentralized management) ▪ Communities are provided with tools ▪ Administrators tailor communities according to their needs ƒ Collaboration and Teaching Tools ▪ General Collaboration Tools: Calendar, Announcements, Chat, Forum, File-Store, Weblog, Wiki, ... ▪ Teaching Tools: Syllabus, Homework, Problem Based Learning, Room Reservation, ... ▪ Decentralized Management: e.g. teacher configures a class community with tools suitable for his teaching concepts <6> Snapshot <7> TECHNOLOGY-ENHANCED LEARNING IN AUSTRIA AND (MAINLY) NEW EUROPE <8> Open Questions ƒ Characteristics of Learning Tools Portfolio? ƒ Support for Collaboration and Social Awareness? ƒ Sharing & Re-Use? ƒ In Austria? ⇒ In Comparison to Other Countries? <9> Methodology ƒ Descriptive Study (Picciano, 2004) ƒ Part of Study on Higher Education Institutions in (mainly) New Europe ƒ Performed by iCamp & ProLearn ƒ Overall: 26 countries (varying completeness) ƒ 44 Questions in 7 groups, breaking down to 118 variables ƒ 23 responses from Austria, 100 overall ƒ Developed: Jan/Feb 2006 http://www.prolearn-project.org/ ▪ Pre-Test ▪ Analysis through Advisory Board ▪ Adaptation by Core Team ƒ Conducted: Mar-July 2006 http://www.icamp.eu/ <10> Contributing Countries <11> Characteristics (Tools Portfolio) L(C)MS: Blackboard, CeWEbs (S), CIS (S), DMA, Dynamic PowerTrainer, ELGG, eNcephalon (S), Hyperwave, Ilias, KUG-Online, Learn@WU (S), lerndorf (S), Moodle, MS Class Server, n.n. (S), n.n. (S), n.n. (S), n.n. (S), n.n. (S), Portal (S), TUWIS (S), Virtual Medical Campus (S), VUW++ (S), WBT Master (S), WebCT Learning (Content) Management … Moodle 12 (Moodle + Other 9) (Moodle + Commerc. 5) Pure AUTH: cmap, Dreamweaver, exe, Framemaker, Hot Potatoes, nvu, xmlSpy Pure ASS: DHS, Speedwell Question Bank, Forms 5, n.n. (S), n.n. (S) Pure LOR: COL learning object repository, Cumulus, easyDB, EducaNext (S), HCD-Suite (S), IAEM (S), MBOX, n.n. (S), pakXchange 1 2 WebCT Blackboard Hyperwave 3 3 2 Institution has (one or more) … Pure AIS: BACH (S), LPIS (S), i3v, n.n. (S) Open-Source LMS Self-Developed LMS Commercial LMS Pure CMS: aloha (S), communicom, Drupal, ELK - CMS (S), ePrints, Plone, Stream Server, Typo3, Wordpress, XIMS (S) Pure COLL: Breeze, BSCW, Campus Pack for Blackboard, CGI:IRC Chat, concert chat, digalo, Interwise, mediaWiki, phpBB, Quicktopic, Tikiwiki, Video Conferencing Software, Virtual Network Computing - VNC, XchangeBoard .LRN Ilias 14 14 11 L(C)MS CMS AIS AUTH LOR ASS COLL Types 25 10 4 7 9 5 14 Occurences 43 13 5 7 9 5 20 Self-Dev 15 3 3 0 4 2 0 <12> Characteristics (Tools, Overall) <13> Characteristics (Users) Registered Users Arithmetic Mean: 6894 Standard Deviation: 13878 Active Users Arithmetic Mean: 3988 Standard Deviation: 6717 Registered Users Arithmetic Mean: 5236 Standard Deviation: 8806 Active Users Arithmetic Mean: 2828 Standard Deviation: 4643 <14> Characteristics (Courses) Courses in System (n=13) Arithmetic Mean: 501 Standard Deviation: 582 All Courses (n=12) Courses in System (n=79) Arithmetic Mean: 242 Standard Deviation: 401 All Courses (n=63) Arithmetic Mean: 1234 Arithmetic Mean: 1020 Standard Deviation: 1686 Standard Deviation: 1987 <15> Characteristics (Summary) ƒ Austria ▪ Biggest 3 LMS (Users): ƒ Overall Study ▪ Biggest 4 LMS (Users): • Blackboard • WebCT • learn@WU/.LRN • Eleum or Moodle • Blackboard • learn@WU (.LRN) • WebCT ▪ Moodle as the only LMS: • Ø 663.07 users • Max 3600 users ▪ All Moodles in Survey: • Ø 1800.73 users • Max 28.500 <16> Collaboration & Social Awareness <17> Collaboration (Usage Frequency) = Overall = Austria Indicates lack of support for (social-)constructivist education theories But: better than in overall study Var1 Var2 1 frequent use Course Management 2 moderate use Course Management 3 infrequent use Course Management 4 experimental use Course Management 5 never Course Management 6 no answer Course Management 7 don't know Course Management 8 frequent use Authoring 9 moderate useRare authoring Authoring in hand 10 infrequent usegoes hand Authoring 11 experimental usewith rare Authoring 12 nevercollaboration Authoring 13 no answer Authoring 14 don't know Authoring 15 frequent use Delivery 16 moderate use Delivery 17 infrequent use Delivery 18 experimental use Delivery 19 never Delivery 20 no answer Delivery 21 don't know Delivery 22 frequent use Collaboration 23 moderate use Collaboration 24 infrequent use Collaboration 25 experimental use Collaboration 26 never Collaboration 27 no answer Collaboration 28 don't know Collaboration Overall Austria 54 52 18 17 18 26 5 4 0 0 5 0 0 0 25 17 22 13 19 22 13 30 2 0 13 4 6 13 49 48 19 26 16 17 6 4 2 0 6 0 2 4 22 22 25 35 22 22 14 13 1 0 12 9 4 0 <18> Coll. & Soc. Awareness (Opinions) = Overall Material sharing generally considered important. = Austria More agreement to have a big tools portfolio for instructors to choose from. Shows demand for support (social networking functionality supported by tools portfolio: 30% Austria, 34% Overall)! Opinions vary about tools portfolio sizes for students. <19> Repository Interoperability LOR accessible from outside (18/100) Cross-organisational repository network (16/100) But still …! Austria: Unis (11) Colleges (12) Cross-Organisational Repository Network 4 1 Repository is accessible 3 0 But still …! 81% U! 67% C! Primary Mode for Sharing of Learning Resources <20> Summary Study (I) ƒ Quite Heterogeneous Landscape of Tools: ▪ 182 different tools (100 Universities!) ▪ In 290 installations ƒ Strengths ▪ In General: • Text-based Communication, Assessment Features • Quality Assurance, Individual Publishing Features ▪ In Austria: • Stronger in Collaborative Publishing Features • More than 50% (Overall: 46%) have more than 60% of their students registered in the systems • In average more than double as many TEL-enriched courses than Overally Attention: compared against only (mainly) New Europe! <21> Summary Study (II) ƒ Shortcomings ▪ In General: • Social Networking Features (but: judged very important!) • Interoperability of Repositories (but: judged important!) • Collaboration – Features missing – e.g. AV-Broadcasting / Conferencing – Existing Collaboration Facilities rather rarely used ▪ Additionally weaker in Austria: • Features for Authoring of Learning Designs <22> RESEARCH & RESEARCH PROJECTS <23> Research Headed by Gustaf Neumann ~ 33 people (plus several open positions) <24> Research Projects ƒ Past Projects ▪ Universal, Elena, TEN-A, … ƒ Current Bigger Projects ▪ ▪ ▪ ▪ Prolix (IP, IST) ProLearn (NOE, IST) iCamp (STREP, IST) E-Learning / e-Teaching Strategy (bm:bwk) <25> iCamp Vision … … to become THE Educational Web for Higher Education in an Enlarged Europe of 25+ <26> Organisation of Work … later: competence assessment … now :) … beginning: survey on tools … later: interoperability <27> Subject - object relationships Competence Achievement in … Areas of challenge... <28> augmented landscape Missing link from the models to the infrastructure: … activity theory <29> SELECTED HOT TOPIC IN MORE DEPTH: TEL INTEROPERABILITY <30> TEL Interoperability ƒ Repository Interoperability ▪ Retrieval Interface (SQI) ▪ Federated Search for Digital Libraries / Learning Object Repositories ▪ Aggregation / Mediation Services ƒ Next Step: Collaboration oriented Interoperability ▪ Blog interoperability ▪ Aggregation ▪ Rip, Mix & Feed Steering Interfaces <31> Interoperability is a property that emerges, when distinctive information systems (subsystems) cooperatively exchange data in such a way that they facilitate the successful accomplishment of an overarching task. <32> (modified from Kosanke, 2005; IEC, 2005) Concept of Interoperability <33> Theoretical Approaches ƒ Information Integration & Dissemination for Learning ▪ Data Integration vs. Data Exchange ▪ Information Querying ▪ Information Filtering ƒ Remoting: Service Orientation for Learning Services ƒ Presentation Integration: Portlets ▪ i.e. the learner‘s front-ends ▪ e.g. Web Services for Remote Portlets(WSRP) <34> <35> Interoperability Stack: Combines Remoting and Information I & D <36> Interoperability: Retrieval via SQI <37> Federated Search <38> Patterns <39> Examples for Patterns <40> Next Step ƒ Integration ▪ of federated search („retrieve“) technology ▪ with feed structures („publish/subscribe“) ƒ i.e. Integration ▪ of ad-hoc retrieval ▪ and information filtering (standing queries) <41> From Federation to Information Integration & Dissemination <42> <43> Search is an Iterative Process begin Turn into Search Portlet to encapsulate end-user functionality in an interoperable building block Query Formulation Processing Results Display Reformulation Evaluate Results yes no Satisfied? end <44> SELECTED HOT TOPIC IN MORE DEPTH: AUTOMATED COMPETENCE ASSESSMENT <45> << … just a selection … The History of Competence >> <46> Definition “A competence is defined as the ability to successfully meet complex demands in a particular context through the mobilization of psychosocial prerequisites (including both cognitive and noncognitive aspects)” (Rychen & Salganik, 2003b, p. 43) <47> Definition (II) ƒ Competence is ƒ a human potentiality for action, ƒ which is: ▪ Demand oriented (= abilities required for e.g. task) ▪ Refers to abilities that can be learned ▪ Involves cognitive and noncognitive elements • factual knowledge, procedural skills, internalised orientations, values, attitudes, volitional aspects, … <48> Competence Classes (I) ƒ Excerpted from empirical, political, and theoretical perspectives (see paper) … ƒ Professional competence ▪ basic and specialized general knowledge, basic psychomotor and mechanical skills, and disciplinary and interdisciplinary knowledge (Jäger, 2001) ƒ Methodological competence ▪ ability to independently acquire, structure, critically evaluate, and exploit knowledge in a creative way (Kauffeld et al., 2003) <49> Competence Classes (II) ƒ Social Competence ▪ facilitate communicative and cooperative action and that aim at identifying, managing and mastering conflicts (Erpenbeck, 2003) ƒ Personal Competence ▪ concerned with those attitudes and character attributes required to perceive and utilize one’s own competencies and to act in a reflective and self-reflective way (Erpenbeck, 2003) <50> Important Competences <51> Automated Measurement ƒ Four Different Types of Approaches ▪ Multiple-Choice Approaches ▪ Simulations • Virtual labs, online experiments, games • From simple click-thru to sophisticated MM • Underlying model used to evaluate performance ▪ Graph-Based Approaches • Based on formalisms such as: concept maps, knowledge maps, mind maps, topic maps, ontologies, Petri nets, adjacency networks, and affiliation networks (plus many others) • Mining approaches (e.g. SNA on eMail interaction) • Construction approaches (fill-in-the-map vs. construct-a-map) <52> Automated Measurement (II) ƒ Natural Language Processing Approaches (NLP) ▪ Syntax-based: structural analysis regardless meaning • Shallow counting (orthography, e.g. Page, 1966) • Structural Analysis (e.g. POS-tagger & discourse structure parser) ▪ Semantics-based: analysis of the meaning • Concept-based • Context-based <53> State of the Field… ƒ See paper for more ƒ But: Two examples ▪ Social Competency Aspects measure with SNA • over interactions within learning communities • within scientific communities ▪ Professional Competence measure with LSA • Essay Scoring <54> COMPETENCE ASSESSMENT EXAMPLES FOR AUTOMATED <55> SNA over Discussion Boards Data message_id forum_id parent_id message_id author forum_id parent_id \N author 130 2853483 2853445 \N 2043 60 734569 31117 2491 131 1440740 785876 \N 1669 221 762702 31117 132 2515257 2515256 \N 5814 317 762717 31117 762702 1927 133 4704949 4699874 \N 5810 1528 819660 31117 793408 1197 134 2597170 2558273 \N 2054 1950 840406 31117 839998 1348 135 2316951 2230821 \N 5095 1047 841810 31117 767386 1879 136 3407573 3407568 \N 36 2239 862709 31117 137 2277393 2277387 \N 359 2420 869839 31117 138 3394136 3382201 \N 1050 2694 884824 31117 139 4603931 4167338 \N 453 2503 896399 31117 862709 1982 140 6234819 6189254 6231352 5400 2846 901691 31117 895022 992 141 806699 785877 804668 2177 3321 951376 31117 142 4430290 3371246 3380313 48 3384 952895 31117 951376 1597 143 3395686 3391024 3391129 35 1186 955595 31117 767386 5724 144 6270213 6024351 6265378 5780 3604 958065 31117 145 2496015 2491522 2491536 2774 2551 960734 31117 146 4707562 4699873 4707502 5810 4072 975816 31117 147 2574199 2440094 2443801 5801 2574 986038 31117 862709 2043 148 4501993 4424215 4491650 5232 2590 987842 31117 862709 1982 1 \N 1982 862709 \N 2038 5439 \N 5174 \N 716 862709 \N 1939 584 <56> SNA over message boards ƒ Message Board: Business English ƒ Most central Author 1083 (Highest Degree Centrality, Highest Betweenness) ƒ => a student! n 6 5 4 3 k 2 2 2 2 no groups 0 2 64 2691 Calc‘ed with k-plex: - n: number of members to be connected with - k: number of members no connection is neccessary <57> SNA over scientific community author Neuman Simon Scharl Wild Treiblmaier Mendling Zdun Olemedilla Madlberger Kieslinger local centrality 188 139 79 77 67 63 57 49 44 44 <58> Essay Scoring with LSA <59> Folding-In in Detail (cf. Berry et al., 1995) mi = Tk S k d iT 2 Mk vT (2) convert „Dk“-format vector to „Mk“-format Tk (1) convert Original Vector to „Dk“-format Sk Dk d i = vT Tk S k−1 1 <60> Essay Scoring (Code) library( "lsa“ ) # load package # load training texts trm = textmatrix( "trainingtexts/“ ) trm = lw_bintf( trm ) * gw_idf( trm ) # weighting space = lsa( trm ) # create an LSA space # fold-in essays to be tested (including gold standard text) tem = textmatrix( "testessays/", vocabulary=rownames(trm) ) tem_red = fold_in( tem, space ) # score an essay by comparing with # gold standard text (very simple method!) cor( tem_red[,"goldstandard.txt"], tem_red[,"E1.txt"] ) => 0.7 <61> Benchmarking Effectiveness ƒ Compare Machine Scores with Human Scores ƒ Human-to-Human Correlation ▪ ▪ Usually around .6 Increased by familiarity between assessors, tighter assessment schemes, … ▪ Scores vary even stronger with decreasing subject familiarity (.8 at high familiarity, worst test -.07) ƒ For the whole essay collection from the last slide: Rho = 0.687324, compared to pure vector space model with Rho = 0.4475188 <62> BEWARE THE END IS NEAR. <63>