How To Measure Innovation in Elearning The I-Afiel Methodology
How To Measure Innovation in Elearning The I-Afiel Methodology
How To Measure Innovation in Elearning The I-Afiel Methodology
Lucilla Crosta, Kelidon Association, Italy Victor Prieto, Cevalsi, Spain Summary This paper presents and discusses innovation in relation to the European project Innovative Approaches for a Full Inclusion in eLearning (i-AFIEL) and, in particular, the methodology applied to measure innovation in eLearning projects developed by the Spanish OVSI Foundation. The paper focuses on the concept of innovation and the three key aspects related to the eLearning context: technological innovation, sociological innovation and service customization. A series of Innovation Criteria are proposed and discussed as elements to be measured in order to understand the degree of innovation of each eLearning project. The challenge of the iAFIEL methodology is the transformation of these indicators into numbers and scores, and thus to evaluate quantitatively the success and the novelty of the projects. The evaluation system involves the transformation of the Innovation Criteria into Indicators, and the transformation of the latter into questions and tools addressed to two different target groups, a students' questionnaire and an organization's questionnaire, allowing to have broader and more objective views and measurements. Some examples of the application of the i-AFIEL methodology are provided in the paper with some final reflections and conclusions about the need of further improvements and development of the methodology itself. The i-AFIEL model can be considered in deed a concrete example of how a research study in eLearning may be carried out in a creative way. Keywords: e-learning, i-AFIEL, innovation, methodology
1 Introduction
This paper is aimed to present the European project entitled Innovative Approaches for a Full Inclusion in e-Learning (i-AFIEL) and to describe how creativity and innovation can find an example of good practice and application in the project itself. Nowadays, stakeholders, educators, content providers, software providers and e-learning developers have come to understand that for e-learning, in order to be effective and inclusive for all, it is crucial to create awareness around the added value which it brings with it, which does not mean simply transposing face-to-face classes online, on the Internet or on a CD Rom. E-learning should bring novelty, should support users creativity, active participation, relationship building, connections, information sharing, collaboration inclusion and should innovate the old educational frameworks. Everything still needs to be reinvented. The i-AFIEL project was carried out during 18 months in 2007 and 2008, by seven European institutions, from three different countries: Spain, Italy and UK. The main aim of the project was to seek out best practices and new ideas to make e-learning possible for people who for socioeconomic reasons, age, disability or any other marginalising factors have not found ICT to be a part of their daily lives yet.
The i-AFIEL project 1 can be considered as a concrete example of innovation and creativity in elearning 2 for a full access and inclusion for all. In this context innovation is described as:
a concept which should amalgamate both technological novelties (tools, programs, hardware, etc.) and sociological novelties (target audience, social integration, social interaction) and also the improvements in the quality of the service (educational improvement, learning support, teaching support, etc.) 3
A clear reference is here made to all learning 2.0 technologies, (social networks, wiki, weblogs, etc.) which, in a very simple way, can easily and constantly connect users with each others, allow them to share and create new contents and which make them their own actors of the learning process. The project was also aimed to build a methodology to measure innovation in e-learning. However, the methodology should be understood in the context it was created: its goal is to measure innovation based on the projects presented by the partners (see table 1) and it is not enough prepared to measure all e-learning initiatives. For this the methodology should be updated to include the latest trends on e-learning and also adapted to the different situations which should be measured. As first premise, this methodology should be understood as a starting point for the measurement.
i-AFIEL project www.iafiel.gva.es For definition of e-learning concept see for example OECD's report eLearning in tertiary education: where do we stand? (2005) http://www.oecd.org/dataoecd/27/35/35991871.pdf and European Commission's eLearning Action Plan (2001) http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2001:0172:FIN:EN:PDF 3 i-AFIEL guidelines www.iafiel.gva.es
2
eLearning
Innovation
Results
Technological innovation
Sociological innovation
Service customization
Project achievements
Student satisfaction
Short description The Qual-ELearning project deals with quality of eLearning providing European professionals from the creation to the application of training programmes with knowledge, tools and methods to improve evaluating the effectiveness and impact of training cycles.
Link www.qual-eLearning.net
The BENVIC project is aimed at developing standards for www.benvic.odl.org/indexpr. html quality assessment of the European virtual campuses and quality criteria of this type of training. www.ihep.com/organizations. php3?action=printContentIte m&orgid=104&typeID=906&it emID=9239
Quality on the Line. This project examines case studies of six colleges and Benchmarks for universities which provide Internet-based degree success in internet programs. The final outcome is a list of 24 benchmarks. based distance education. DLAE (Distance learning accreditation in Europe)
The SIG-DLAE project aims at developing a model similar http://dlae.enpc.fr to the U.S. Accreditation Model (DETC), but adapted to European experiences and practices. www.adeit.uv.es/mecaodl/
MECA-ODL The aim of the project is to identify and analyse different Compendiun of quality criteria of Web-based distance learning. reference materials on quality in open and distance learning delivered via internet Distance education DETC promotes the use of quality standards of distance and training council learning and offers official accreditation to those (DETC) accrediting institutions which comply with them. commission Information advice and guidance
www.detc.org
Provide Deaf people with access to information through www.walsalldeaf.org.uk Deaf advisers that Walsall Deaf Peoples Centre (UK) won support from the UK government to develop the learning package NVQ2/3.
E-learning for adults The project consists in establishing a partnership medium www.elfora.net where best practises related to vocational systems, eLearning, online courses, software platforms, etc. can be shared. ELBA Electronic book for adults The project is aimed at elaborating an electronic version of a European social anthology in an electronic book format which will be incorporated into some digital libraries and which will be accessible to anyone through the internet. The project aims at using Internet to help adult learners get access to basic education and developing subsequent lifelong learning activities, while improving their personal and/or professional skills. The project aims at developing a complex training tool enabling secondary school teachers to rise the pupils interest on science and technology by promoting sustainable energy in practical-orientated training. www.elbaproject.net
www.sec.ro/iall
www.unitbv.ro/Default.aspx?t abid=561
Objectives of the online formation are the improvement of http://cirm.sirse.net/fad/corsi/ the radioprotection acquaintances to fine radioprotection 1_Radio_MAC.pdf preparation of the professionals who follow curative performances or diagnostic surveying which imply the
use of ionizing radiation. eLane eLearning European Latin America New Education CompeTIC Provide society with high quality & low cost educational material, maximizing impact both at academic and non academic fields. Focus on the support of educational programs at isolated Latin-American communities through distance learning systems. CompeTIC objectives are to provide access to Internet and a website to the SMEs. CompeTIC aims to promote productive intelligent environments (EPIS), avoid SME digital divide and develop SME complete programmes. Internauta, aims to allow the citizens of the Valencia Region to use Internet in an intelligent way. It is aimed at groups at risk of digital exclusion. http://e-lane.org
www.competic.es
INTERNAUTA
www.internauta.gva.es
15. Excellence in the training resources supply. All involved resources in the system should look for offering added value services: manuals, instructions, teachers preparation, etc. 16. Effective communication strategy. Dissemination and information about the programme/course should be congruent with reality in order to avoid misunderstandings as well as to enable an effective communication. As expected, the technological criterion is neither the only one nor the most important, there are many more which were elaborated after the review and reading of several initiatives, including that ones involved in the i-AFIEL consortium. However, it is important to consider these sixteen facts or ideas as the main challenges in the field of e-learning. The existence or non existence of these criteria in the projects will be evaluated through a set of indicators. This will be the key point of the evaluation model; the simple success of a project is not the main interest for i-AFIEL partnership, but the contribution of innovation to success. IAFIEL methodology tries to study the relationship between innovative tools, models, programmes, courses with the progress of e-learning and its application and usage in new scopes and new audiences. The final step, as shown in figure 1, is the construction of the indicators, which will take the form of two questionnaires, one devoted to the organisation offering the e-learning course and the other, addressed to the students. It is necessary to take into consideration both actors due to the fact that both are equally involved in the e-learning process, thus, there will be indicators related to students and also to the organisation offering the course or programme. The system is supported in the usage of quantitative methods which allows us to achieve an objective model. The final goal of the methodology advises the usage of the most aseptic ways of measuring, that is, the same criteria and indicators should be used for every project evaluated and the observation of these characteristics must be equally measured. However, qualitative interviews are also welcome and could be carried out by evaluators if they think it is needed, remarks and explanations on the projects are supposed to be present during the evaluation. In fact the final questionnaire allows the collection of further information although it is not taken into account for scoring the project, this means that the final assessment report could include that information, but comparative results and final evaluation of the project will come from quantitative indicators 4 .
The information here reported on the i-AFIEL benchmarking methodology are extracted from the i-AFIEL guidelines available at www.iafiel.gva.es
Furthermore not all indicators are related to these criteria; this technical framework divides the indicators in innovation and results indicators. The latter ones are independent on the innovation criteria, they are just giving to the evaluators an approximation to the satisfaction of the users and the success of the e-learning project based on the students experience and their behaviour measured through this figures. The following factsheets present the indicators following the fields described before. First, here are presented just two examples of the indicators which will be asked to the organisation in charge of the e-learning programme analysed, and then other two examples of those related to the students.
Indicator: Respondent: Criteria: Rationale: Field: Question: Extra Question: Notes: Indicator: Respondent: Criteria: Rationale: Field: Question: Total amount of Students expected to be Code: R01 enrolled Organisation Not applicable Shows the starting enrolment expectations and serves as basis for other indicators Results (Project achievements) 1. How many Students did you expect to be enrolled before the beginning of the course/programme?
Students expectations testing Code: I11 Organisation 4 Shows the attention paid to expectations of the Students Innovation (Service customization) 2. Is there a test to know the expectations of the Students in the course/programme? Yes No
Assessment of the teachers support Code: IR01 Student 15 Shows the opinion of the Students about the teachers support Innovation (Service customization) / Results (Student satisfaction) 1. Did the teacher / mentor give enough help during the course? Please answer scoring
this scale where 0 means not enough at all and 4 totally enough
Not Enough. 0 Enough 1 2 3 4
Extra Question: Notes: Indicator: Respondent: Criteria: Rationale: Field: Question: Adequacy of the course/programme to Students expectations based on Code: IR02 the information provided by the Organisation Student 6, 16 Shows the adequacy between the communication of the programme/course and the final contents Students got Results (Satisfaction) / Innovation (Service customization) 4. Did the course deliver what you expected from it? Please answer scoring this scale
where 0 means the course does not deliver what you expected at all and 4 the course
will facilitate the customization and will allow the evaluators to detail the particularities of a project and point out their singular and most relevant aspects. However, this individual qualitative information obtained by each project can not be included in the scoring for the innovation assessment because this is based on quantitative methods. Once decided the scoring strategy it was crucial to make an easy-application system which would take into consideration all questions equally. Lets explain the innovation evaluation system. Once gathered the information through the answered questions for each innovation indicator, the data will be treated adding up all scores obtained in the questionnaires after their application and then dividing this addition by the total score which could be obtained.
Example: If we apply the system we will obtain a percentage, e. g. we have 23 questions in the organisations questionnaire and 8 in the students one, this means a project can obtain, as a maximum, 92 points (23x4=92) in the first and 32 (8x4=32) in the second, that is, a total score of 124 points. Imagine the project we are evaluating obtains a total score of 93 points: (93/124)*100 = 75%
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
We can not add the scores of every student, it would not be logical because the final score of a project would depend on the total amount of students enrolled answering the questionnaires, and it would not be a good measure of innovation, but of the students registered in the programme or course. We have to analyse the answers, specifically we should calculate the average of each question:
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Student 1 1 3 3 3 1 3 1 2
Student 2 1 2 1 2 1 2 1 1
Student 3 2 1 1 2 1 1 1 1
Student 4 2 3 3 4 2 4 1 1
Student 5 1 3 3 3 1 3 1 2
Once we have the average score of every question, we have to add them up in order to obtain the whole score of the students questionnaire:
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
Then we add up this score to the organisations one, what gives us the whole score of the project: 78 + 15 =93 (95/124)*100 = 75% In this case, the final score is 75%.
information they are capable of carrying out a full report of the project pointing out not only its innovation level, but also how it is related to the results obtained.
3 Conclusions
Although the i-AFIEL methodology might represent just an attempt to measure innovation in elearning, it can be considered as an example of concrete application of how a research study in e-learning could be creatively carried out. The methodology was created and tested among the e-learning projects illustrated in table 1. However, the methodology was not tested on other wider settings and, hence, the methodology still needs further research and study in order to develop its full potential at European level. While the methodology brought a lot of innovation, some aspects are still obscure and need further improvements. For example, it was not possible to define well how the relevant qualitative elements collected could be transposed into the quantitative system of measurement. Hence, some questions are here due: how much is the weight of qualitative elements on the final quantitative score? How much the former might influence the latter? How to translate this into practice? The fact that the methodology was not unfortunately fully applied on the expected number of projects was another unsolved issue. A pre-test was made but its full and concrete application is what really would have given the real results of the methodology which could be considered partial, at this stage. Furthermore, among the innovation criteria there is not a specific section dedicated to pedagogy in e-learning although probably this issue is touched more or less indirectly by some of the criteria presented in the list. Pedagogy is very important while discussing about elearning and innovation and would have needed to be treated with a special attention as a single item among the other innovation criteria. Innovation is also about pedagogy and the approach to e-learning used for guaranteeing a full inclusion for all. Finally, the i-AFIEL is an open and live methodology, which must be continuously updated with the latest technologies and its applications to education which will come in the future. That is why this methodology can be considered in continuous evolution also because of its possible different application in different contexts and countries. Hence other contributions in the field are then expected for further improvements and developments 5 .
References
Falcao, Rita et al, 2007, A proposal for Benchmarking learning objects, en eLearning Papers, 3, March, in www.eLearningpapers.eu Pittinsky, Matthew et al, 2000, Quality on the Line. Benchmarking for success in Intenet-based distance education, Institute for higher education policy. Sangr, Albert et al, 2002, Benchmarking of Virtual Campuses (BENVIC), Minerva Project. VV.AA, 2004, Handbook of best practice for evaluation of eLearning effectiveness, Qual eLearning project. VV.AA, 2005, Accreditation in distance learning Processes and criteria, Special Interest Group for Distance Learning Accreditation in Europe.
VV.AA, 2006, 206 ELearning Benchmarking Project. Final Report, en Australian Flexible Learning
A special thank is here due to the i-AFIEL project partners, especially to OVSI Foundation the creator of the i-AFIEL methodology, who provided us with the authorization to report in this paper, part of the content already published in the i-AFIEL project guidelines. For more reference see www.iafiel.gva.es
11
Authors
Lucilla Crosta E-Learning Specialist Kelidon Association Lcrosta@kelidon.org Victor Prieto Coordinador Cevalsi (Observatorio Valenciano de la Sociedad Tecnolgica y del Conocimiento) vprieto@ovsi.com
Copyrights
The texts published in this journal, unless otherwise indicated, are subject to a Creative Commons Attribution-Noncommercial-NoDerivativeWorks 3.0 Unported licence. They may be copied, distributed and broadcast provided that the author and the e-journal that publishes them, eLearning Papers, are cited. Commercial use and derivative works are not permitted. The full licence can be consulted on http://creativecommons.org/licenses/by-nc-nd/3.0/
12