University of Wollongong Research Online Academic Services Division - Papers Academic Services Division 2007 ascilite Report 4 for the Carrick Exchange Project: Final Report R. Philip D. Parrish Geraldine E. Lefoe University of Wollongong, glefoe@uow.edu.au Meg O'Reilly Southern Cross University Recommended Citation Philip, R.; Parrish, D.; Lefoe, Geraldine E.; and O'Reilly, Meg: ascilite Report 4 for the Carrick Exchange Project: Final Report 2007. http://ro.uow.edu.au/asdpapers/81 Research Online is the open access institutional repository for the University of Wollongong. For further information contact Manager Repository Services: morgan@uow.edu.au. ascilite Report 4 for the Carrick Exchange Project: Final Report Abstract Whilst the development of repositories for sharing learning and teaching resources has expanded considerably in the last few years in Australia, many repositories have a particular institutional or discipline focus. In Australia there is no dedicated national repository or community available for the higher education sector that provides a cross disciplinary forum for sharing ideas and resources about learning and teaching, and for cross-institutional collaboration. The Carrick Institute for Learning and Teaching in Higher Education is well placed to address this gap by developing the Carrick Exchange (see Phillips, Orrell & Millea, 2007). Utilising the current expansion and availability of Web 2.0 services, the Carrick Exchange can support the development of a national and international community to achieve these goals and meet the Institute’s objectives. In particular the Carrick Exchange will provide an avenue to demonstrate the values of inclusivity by “assisting the development of networks and communities which support higher education staff who have a direct impact on the advancement of learning and teaching” and through the focus on long-term systemic change (Carrick Institute website).The Carrick Exchange is seen by many in the sector as a major initiative with the potential to significantly improve the profile of teaching and learning. Disciplines Education Publication Details This report was originally published as Philip, R, Parrish, D, Lefoe, G and O'Reilly, M, ascilite Report 4 for the Carrick Exchange Project: Final Report, Wollongong, NSW, The Australasian Society for Computers in Learning in Tertiary Education, 2007. Original report available here This report is available at Research Online: http://ro.uow.edu.au/asdpapers/81 ascilite Report 4 for the Carrick Exchange Project: Final Report September 2007 Prepared by: ascilite The Australasian Society for Computers in Learning in Tertiary Education 1 Document details Document name ascilite Report 4 for the Carrick Exchange Project: Final Report Authors Robyn Philip, Dominique Parrish, Geraldine Lefoe, and Meg O’Reilly For ascilite (The Australasian Society for Computers in Learning in Tertiary Education) File name ascilite_Report 4 for CE_v1_0.doc Other documents to be read in conjunction with this document ascilite/Carrick Exchange Proposal and Project Plan 2007 ascilite Reports 2, 3 and 4 for the Carrick Exchange Project 2007 ascilite Resource Identification Network (RIN) Project, Stage 1 ascilite Component Think Tank Issues Paper, 2006 Version History Date 04/10/07 Version 1.0 Comment Final authorised by Geraldine Lefoe Project team Project Leaders Dr Geraldine Lefoe, University of Wollongong Meg O’Reilly, Southern Cross University Project Manager Dominique Parrish, Learning Achievements & Solutions Research Manager Robyn Philip, Consultant Project Team Dr Cathy Gunn, University of Auckland Dr Mike Keppell, Hong Kong Institute of Education Project Steering Group (ascilite Executive) Dr Joe Luca, Edith Cowan University, Kar-Tin Lee, Queensland University of Technology, Dr Barney Dalgarno, Charles Sturt University, Dr Bob Corderoy, University of NSW, Caroline Steel, University of Queensland Allan Christie, Netspot Pty Ltd International Adviser Dr Allison Littlejohn, Glasgow-Caledonian University Acknowledgement The research team would like to thank all the interviewees, focus group participants, reference group members and other advisors who took part in the research and willingly shared their experience, ideas and time. We also wish to acknowledge the additional support provided by the ascilite executive team, and the Carrick Exchange Project Team. Support for this research has been provided by the Carrick Institute for Learning and Teaching in Higher Education, an initiative of the Australian Government Department of Education, Science and Training. The views expressed in this report do not necessarily reflect the views of The Carrick Institute for Learning and Teaching in Higher Education. 2 TABLE OF CONTENTS Key outcomes .......................................................................................................................... 5 Introduction.............................................................................................................................. 8 Background.............................................................................................................................. 8 Methodology .......................................................................................................................... 10 Aims and outcomes............................................................................................................... 10 The design-based methodology .......................................................................................... 11 Literature review .................................................................................................................... 16 Engagement guiding principles and recommendations ................................................... 23 Recommendations related to the purpose of the carrick exchange........................... 24 Recommendations related to multiple levels of engagement ..................................... 27 Recommendations relating to the management of engagement with the carrick exchange ........................................................................................................................... 33 Recommendations related to ease of use ..................................................................... 38 Recommendations related to promoting engagement with the carrick exchange ... 40 Recommendations for engagement policies................................................................. 43 Resource identification and contribution guiding principles and recommendations ... 45 Principles related to resource identification ................................................................. 46 Principles related to addressing the culture around sharing in higher education.... 49 Principles related to rewards, incentives and recognition for contribution .............. 52 Principles related to intellectual property and digital rights management ................ 55 Principles related to management of resource contribution ....................................... 58 Recommendations for resource identification and contribution policies.................. 62 Peer review and commentary guiding principles and recommendations ....................... 65 Protocols and mechanisms related to the goals of peer review ................................. 67 Protocols and mechanisms related to the process and procedures for peer review 72 Protocols and mechanisms related to informal commentary...................................... 79 Protocols and mechanisms related to rewards and incentives for formal peer review and informal commentary ............................................................................................... 82 Recommendations for peer review and commentary policies .................................... 84 Conclusion ............................................................................................................................. 86 Benefits................................................................................................................................... 86 Challenges.............................................................................................................................. 87 Future research...................................................................................................................... 89 References ............................................................................................................................. 91 Appendices ............................................................................................................................ 96 Appendix A: Glossary of terms....................................................................................... 96 Appendix B: Repositories, associations and projects referred to in the report........ 98 Appendix C: Key practitioners interviewed in cycle 1................................................ 101 Appendix D: Focus group participants – cycle 2 ........................................................ 103 Appendix E: Engagement table specific example – carrick grant holders .............. 104 Appendix F: Roles identified for the carrick exchange .............................................. 107 Appendix G: Examples of formal peer review processes for learning and teaching resources ................................................................................................. 109 Appendix H: Peer review criteria and standards ........................................................ 113 Appendix I: Protocols for peer review.......................................................................... 115 3 Executive Summary ______________________________________________ Whilst the development of repositories for sharing learning and teaching resources has expanded considerably in the last few years in Australia, many repositories have a particular institutional or discipline focus. In Australia there is no dedicated national repository or community available for the higher education sector that provides a cross disciplinary forum for sharing ideas and resources about learning and teaching, and for cross-institutional collaboration. The Carrick Institute for Learning and Teaching in Higher Education is well placed to address this gap by developing the Carrick Exchange (see Phillips, Orrell & Millea, 2007). Utilising the current expansion and availability of Web 2.0 services, the Carrick Exchange can support the development of a national and international community to achieve these goals and meet the Institute’s objectives. In particular the Carrick Exchange will provide an avenue to demonstrate the values of inclusivity by “assisting the development of networks and communities which support higher education staff who have a direct impact on the advancement of learning and teaching” and through the focus on long-term systemic change (Carrick Institute website).The Carrick Exchange is seen by many in the sector as a major initiative with the potential to significantly improve the profile of teaching and learning. Most importantly, it is about connecting people with people . . . It is a space to 1 contribute, discuss and debate. (Carrick Exchange website) ascilite, the Australasian Society for Computers in Learning in Tertiary Education, was engaged to assist with the development of the Carrick Exchange (2006 – 2007) by conducting research into: • • • Users’ needs within the higher education sector for collaborative and communication spaces for learning and teaching; The conditions and contexts of use of resources available for sharing and reuse within the higher education sector; and The policies necessary to facilitate engagement of the higher education sector with the Carrick Exchange. The challenge for the Carrick Institute in developing the Carrick Exchange is that the literature, along with national and international practice, indicates that uptake of repositories of learning materials has been slower than expected. Poor engagement by target users may be related to lack of clarity about purpose and lack of strategies and resources directed to fully engaging the community for which the repository has been designed. The ascilite research has focussed on identifying the protocols, procedures and strategies that will be required to address these issues. A design-based methodology was adopted using interviews, focus and reference groups methods to gather data which was analysed in a cyclical and iterative way along with the current literature. In this way ascilite was able to document the experience, opinions and advice of the target users from the Australian higher education sector. The findings will inform the next iteration of Carrick Exchange development. 1 http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381 4 Key outcomes The findings derived from the ascilite research in Stage 2 are listed in terms of the recommendations, principles, protocols and mechanisms. User engagement Recommendations related to the purpose of the Carrick Exchange 1.1.1. The purpose of the Carrick Exchange should be well articulated to the sector. 1.1.2. The quality of the processes, resources and functionality of the Carrick Exchange should meet the expectations of the sector. Recommendations related to multiple levels of engagement 1.2.1. A strategic plan for the engagement of the sector at multiple levels should be developed and implemented. 1.2.2. Membership and the conditions of membership for the Carrick Exchange should be specified. 1.2.3. The Carrick Exchange should meet the diversity of members’ needs and contexts of use. Recommendations relating to the management of engagement with the Carrick Exchange 1.3.1. The Carrick Exchange policies and practices should be aligned strategically with institutional policies and practices. 1.3.2. The Carrick Exchange should engage with the National Authentication Framework to ensure integration. 1.3.3. A strategy should be developed to support, encourage and manage champions to promote the Carrick Exchange. 1.3.4. Partnerships with Australian universities should be established to ensure longevity and sustainability of the Carrick Exchange. 1.3.5. Collaboration should be encouraged and supported as a means of fostering engagement and contribution to the Carrick Exchange 1.3.6. The Carrick Exchange should support the needs of networks and communities of practice. 1.3.7. An evaluation plan should be developed to measure levels of engagement including participation, collaboration and contribution. Recommendations related to ease of use 1.4.1. The Carrick Exchange should devise strategies of engagement for the varying levels of technical capability of users. 1.4.2. The Carrick Exchange should be easily accessible and intuitive to use. Recommendations related to promoting engagement with the Carrick Exchange 1.5.1. There should be a strategic plan and adequate support for the promotion and publicity of the Carrick Exchange to ensure successful engagement by the sector. 1.5.2. There should be regular opportunities to disseminate and encourage engagement of the higher education sector with the Carrick Exchange. 1.5.3. Web 2.0 technologies should be promoted and supported as a mechanism for enhancing engagement with the Carrick Exchange. 5 Resource contribution Principles related to resource identification 2.1.1. The Carrick Exchange should strategically target resource contributors. 2.1.2. Existing resources that could be contributed or linked to the Carrick Exchange should be identified. Principles related to addressing the culture around sharing in higher education 2.2.1. A strategic plan for establishing and maintaining institutional partnerships for sharing between institutions and the Carrick Exchange should be developed. 2.2.2. The Carrick Exchange should systemically address the culture surrounding sharing in higher education. 2.2.3. Networks and partnerships should be nurtured as a means of fostering contribution to the Carrick Exchange. Principles related to rewards, incentives and recognition for contribution 2.3.1. The Carrick Exchange should establish a system of rewards, incentives and recognition that is valued, appropriate and manageable, and aligned to institutional policies and practices. 2.3.2. The Carrick Institute should ensure that the established system of rewards, incentives and recognition is adequately resourced and supported into the future. 2.3.3. A strategy for impact and process evaluation should be developed to assess the relevance and value to the sector of the rewards, incentives and the system of recognition. Principles related to Intellectual Property and Digital Rights Management 2.4.1. The Carrick Exchange should establish IP and Rights Management policies and practices that align to institutional policies and practices. 2.4.2. IP and Rights Management should address the concerns and needs of contributors 2.4.3. The Carrick Exchange should develop systems and structures to ensure that IP and Rights Management policies and practices are implemented and appropriate. Principles related to management of resource contribution 2.5.1. The Carrick Exchange should develop clear policies, guidelines, and procedures for resource contribution. 2.5.2. The Strategic Plan should include guidance on resource management, archiving, sustainability, and support and encouragement for resource contribution. 2.5.3. An evaluation plan should be developed to assess and report on resource contribution to the Carrick Exchange. Peer review and commentary Protocols and mechanisms related to the goals of peer review 3.1.1. Formal peer review policies and procedures of the Carrick Exchange should align with institutional policies and practices, and promote the scholarship of teaching and learning. 3.1.2. The goals of formal peer review should be established and inform related policies and procedures. 6 3.1.3. Peer review should enable judgements about the quality and usefulness of a resource. 3.1.4. Resources contributed to the Carrick Exchange should be classified according to the level of peer review received. 3.1.5. The formal peer review process should be appropriately resourced and adequately rewarded Protocols and mechanisms related to the process and procedures for peer review: 3.2.1. The Carrick Exchange should establish processes, criteria and standards that enable consistent, equitable and fair peer review. 3.2.2. The peer review process should meet the expectations of the sector and the needs of members. Protocols and mechanisms related to informal commentary 3.3.1. Guidelines and protocols for informal commentary should be established to ensure that the needs and expectations of contributors and users are met. Protocols and mechanisms related to rewards and incentives for formal peer review and informal commentary 3.4.1. The Carrick Exchange must implement rewards and incentives for participation in either formal peer review or informal commentary 7 Introduction ______________________________________________ Background The Carrick Exchange is a major initiative within the Australian higher education sector. The Carrick Exchange is a new online service that will provide learning and teaching resources and support communication and collaboration across the national and international higher education sector. The Carrick Exchange is a hub for the exchange of ideas about teaching practice in the Australian higher education sector. It is a place to explore, discover and experiment with issues, technologies, processes and ideas. Most importantly, it is about connecting people with people . . . It is a space to contribute, discuss and debate. (Carrick Exchange website) 2 ascilite, the Australasian Society for Computers in Learning in Tertiary Education, was engaged to assist with the development of the Carrick Exchange (2006 – 2007) by conducting research into: • • • Users’ needs within the higher education sector for collaborative and communication spaces for learning and teaching; The conditions and contexts of use of resources available for sharing and reuse within the higher education sector; and The policies necessary to facilitate engagement of the higher education sector with the Carrick Exchange. The ascilite research was funded by the Carrick Institute for Learning and Teaching as a subproject of the Carrick Exchange. This final report on guidelines and recommendations is one of four documents which present the findings derived during Stage 2 of the ascilite research (January to September 2007). Report 1: The Literature Review Report 2: Themes, Issues and Concerns Emerging from Key Practitioner Interviews Report 3: Themes, Issues and Concerns Emerging from Focus Groups Report 4: Final Report Preliminary work was undertaken in 2006 as a Stage 1 investigation, examining the background issues which might impact on the development of the Carrick Exchange. Outcomes of Stage 1 of the project are reported in the Resource Identification Network (RIN) Project, Stage 1 - ascilite Component Think Tank Issues Paper 2006. The challenge for the Carrick Institute in developing the Carrick Exchange is that the literature, along with national and international practice, indicates that uptake of repositories of learning materials has been slower than expected. Poor engagement by target users may be related to lack of clarity about purpose, and lack of strategies and resources directed to fully engaging the community for which the repository has been designed. 2 http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381 8 With this in mind, the key drivers for the ascilite Stage 2 research were: • • • consolidation of knowledge and current practice; investigation into engagement of the higher education sector; and the development of communities of practice. Consolidation: Stage 2 of the ascilite/Carrick Exchange project built on Stage 1 research informing the technical development of the Carrick Exchange by reporting on a broad range of user needs and contexts of use of resources and communication environments. Engagement: Through an examination of professional groups that have knowledge and expertise in developing and adopting learning and teaching resources, the ascilite investigation aimed to find some solutions to the general lack of user engagement in repositories and online services in education world-wide reports. The research focussed on methods of engagement with informal repositories of knowledge using Web 2.0 technologies, and explored alternative methods of capturing information and networks of users in this context. Developing communities of practice: Consideration of how networks and communities of practice are built and sustained is key to the on-going success of the Carrick Exchange. Communities of practice are not formal arrangements or organisational bodies but are informal entities that exist due to the members’ shared domain of knowledge, their sense of community and shared practice. Communities and networks such as those around existing online services were investigated (MERLOT (North America), CD-LOR (UK), the Minister of Communications Digital Strategy Advisory Group (NZ), the Australian Partnership for Sustainable Repositories (APSR), and Australian Research Repositories Online to the World (ARROW) etc.). Factors impacting on these communities and reported here include organisational, socio-cultural, information management and technological issues. Full details of the background to the ascilite project are provided in the ascilite/Carrick Exchange Proposal and Project Plan 2007. Further analysis of the Carrick Exchange is provided in reports by Phillips, Orrell and Millea (2007), and Lefoe, O'Reilly, Parrish, Bennett, Keppell and Gunn (2007). A glossary of the key terms used in this research is provided as Appendix A. Presentation of findings, recommendations and guiding principles In presenting the consolidated findings as recommendations and guiding principles in this report in the three key areas (engagement, resource contribution and identification, and peer review and commentary), the authors note that there may appear to be some repetition. This is unavoidable as some issues have implications that cross all three areas and manifest in different ways according to the focus area. 9 Methodology ______________________________________________ Aims and outcomes Aim The aim of Stage 2 of the ascilite research was to inform development of the Carrick Exchange through further research into the three key areas: engagement, resource identification and contribution, and peer review and commentary. The data and analysis was to be derived from a coordinated and collaborative series of activities, distilling the available knowledge and expertise from: • • • • The literature Current practice Exemplary community development to foster engagement with online services, and Key practitioners and networks in both national and international settings. Outcomes The outcomes of the Stage 2 research are: 1. Recommendations and procedures to foster engagement and uptake of the Carrick Exchange by the sector. 2. Guiding principles for resource contribution and identification, networking and engagement that extend beyond the development of technical infrastructure and standards to consideration of the ease of use and the ability to customise the resources; 3. The development of Peer Review and commentary protocols, and mechanisms for resource contribution and identification to be incorporated in the Carrick Exchange. Key terms See Appendix A for a glossary of key terms used in this research. 10 Table 1: Research questions for Stage 2 Research questions 1. What Peer Review and commentary protocols and mechanisms can be derived from the higher education community? Sub-questions • • • 2. What resource contribution and identification methods will engage and encourage users to contribute and collaborate within the Carrick Exchange? 3. What are the successful methods for establishing and maintaining engagement (i.e. use, reuse and recognition) by the higher education community with the Carrick Exchange? • • • • • • • What new and current methods exist for peer review and commentary of resources that can be adapted for use for the reviewing of teaching and learning resources? o What gets peer reviewed? o What methodologies apply to different types of resources? o What process for peer review is used? o Who are the reviewers? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? What are the key factors to support resource contribution, identification and reuse? What types of resources will be contributed? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? What are the key factors for establishing and maintaining engagement in academic and nonacademic online communities e.g. online support groups, interest and resource sharing groups etc? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? The design-based methodology A design-based research methodology was adopted for Stage 2 of the ascilite research. This required an iterative approach to data collection and analysis, similar to action research in its inclusion of stakeholders during each stage. The difference, however, is that the outcomes of design-based research should be a series of recommendations and/or principles for design (Design-Based Research Collective, 2003; van den Akker, 1999). (See Figure 1.) Figure 1: The design-based research cycle. Adapted from Reeves (2000) Overall, the design-based research cycle was completed three times during Stage 2 of the Carrick Exchange project (Figure 2). The final cycle will be completed in December, 2007. 11 Each cycle focused on one type of stakeholder group and each successive cycle was informed by interpretation of the data from the previous cycle/s. Cycle 1 Key practitioner interviews In Cycle 1 the project team approached thirty five key members of the higher education community, liaising with organisations within Australia and overseas as part of the process of developing a well grounded understanding of repositories and their communities based on evidence and experience (See Appendix C). Twenty-nine of these individuals agreed to interviews. National and international interviewees were chosen from across disciplines, institutions and fields of interest and identified as those with: (a) a need for online collaboration and communication services for learning and teaching; and/or (b) significant experience in the field. Interviewees included representatives of international projects (6), national initiatives (3), vocational education and training initiatives (4), institutional repositories (3), discipline and professional associations (9), institutional users (2), Deputy/Pro Vice-Chancellors (1), and one Carrick Institute Fellow. The research questions underpinning all interviews were: 1. What peer review and commentary protocols and mechanisms can be derived from the higher education community? 2. What resource contribution and identification methods will engage and encourage users to contribute and collaborate within the Carrick Exchange? 3. What are the successful methods for establishing and maintaining engagement (i.e. use, reuse and recognition) by the higher education community with the Carrick Exchange? Each interview was conducted by a single interviewer (one of the project team members) either face-to-face or by phone, and recorded with the participant’s consent. Interviewers took notes during the session and a summary transcript of the interview was made. In five cases transcripts were created from notes only, due to failure of the technology. Transcripts were returned to interviewees for validation and then entered in qualitative analysis database software for identification of emerging themes and analysis. Cross-checking of themes by two ascilite project team members was undertaken to increase reliability. This assisted in the validation and preliminary interpretation of findings. A full report of the interviews is included in ascilite Report 2 for the Carrick Exchange Project: Themes, Issues and Concerns Emerging from Key Practitioner Interviews Cycle 2 Focus groups Cycle 2 was undertaken through three focus groups (Brisbane, Sydney and Melbourne). A total of twenty-two practitioners participated. The aims of the focus groups were to build on the outcomes of the data already gathered from the individual interviews in Cycle 1 and: • • • • • Review the data and report from the key practitioner interviews; Discuss the issues that arose from the data; Prioritise issues raised and generate possible strategies for addressing these; Generate ideas for policy development and implementation; and Identify missing data or where further research was required. 12 Representatives were invited from each Australian university and where possible included: • • • • • • • Academics interested in learning and teaching but with little experience of technology; Members of cross-institutional teams (Carrick grant holders, consortiums etc.); Educational and staff developers; Librarians; ‘Early adopters’; Repository representatives; and A mix of gender and cultural backgrounds. Availability of staff meant that it was not always possible to achieve this diverse mix in each focus group. However we did achieve representation from five of the six States, including Western Australia and South Australia. There were no representatives from the Northern Territory, The Australian Capital Territory or Tasmania. Four to twelve participants were in each group, and each focus group included some representation of educational and staff developers, and members of cross-institutional collaborative project teams. In at least two of the focus groups there were librarians and repository representatives. Two participants represented the vast majority of staff in higher education who have an interest in learning and teaching but may have only had limited use of the technology. In addition, within the groups there were varying degrees of expertise from those highly skilled in the use of technology through to those with a more limited experience. Two participants had also attended the Carrick Exchange Think Tank forum in Melbourne, September 2006 (Stage 1). Some focus group participants aligned to more than one representative category. Prior to the focus groups participants were sent: • • • the focus group questions, derived from the data in Cycle 1; promotional material providing a background and context to the Carrick Exchange; and the Executive Summary from the Report on Cycle 1 – Key Practitioner Interviews. The findings from the key practitioner interviews were used to underpin guided reflection by the focus group participants. The two-hour focus groups were based on a structured discussion process whereby participants interrogated the findings of the key practitioner interviews and worked towards the development of possible strategies, solutions, recommendations and policies in the three areas –engagement, resource identification and contribution, and peer review and commentary. At the beginning of each session, a very brief presentation was provided as to the nature and functions of the Carrick Exchange under development. The questions were posed by one member of the research team while a second member took written notes of the session. The focus groups were also recorded with the consent of participants and the recordings transcribed. These transcripts were analysed with the same qualitative approach in Cycle 1 of using software to assist in deriving key themes and cross checking between two members of the research team. The report of focus group themes and illustrative quotes were subsequently shared with the ascilite reference groups (see Cycle 3) to further inform the ongoing development of the Carrick Exchange. A full report of the focus groups is included in ascilite Report 3 for the Carrick Exchange Project: Themes, Issues and Concerns Emerging from the Focus Groups. Cycle 3 Reference groups To establish membership of three ascilite reference groups, in all 24 ascilite members were selected from expressions of interest received in the period December 2006-March 2007. Membership of reference groups was determined on the basis of stated interest and expertise, and members were allocated to one of the following research areas: (1) user engagement, (2) resource identification and contribution and (3) peer review and 13 commentary. To ensure representation across the sector both geographically and institutionally, the following members were selected: - 12 from NSW, 4 from Victoria, 2 from Queensland 1 from Western Australia, and 2 from the TAFE sector, NSW. In addition reference group members were also selected from New Zealand (1), Singapore (1) and Hong Kong (1). Reports from Cycle 1 and Cycle 2 containing the data, analysis and recommendations for each cycle were uploaded to the EdNA 3 site for reference groups to peruse and provide comment. Comments submitted were thus for open view to all subscribers of the reference group. ascilite reference group members were requested to comment within a 3-week period and 10 of the 24 members submitted constructive and well considered responses to the forum (4 of 10 engagement; 2 of 7 peer review, and 4 of 6 resource identification and contribution). Some interaction occurred among the reference group members as evident by the average number of postings per respondent within forums being between two and three. One member of the reference groups reported they had read the reports discussed with colleagues in their institution and subsequently provided a synthesis of responses for the forum. An additional three people read through the comments but did not contribute themselves. All those who did not respond were encouraged to bring their perspectives to the symposium to be held at the ascilite conference (see Cycle 4). All comments submitted to reference group forums were analysed using the approach to deriving themes that was used throughout previous cycles. The key themes, recommendations and guiding principles were included in the overall report for Cycle 3 (ascilite Report 4 for the Carrick Exchange Project: Final Report). The final set of design principles for Cycle 3 are included in the Conclusion to this report. Cycle 4 Symposium An international symposium is to be hosted during the 24th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ascilite). In the lead-up to this symposium the ascilite Carrick Exchange final report will be circulated to reference group members. In addition three papers have been prepared and will be provided to symposium registrants for stimulus. The authors will report on current issues in the repositories and their communities developed in UK, USA and Australia. Using these stimulus papers for pre-reading by participants, the symposium will be structured to allow for small group interaction and an active approach to deriving issues themes and concerns that will guide future research into the Carrick Exchange. The symposium will also be recorded with the consent of participants and one member of the research team will take notes including any that are put up on a whiteboard. These recordings and notes will be transcribed and the key themes will be derived. Illustrative quotes will also be used in the report that will be delivered following the symposium. Overall, the aim of all four cycles was to inform the development of possible strategies, solutions, recommendations and policies in the three key areas: engagement; resource contribution and identification; and peer review and commentary. 3 Education Network Australia, collaborative workspace for project, http://www.edna.edu.au/edna/go 14 Figure 2: Project Methodology Showing Four Cycles 15 Literature review ______________________________________________ Further review of the literature is to be found in ascilite Report 1 for the Carrick Exchange Project: The Literature Review. Engagement of the sector and communities of practice To promote knowledge sharing in the domain of learning and teaching, and engage the Australian higher education community within the Carrick Exchange, it is not sufficient to build a repository of resources, nor merely to provide social networking software and spaces for collaboration and community building. The literature increasingly documents a general lack of user engagement with repositories and online services (e.g. Gunn, Woodgate & O’Grady, 2005; Hummel, Tattersall, Brugos, Brouns, Kurvers and Koper 2005; Littlejohn, 2003; Phillips, Aspin, Hull, & Oxley, 2004). It is acknowledged that effort should be directed towards the process of engaging the target audience, familiarising groups with the affordances of the Carrick Exchange and its potential to support user needs, and proactively facilitating the development of networks and communities of practice engaged in sharing and developing high quality resources. The potential benefits of repositories are generally well documented. The following list, generated by combining Deakin University institutional repository information, (Monahan & Owies, 2005, pp.1-2) and the Jorum national repository information from the UK (Jorum, 2007), indicates significant advantages. • • • • • • • • • • • • • • • • Participation in the advancement and sustainability of e-learning at a national level; Conservation of time and effort through reuse and sharing of knowledge and resources; Facilitation of communities of practice and online professional networks at a national level; Project dissemination as a requirement of funding obligations; Archival functionality for publicly funded project output; Seamlessly integrated knowledge management systems; Easy cost effective searching and retrieval of resources; Improved sharing and storage systems; Improvements to the quality and currency of courseware and methodologies in units of study; Automated review notifications for content expiry; Workflow visibility and effectiveness - documented processes for contribution and reuse; Authenticated and secure access; Automated tracking and control over the reuse of materials; Legal security for contributors through a robust tagging and licensing system; Management and protection of copyright; and Metadata and cataloguing conforming to accepted standards, enabling easy search and retrieval. However, realising these advantages is a complex task. Ignoring the sociocultural issues relating to learning object repositories is to run the risk of creating an under utilised service. It cannot be assumed that reuse will follow existence of the repository (Margaryan & Littlejohn, 2007; Philip, 2007). Margaryan and Littlejohn point to the effect various cultural dimensions (organisational, professional, disciplinary and national) may have on the impact, uptake and usage of the system. These factors are further influenced by community size, member proximity to the resource, the roles of stakeholders and types of tasks for which the resources in the repository are intended and used. In addition, barriers to uptake will cluster around socio-cultural, pedagogic, organisational and informational management, and technological issues (Margaryan, Currier, Littlejohn, & Nicol, 2006). Margaryan et al. offer the following general solutions to each of these areas (pp.4-5): 16 Solutions to socio-cultural issues • Design of the LORs (learning object repositories) should be based on understanding of cultural norms and expectations of the user communities. Solutions to pedagogic issues • • • • • • Emphasis on pedagogy pull vs. technology push. Development of LORs by multidisciplinary teams (including teaching practitioners as well as learning technologists and librarians). Provision of examples of successful use of LORs related specifically to teaching and learning. Development of LOR models involving co-construction of resources by the students. Demonstration of impact for learning and added value for individual users. User development and support in information literacy. Solutions to organisational and information management issues • • Incentives and rewards linked to community needs and goals. LORs linked to organisational strategy and objectives. Solutions to technological issues • • • Facilitating ease of use, engagement, efficiency and pedagogic effectiveness. Conceptualisation of LORs as a context rather than isolated tools. Effective policies and practices for metadata creation. For the Carrick Exchange to become integrated into the everyday work practices of the Australian higher education sector, there are considerable challenges to address. Staff are habitually time poor (Koppi & Lavitt, 2003) and there is no well developed culture of sharing and reuse. Reusing materials developed by others and dissemination of one’s own personal teaching materials are not necessarily seen as incentives to effect the practice of reciprocal sharing (Wetterling & Collis, 2003; Koppi & Lavitt, 2003). Barriers to change are not easily surmounted as the VET sector has found in the evaluation of the initial phases of the Australian Flexible Learning Framework (Phillips et al., 2004). The Carrick Exchange will be initiating technical change, and educational and cultural change. The literature suggests that change management issues should be addressed early and that to effect major educational change commitment at the highest organisational level will be required (e.g. Ely, 1999; Kenny, 2002, in Kenny 2003; McKenzie, Alexander, Harper, & Anderson, 2005). Support championed and led from the top, with “bottom up” support for innovators, and provision of well supported resource development for those in the middle is the best model (Nicol et al., 2004 in Weedon, Bricheno & Chidwick, 2004). Scott (1999, p.10) notes that one of the reasons why many “intrinsically worthwhile” innovations founder in educational settings is that administrative and support personnel are not sufficiently apprised of key information and integrated well enough into change processes. The study by Ehrmann, Gilbert and McMartin 4 (2007) into the impact of icampus in five American universities noted that “the ‘market’ for higher education responds weakly, if at all, to changes in educational effectiveness" and that “widespread dissemination of such activities is very difficult” (p.2). Also: External forces and internal dynamics both tend to create a short institutional attention span for educational reform. . . It can easily take 8-12 years or more to create a visible, rewarded change in the outcomes of a degree program or a university education. (Ehrmann et al., 2007, p.3). Ehrmann et al. (2007) emphasise the need for rewards to “feed” the demand for adoption of new 4 iCampus is an initiative between Microsoft Research and MIT. The consortium aimed to build and research technologies that could make a significant difference in university teaching and learning. Because of the failure of a number of large-scale projects which had not moved beyond the pilot phase, Ehrmann et al. (2007) and iCampus decided to evaluate five successful projects. 17 ideas (p.iii); and importantly suggested that cross-institutional and intra-institutional links should be “nurtured” amongst central services that offer educational and professional development support (p. iii). The use of social networking software to facilitate communication, networking, collaboration and sharing amongst users will be an important aspect of the Carrick Exchange. In general, social software supports collaborative online spaces such as blogs and wikis; sharing of and commentary on photo, audio and video files; digital storytelling; 3D virtual worlds (such as Second Life, http://secondlife.com/); and social spaces such as Facebook (http://www.facebook.com/). Technology usage by individuals is undergoing a change from consumption of content to usercentric creation of content and collaboration via networks. However, there is often an unchallenged assumption that the software on its own promotes communities. The critical elements for successful use of social software are identified in the Australian Flexible Learning Framework report for the VET sector (Evans, 2007, p.13) as, • • • “Authenticity” – there is a real and established need to use the software; “Relevancy” - its use is “relevant to the need, and appropriate for the client”; and “Support” - there is support for the software within an “enabling culture”. Furthermore, Evans (2007) indicates that the best strategies to help staff learn how to use social software tools are action learning, just-in-time mentoring, coaching and work-based learning. The tools considered to be most useful in assisting staff to learn how to use social software are virtual conferencing and online forums. The best professional development strategies were reportedly blended ones using a variety of opportunities and strategies. The importance of modelling as an enabler as well as a professional development strategy was emphasised. Digital storytelling through the use of multimedia case studies was considered very effective, along with “learning by stealth”, i.e. blending usage into everyday practices and inviting managers to ‘see, hear and feel what happens’ (Evans, 2007, p.22). The Carrick Exchange may well support fully formed communities of practice plus other looser and more brittle networks. Wenger, McDermott and Snyder (2002), in defining communities of practice, support the notion that a website on its own is not a community of practice. Communities of practice are “groups of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise . . . by interacting on an ongoing basis” (p.4). Development of the necessary domain of knowledge, the community and shared practice over time (through sharing of stories, experiences and methodologies) will present ongoing challenges to the community. Wenger et al. suggest that domain, community and practice are not only defining terms for a community of practice, but that they represent “different aspects of participation that motivate people to join a community” (Wenger et al., 2002, p.44). Some members will join because they are interested in the domain, the specific area of knowledge; some will join because it is the community aspect they seek; others will be seeking to learn about practice, the standards, the tools and the lessons learned by experts in the field. McDonald and Star (2006) point to five key challenges in the formation of a community of practice: the need for financial support; issues of academic time poverty; the need for well-placed institutional champions; the difficulty of identifying and quantifying outcomes from communities of practice; and the question of sustainability and ongoing support. In addition, policies should be stated clearly “and not form unwanted obstacles” for communities to develop and the threshold for entry into the system should not be unnecessarily complicated or high (Hummel et al., 2005, p.66). There must be sufficient room for the community to self organise its own structure and facilities. Hummel et al. also suggest that the system should begin with a minimal set of activities and forums and, when more participation is required, this might be better promoted through synchronous collaboration rather than through asynchronous means. “Clear policies, usability and reward systems are of importance when facilitating a learning network” according to Hummel et al. (Hummel et al., 2005, p.55). The importance of systems of reward and recognition for effort in teaching, and activities such as contribution to the development, sharing and reuse of resources is gradually being recognised within the higher education sector. Some institutions are beginning to take action which recognises the important role of teaching: for example, the University of Queensland, in an internal review (University of Queensland, 2007), 18 has recognised that different staff roles demand varying amounts of emphasis on teaching and research, which should be acknowledged. The argument is that both are of value and worthy of recognition and reward. In addition exemplary repositories and exchanges such as MERLOT and CLOE, and the Carrick Institute itself, have systems of reward in place that provide models for recognising and rewarding members through awards and conferences. The collaborative RUBRIC project for regional universities also has in place recommended methods for tying contribution to their repository with promotion and career advancement processes. The issue of rewards and recognition is not dealt with in depth in this report as it is to be researched further in another study; however, the ascilite research recognises its central importance in the development and sustainability of the Carrick Exchange. Resource contribution, identification and models of use As well as the communities, networks and workspaces expected to establish around the Carrick Exchange, there will be a repository of quality learning and teaching materials either deposited on the site or linked from websites and databases elsewhere. In a review of repository development in the UK (the CD-LOR Report), the authors warn all repository managers to clearly establish the need for any collection (Margaryan, Milligan & Douglas, 2007; Margaryan & Littlejohn, 2007). This view is shared by findings from the research of Gosper, Woo, Gibbs, Hand, Kerr and Rich (2005), and Ringan, Corley and Campbell (2005). Communicating the purpose of the repository and its community will be important to the success of the Carrick Exchange. The Australian Partnership for Sustainable Repositories (APSR) investigation into the ten major issues facing repository service providers, relating to research output in Australia (Henty, 2007), noted that defining the collection was essential communication task in the process of repository development and user engagement. In addition, the APSR research uncovered similar issues of lack of engagement to those noted in the CD-LOR report. Senior academics responsible for repository services and data management were interviewed for the APSR research. It was noted that open access repositories have not been taken up with great enthusiasm: an exception was the Cornell University Physics eprint archive (http://www.arxiv.org). An interesting finding was that in the research context, mandating article deposit resulted in high levels of contribution, as illustrated by the Queensland University of Technology experience. However the high cost and effort required to deposit articles in the system was seen as a barrier by some senior managers. This lack of engagement with repositories has been reported elsewhere (Margaryan et al., 2007; Wenger et al., 2002). While studies such as Najjar, Ternier and Duval (2004) may document usage patterns from the logs of repositories, they do not reveal the broader picture of engagement. Some studies of digital repositories and engagement (Bradley & Boyle, 2004; Littlejohn, 2003; Hand, Gosper, Woo, Gibbs, Kerr & Rich, 2004) list incentives for use put forward by target users, but engagement still remains relatively low. Repository managers catering to more diverse users, e.g. around national rather than discipline-based repositories, are likely to face greater problems in this area (Margaryan & Littlejohn, 2007). Furthermore, the way repositories are used depends not only on the “dimensions of repositories” (purpose, scope, target audience etc.), but also on “key characteristics” of the communities (p.4336). In their study of the needs of academics in the research domain, Foster and Gibbons (2005) noted that academics characteristically require the following: [To be able to] work with co-authors; keep track of different versions of the same document; work from different computers and locations, both Mac and PC; make their own work available to others; have easy access to other people's work; keep up in their fields; organize their materials according to their own scheme; control ownership, security, and access; ensure that documents are persistently viewable or usable; have someone else take responsibility for servers and digital tools; be sure not to violate copyright issues; keep everything related to computers easy and flawless; reduce chaos or at least not add to it; and not be any busier. (Foster and Gibbons, 2005, p.4 of 12) 19 This suggests that collaborative workspaces offered through the Carrick Exchange, where resources can be shared and contributed may well meet a need within the higher educational community. The research of Breslin, Nicol, Grierson, Wodehouse, Juster and Ion (2007) within the discipline of Engineering in the DIDET project at Strathclyde University also indicates the value of such shared workspaces for teachers and students to share and manage resources. Further, the need for a critical mass of resources within the repository is reported as critical for a viable implementation (Foster & Gibbons, 2005; Breslin et al., 2007). Breslin et al. also suggest that it may take two to three years for a cross-institutional repository project to become established and be able to offer benefits to others beyond the core implementation group. One of the design aims of the Carrick Exchange is to promote processes characterised by Stuckey and Arkell (2006, p.7) as “connection” not just “collection”. Therefore, diverse communities and database or repository models are being explored to determine the elements that contribute to success. Examples as varied as amazon.com (eCommerce), domain.com.au (real estate), Wikipedia.org (a collaborative encyclopaedia), Youtube.com (video sharing), Edna (collection and collaborative network) and MERLOT (educational exchange) are being reviewed. Their strategies for engagement and networking, using Web 2.0 technologies, may inform the development of personal and group workspaces on the Carrick Exchange. On amazon.com, for example, registered users are greeted by name and have delivered to them recommendations and updates about available products. The user can collate chosen resources from the site, and develop their own profile of recommendations, ratings and preferences. The system monitors users’ preferences, locates resources based on previous searches and feeds back information that connects users with other like-minded members. A mix of these features may be of value to the Carrick Exchange. MERLOT (http://www.merlot.org) uses similar techniques to support its educational community. Peer review and commentary The quality and currency of resources added to a repository is critical to ensure user needs are met. A national repository is well-placed to implement a formal peer review system. Peer review as quality assurance could include an assessment of the currency, educational design and construction of resources; compliance with copyright, intellectual property and digital rights management policies; and technical accuracy and reliability. Formal peer review, a lengthier and more demanding review process, might replicate the scholarly peer review process which leads to publication in the higher education sector. Peer review may also be an informal process whereby members of the community voluntarily respond to others’ contributed resources, or resources stored elsewhere but linked to via the Carrick Exchange. These resources might be finalised products which the authors publish and share widely or to a narrower select group of colleagues, or resources “under development”. This informal sharing of ideas could be an important element of the Carrick Exchange and the basis for various communities of practice. One of the best models of formal peer review for educational resources is that of the MERLOT system in the USA (McMartin, 2004; Nesbit, Belfer & Vargo, 2002). This system is based on the academic peer review practices for scholarship and publication in higher education, an “expertiseorientated” approach (Worthen, Sanders & Fitzpatrick, 1997). Peer review is recorded for formal recognition of contributors, and usage pattern updates are sent to contributors each year. Building on the MERLOT model, Nesbit et al. (2002) have created a convergent participation model for evaluation of learning objects where resources undergo a two cycle process: two individual experts assess the resource, and then a combined group assessment is made, amalgamating the feedback from both assessments. Students are included on the panel of reviewers. The Australian ACELL project, Advancing Chemistry by Enhancing Learning in the Laboratory (http://acell.chem.usyd.edu.au/), is an example of a discipline-based repository which also makes use of learner feedback in the evaluation process. Other repositories which implement peer review are Intute in the United Kingdom (http://www.intute.ac.uk/policy.html), and Educause in the USA (http://www.educause.edu). The Jorum national repository in the UK (http://www.jorum.ac.uk/) is currently investigating peer review processes. 20 Though peer review is seen as a value adding process, realisation of peer review processes is problematic. The Taylor and Richardson report on Validating Scholarship in University Teaching (2001) concluded that at the time of publication, there was a “window of opportunity” to establish a peer review scheme to assess information and communication technologies in Australian universities (p.87). The project set out to develop conceptual and procedural bases for a national scheme. Reviewers would be editors of journal and conference proceedings and it would be the responsibility of these editors to advertise the scheme. A supporting professional development strategy was to be developed and resourced. The project generated a number of options for the peer review process. Despite the positive outcomes of the project, the scheme has not been adopted nationally. Taylor and Richardson (2001, p.7) offer four reasons for peer review: • • • • Need for the evaluation of quality in ICT-based resources; Need for recognition for the developer of ICT-based resources; Need for the collaboration and dissemination of resources and knowledge relating to the design and construction of these resources; and Need for this quality assurance, recognition and collaboration to be grounded in the concept of scholarship. Taylor and Richardson stress that it is the focus on scholarship that is central to the “value-adding” work of academics (p.7). Peer review is seen as a means of validating and recognising that scholarship of teaching is important, and this is essential if effort around teaching is to receive parity with scholarly research endeavours in higher education. Recent changes to academic publishing have seen some journals try a move towards a more “open” approach to publishing which may be of interest to those conducting peer review of educational resources. One of the arguments against traditional blind peer review is that the reviews produced and the research accepted favour tradition and not innovation (Rogers, 2006). The peer review system evolved when dissemination was difficult and expensive. There is an argument that open publication may allow innovation to be disseminated more speedily. Anderson (2007) argues that the Web may become the first place of publication in the future, and only the very best and most enduring works will be published in paper. Cornell University’s open access 5 Library e-print site, arXiv (“archive”), has allowed pre-publication of papers in the sciences since 1991, and for these papers to be open for comment. The prestigious science journal Nature 6 began trialling a more open peer review process (Rogers, 2006) in June 2006. Authors can choose a 'pre-print' option. This pilot is now out of the beta stage and is called Nature Precedings (Nature, 2007). Once a research article has passed an initial quality check, the author posts their paper on the journal’s website, and anyone, provided they give their name and email address, can comment on the research; the traditional blind peer-review process continues in the background. The traditional method of review is lengthy, and can take between four and twelve months, after two or three reviewers have made their assessment, the editor has compiled and reviewed the reports, amendments are requested from the author and finally the work is published. The open approach where a pre-print is out in the public domain immediately the research article is completed allows for much quicker access. This could become a model for repositories like the Carrick Exchange which undertake peer review, where currency of resources is a considerable issue. Nature warns readers to treat the findings in any of the research published in Nature Precedings with caution as results may be preliminary or speculative, but all documents are citable (have a DOI), and are archived under a creative commons license where derivatives are allowed (Nature, 2007). Whilst the open review process speeds publication, the process of soliciting informal peer review comments was less than successful. Reaction to the trial from authors and scientists was mixed. While there was considerable traffic on the site where the pre-prints were posted during the four month trial, and the concept was well received, few readers commented on the papers. Survey results and analysis of the comments (Nature, 2006) indicated that: obtaining comments was 5 6 http://arxiv.org/ http://www.nature.com/nature/index.html 21 difficult; attempts to solicit comments produced only limited success; competition in the field may have been a reason for lack of input; potential commenters thought that open peer review was a good concept but still did not provide feedback; and editors’ analysis of the comments indicated that generally comments were low level and did not add to the review process. This reluctance to provide substantive comment in general on other sites was noted by some participants in the ascilite research. Nonetheless, the argument for early publication and dissemination provided in the Nature example probably overshadows the counter argument that informal peer review is likely to be minimal and of little real value for development. In furthering the debate on new directions in academic peer review and publishing, Siemens’ (2007) integrated model of formal peer review combined with informal peer review commentary is of interest 7 , and points to the emerging changes brought about by changes to technology: “Quite simply – as academics, we do not have a clear model of implementation for scholarship in light of current online trends”. The new democratic environment online brought about by Web 2.0 technologies raises issues of “identity, fairness [and] civility” according to Siemens, which need to be addressed with adaptable approaches. 7 http://www.elearnspace.org/Articles/journal.htm 22 Engagement guiding principles and recommendations ______________________________________________ Introduction As part of the development of the Carrick Exchange, ascilite was engaged to conduct research into: • What are the successful methods for establishing and maintaining engagement by the higher education community with the Carrick Exchange? o o o What are the key factors for establishing and maintaining engagement in academic and non-academic online communities e.g. online support groups, interest and resource sharing groups? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? The following protocols and mechanisms to foster engagement and uptake of the Carrick Exchange have been synthesised from knowledge and expertise existing in: the literature, current practice, exemplary community development to foster engagement with online services, and key practitioners and networks in both national and international settings. While one of the aims of the Carrick Exchange is to promote knowledge sharing amongst teachers and other professionals within the higher education sector, there is a general understanding that this is not an easy task. There is an expectation that resources of many types that support learning and teaching will be available for reuse and development via the Carrick Exchange, and that networks and communities will evolve around these knowledge sharing systems. However, research and practice suggest that establishing viable communities around repositories and exchange systems requires facilitation and an understanding of the needs of the users. The more diverse the target group, the more issues there are: sociocultural, pedagogical, organisational, technical, and information management issues. Through the ascilite research into engagement of the sector the aims have been to: • • • • • discover what methods and strategies might best apply in the Australian national context, and ultimately work towards improved learning and teaching praxis; investigate models of practice and exemplars for user engagement with both academic and non-academic repositories, including methods for development and maintenance of community; locate stakeholder groups and individuals with a need to collaborate and share resources within a resource information and networking environment; identify the most significant demand-side issues, especially in relation to understanding user needs and contexts of use; and recommend options for establishing and maintaining engagement with the Carrick Exchange of a range of users within the higher education community. With repositories and online services emerging world-wide, a general lack of user engagement has been identified as an ongoing cause for investigation and attention. This is considered the most significant risk to the success of any repository. In its entirety, this report represents a change management strategy to ensure uptake and engagement of the Carrick Exchange. 23 Recommendations in this section are structured around the following key themes that emerged from the ascilite research findings: 1. 2. 3. 4. 5. The purpose of the Carrick Exchange Multiple levels of engagement Management of engagement Ease of use Promotion and publicity Recommendations related to the purpose of the Carrick Exchange 1.1.1. The purpose of the Carrick Exchange should be well articulated to the sector. The Carrick Exchange is “a new online service that will provide learning and teaching resources, and functions to support communication and collaboration across the national and international higher education sector”. 8 The adequate promotion and communication of this purpose to the higher education sector will ultimately be a key factor in the success of the Carrick Exchange. One interviewee echoed this sentiment in commenting: The purpose of the Carrick Exchange and what it intends to achieve needs to be made quite clear to all key stakeholders and the intended audience from the very beginning (Interviewee). Efforts should be directed towards familiarising groups with the features, benefits and affordances of the Carrick Exchange and its potential to support user needs. A clear statement of the purpose of the Carrick Exchange with explicit details of the advantages of the system and how it is differentiated from existing institutional repositories, databases, work spaces and communities should be communicated to potential users. Defining the collection, its standards and quality assurance methods, and effectively communicating these is an essential strategy in the process of developing the Carrick Exchange, and enhancing user engagement. The Carrick Exchange has been developed for those who teach, manage and lead learning and teaching in Australian higher education. As such the target audience for promotion and publicity should not be limited to people applying for and winning awards and grants, but should be open to both academic and non-academic staff in the sector. Its currency and leadership in the education field should be promoted as reasons for using the Carrick Exchange. The Carrick Exchange should be central to all Carrick Institute functions. As one interviewee commented: It should sit across everything that Carrick does. [For] anything to do with the Carrick Institute, the Carrick Exchange is where you go. Even the forms for grant applications should reside on the Exchange. There has to be a degree of integration that makes it work, otherwise it will just be a great idea that’s not sustained over time. (Interviewee) It is recommended that the Carrick Exchange is central to communication and information provision concerning all Carrick Institute functions and activities, including notifications of events and provision of forms for registrations, grant applications and awards. 8 http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381 24 1.1.2. The quality of the processes, resources and functionality of the Carrick Exchange should meet the expectations of the sector. According to the research participants, the expectations of the higher education sector for the Carrick Exchange is that it will be the focal point for sharing ideas, resources and expertise about learning and teaching in the Australian higher education context. Research participants warned that there must be a clear demand or need for the Carrick Exchange, and that it must meet those demands or needs. Quality was seen as a key success factor for the Carrick Exchange. Therefore, a strategic plan and operational policies, procedures and systems should ensure that the Carrick Exchange delivers a superior standard of services and products. This includes: • • • accuracy and currency of resources, requiring ongoing maintenance of resources by originating authors; a reliable functioning system with a sophisticated search engine that is easy and intuitive to use and incorporates functionalities such as a federated search mechanism to search, harvest and use resources in other higher education institutional repositories; and robust “familiar” processes; as one interviewee suggested, “It is important to use familiar processes that the target audience are used to, for example most people are used to Google. So to mirror the search protocols on the Google search mechanism is a good idea” (Interviewee). Engagement and uptake of the Carrick Exchange is dependent on the experiences of users, especially the initial encounter. The experiences according to research participants must be positive and fruitful; which will occur if the Carrick Exchange establishes and ensures quality processes, resources and functionality. Strategies to convey the purpose, quality and scope of The Carrick Exchange Strategies for the Carrick Institute and the Carrick Exchange 1. Develop a strategic plan and associated policies to engage the higher education sector. This plan should focus on a staged process for the implementation of the Carrick Exchange and strategies to support engagement by the sector and should address issues of sustainability into the future. 2. Develop operational policies, procedures and systems to enable the accuracy and currency of resources to be maintained. 3. Ensure all Carrick Institute communication and information dissemination is managed through the Carrick Exchange. Strategies for the higher education sector 1. Enable access to existing higher education and institutional repositories. Technical strategies 1. Develop technical systems to provide: • Automated alerts established to contact originating contributors to update submitted resources, thereby maintaining accuracy and currency of resources. This should be copied to Carrick Exchange administrative staff for follow up; and 25 • a search engine that is easy and intuitive to use (perhaps modelled on Google) that has a federated search mechanism to search, harvest and access resources from other higher education institutional repositories. 2. Ensure the system is robust and well integrated when released. There is a risk in releasing the system to the whole sector before it is ready. Exemplars Intute example document about purpose of the repository and community: Intute (2006). The New Best of the Web. Ariadne, 48 (July). Retrieved September 1, 2007, from http://www.ariadne.ac.uk/issue48/williams/ Many repositories provide an ‘about us’ section that details information about the purpose, affordances, features and benefits of the repository. This feature clearly communicates accurate, relevant and current information about the repository to users. These include: • • • Connexions: Provides detailed information in the ‘about us’ section in relation to the philosophy, project history, key project stakeholders, and technology. http://cnx.rice.edu/aboutus/ EducaNext: Provide detailed information in the ‘about us’ section in relation to the governing organisation the service including case studies to illustrate aspects of use, and related and relevant publications about the service. http://www.educanext.org/ubp; MERLOT: provides an overall statement of the mission of MERLOT with links to additional pages detailing information about the resources, processes and functions of the system and services. http://taste.merlot.org/ Some repositories provide a similar communication of information through a link to “frequently asked questions” such as: • • • Carnegie Foundation for the Advancement of Teaching: Frequently asked questions are related to - Grants & Scholarships, Publications and Previous Work, the Carnegie Foundation and the collection classifications. http://www.carnegiefoundation.org/about/index.asp?key=17 Educause: has an extensive list of frequently asked questions across a broad range of relevant and related aspects. http://www.educause.edu/FrequentlyAskedQuestions/660 Health Education Assets Library (HEAL): questions are in relation to users, user groups, and management of user profiles, cataloging issues, searching, contributing and using resources, and technical issues. http://www.healcentral.org/utils/help/faq.jsp A number of repositories invite feedback from users about their experiences in the repository or invite users to report unsatisfactory aspects of the repository. “Contact us” links that generate an email addressed to repository managers are also provided by some repositories. Feedback functions enable manages to keep abreast of the expectations of users particularly in regard to processes, resources or the repository systems. Repositories providing this include: • Intute: invites users to lodge questions, comments or requests using a feedback form. Users are also invited to ‘suggest a site’ for inclusion in the Intute database of Internet resources. http://www.intute.ac.uk/ 26 • Jorum: has a submission form for feedback that collects the details of the person providing the feedback. http://www.jorum.ac.uk Key References Barton, M. & Waters, M. (2004). Creating an institutional repository, Learning About Digital Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved June 19, 2007, from docushare.lib.rochester.edu/docushare/dsweb/Get/Document-19824/IR%20bibliography.pdf Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning. London and Sterling: Kogan Page. Margaryan, A., Milligan, C. & Douglas, P. (2007). Community dimensions of learning object repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object repositories. Retrieved July 23, 2007, from http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Structured_Guidelines_v1p0.pdf Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible Learning Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian Flexible Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from http://flexiblelearning.net.au/aboutus/keydocuments.htm Recommendations related to multiple levels of engagement 1.2.1. A strategic plan for the engagement of the sector at multiple levels should be developed and implemented The Carrick Exchange needs to foster the engagement of individuals and institutions within the higher education sector. As one interviewee stated “Buy in from all stakeholders is required from the start if the initiative is going to be successful”. Senior staff (Senior Executives, ViceChancellors, Deputy Vice Chancellors, Pro-Vice Chancellors and Deans) will be key to getting the “buy in” of universities. The Carrick Exchange should also target: • • • • the engagement of special groups or individuals with an institutional responsibility for collaborating with teaching staff on development of teaching and learning resources; those disciplines with a history of sharing resources, particularly in the early phases of the project, to promote effective use of the system and provide models of usage and community building; staff support to help with orientation, troubleshooting, educational support and change management; and institutional Carrick Exchange Champions to raise awareness through such things as demonstrations, seminars, and workshops at their local level. Research by Rogers (2003) suggests five factors as critical for the adoption of technological innovations: • Relative advantage: potential users need to see an advantage for using the Carrick Exchange; 27 • • • • Compatibility: the Carrick Exchange must fit in with potential users’ current practice and values Complexity: the Carrick Exchange must be easy to use; Trialability: potential users must be able to test the Carrick Exchange before making a decision to adopt. For example they might be able to view the exchange without having to join; and Observability: potential users need to see the observable results of what the Carrick Exchange can do for them in their specific circumstance. This can be provided through exemplars or case studies of successful reuse of Carrick Exchange contributions. Exemplars could be made available through workshops or training sessions. According to Dormant (1997), when people are adopting a new innovation they move through stages of acceptance. Therefore strategies corresponding to a person’s stage of acceptance is required. Likewise, for optimum engagement in the Carrick Exchange, strategies that address the stages of acceptance across the sector should be planned and implemented. Table 2 provides an overview of the stages of acceptance and suggests possible strategies for the Carrick Exchange in each stage. Appendix E further explores these stages of acceptance and how each might be addressed through the example of a Carrick Exchange user group (Carrick Grant holders). The Carrick Exchange should endeavour to explicitly address the five factors for the adoption of technological innovations (Rogers, 2003), and the stages of acceptance (Dormant, 1997). At the same time the Carrick Exchange should also consider the various key stakeholder groups across the sector when developing a strategic plan for the engagement of the sector. 28 Table 2: Strategies for enabling acceptance and uptake Stage Of Awareness Awareness • • • Passive regarding the change Little/no information about change General Strategy Carrick Exchange Strategy Advertise Advertise Be an advertising agent Provide information on the website, email & phone details for 1st contact Appeal to his or her needs and wants Provide reasons for need for the Carrick Exchange Be credible and positive Have evidence of a critical mass of users, including highly regarded experts & leaders in the disciplines Little/no opinion about change Support Roles & Personnel CE Managers & Champions, CE promotional staff Provide reasons for use and added value provided by the CE in relation to other repositories, networks & communities Have evidence of national & institutional support for the CE. Local Champions Relevant associations (e.g. disciplinary groups with sub-groups focussed on teaching and learning; crossdisciplinary associations such as ascilite, ODLAA, HERDSA). Have evidence of a quality database of professional development & teaching resources derived from Carrick Institute projects Curiosity • • • More active regarding change Express personal job concerns Asks questions about own work and change Envisioning • • • Active regarding change Expresses work-related job concerns Asks questions about how change works Inform Inform Identify specific concerns Identify special needs of discipline groups & use ‘push’ technology to reach target groups (e.g. email, RSS feeds). Provide clear information about concerns Provide information at short f2f sessions or as part of other related activities. CE discipline-based Champions & collections managers Emphasise pluses, acknowledge minuses Demonstrate Demonstrate Give success images Provide digital stories & case studies on the CE CE & Local Champions, Provide demonstrations Provide 1-2 hour face to face sessions Staff & Educational Developers Connect with peer users Offer showcasing – stories from the members of CE sharing & utilising facilities Encourage registration so as to view activities of available communities & networks. 29 Stage Of Awareness Tryout • • • Active regarding change Has opinions about change Interested in learning howto General Strategy Carrick Exchange Strategy Train Train Provide effective training Support & provide local face to face training initiatives Offer training in peer review for the CE Provide job aids, checklists Promise technical followup Provide fact sheets, FAQs, guidelines, tutorial animations, and digital stories on the CE; Provide forums on CE for getting started, for Q&A, pedagogical and technical Support Roles & Personnel CE Trainers Help desk supporters Librarians Educational Developers Technical Writers Provide a database of members available to support others in specialist areas Use • • • Active regarding change Uses change on the job Asks detailed questions about use Support Support Provide necessary technical help Provide help desk and centralised support for resource submission issues Provide pedagogical help Provide support for embedding reuse and sharing into the curriculum Provide support clearing copyright, metadata entry and checking, rights management and version control Conduct conferences and workshops which focus on resource identification & submission, and peer review of submitted material. Provide recognition Provide reinforcement CE and institutional information specialists Educational and staff developers, Carrick and local Champions, CE community members Librarians, copyright managers Carrick Institute managers, Institutional managers Provide awards for contribution and excellence which support career advancement; link with promotional policies and procedures. Provide time, funding and recognition for preparing resources and activities which support the CE; for submission and maintenance of resources on the CE; for ongoing networking and mentoring those using the CE Source: Adapted from Dormant 1997; Key: CE = Carrick Exchange 1.2.2. Membership and the conditions of membership for the Carrick Exchange should be specified. The Carrick Exchange is being developed for those who teach, manage and lead learning and teaching in Australian higher education. This criterion for members needs to be clearly communicated to the sector so that those who qualify for membership are made aware of their entitlement. It is also imperative that the corresponding conditions of Carrick Exchange membership are communicated. Carrick Exchange membership conditions that might need to be developed and communicated include: 30 • • • • • • • Copyright and intellectual property; Rights associated with the various levels of user accounts (e.g. browsing, searching contribution and commentary rights, allowing only registered users certain privileges); Reuse of resources (e.g. licensing agreements, terms of use for materials accessed from the Carrick Exchange); Cost of membership (to the individual and institution); Membership of international stakeholders and professional associations; Code of behaviour guidelines for individuals, groups, communities of practice or institutions (e.g. detailing expectations such as non offensive contributions or commentary, adherence to legal practices, human rights and professional accountability); and Privacy requirements. Research participants saw a cost for membership of the Carrick Exchange as a disincentive to engagement. As such it is recommended that Carrick Exchange membership be free to all staff within the Australian higher education sector and that other methods are explored for sustainability through extended government funding or institutional commitment. 1.2.3. The Carrick Exchange should meet the diversity of members’ needs and contexts of use. An important process for the Carrick Exchange to undertake, in developing a strategic plan for the engagement of the sector is: 1. Identify the needs of the sector; and 2. Ensure that these identified needs are met. The shape of the Carrick Exchange and the networks and communities that develop will emerge with the diversity of users’ needs. Ongoing attention needs to be given to maintaining and growing the communities that develop around the Carrick Exchange, and supporting existing and future working relationships and cultures. To some degree communities should be able to self organise their own structures and facilities within the Carrick Exchange. The use of social networking software to facilitate communication, networking and collaboration amongst users is an important aspect of the Carrick Exchange and there is a need to promote and support the effective use of these tools. While the Carrick Exchange should have a discipline focus, cross-discipline and trans-discipline activities and initiatives are of interest to the sector. The Carrick Exchange should be seen as a mechanism for ongoing support of the diverse needs of members including collaboration. One interviewee noted that: The Carrick Exchange has to assist in the real work that is undertaken by staff in universities, so it has to add to actual real stuff that people are doing on a day-today basis, and not be seen as something that is just an extra. (Interviewee) The introduction of solutions, which the Carrick Exchange can provide, to meet the needs of members should be managed in a strategic manner. Hummel et al. (2005) suggest that this system of introduction should begin with a minimal set of resources, activities and forums that meet the needs of the user or community. Gradually as participation increases and users’ needs develop, appropriate mechanisms and offerings are introduced. The promotion of these introduced solutions may be supported through synchronous collaboration and communication. 31 Strategies to engage multiple levels of users Strategies for the Carrick Institute and the Carrick Exchange 1. Devise strategies of engagement relevant to: • • the varying levels of engagement (See Table 2); and different key stakeholders including international members, active communities and professional associations, and disciplines already engaging within repositories (e.g. engineering and science) 2. Provide access to examples of how communities, groups and individuals are effectively using the Carrick Exchange particularly from teaching and learning support areas such as libraries and student learning centres. Provide user scenarios and case studies indicating how the Carrick Exchange can work from several points of view. 3. Target groups or individuals (e.g. librarians, staff and educational developers) particularly in the early phases of uptake, and assist these groups to effectively use the system including Web 2.0 technology features. Exemplars could be drawn from these targeted cases. 4. Identify key leaders in teaching who do not readily engage with technology and investigate what possible incentives would lead them to engage with the Carrick Exchange. Also investigate the incentives for engagement in the Carrick Exchange through networks of new and sessional teachers, who might have a real need for high quality teaching resources. 5. Provide tips and links on how to, e.g. use particular resources in teaching, implement new pedagogical ideas, use new technologies. 6. Define the terms and conditions of membership of the Carrick Exchange Strategies for the higher education sector 1. Involve Carrick Fellows and grant holders with the Carrick Exchange. Technical strategies 1. Provide multiple levels of access and permissions to the Carrick Exchange e.g. view, browse, review, collaborate. 2. Develop a tour of the Carrick Exchange (e.g. tutorial, overview) with examples of specific actions, searches and details of how the functions of the Carrick Exchange can be used by communities, groups and individuals. Exemplars Examples of multiple levels of engagement are found within repository “Conditions of membership” documents, such as those found on: • • • EducaNext Code of Behaviour and terms of use: http://www.educanext.org/ubp/PUSH/usermgmt@regrTermsOfUse Educause Membership Terms and Conditions http://www.educause.edu/Membership/5 Jorum “User Terms of Use”: http://www.jorum.ac.uk/user/termsofuse/index.html Communities develop and grow in online environments such as Flikr (http://www.flickr.com/) Delicious (http://del.icio.us/?url), Youtube (http://www.youtube.com/), and Facebook (http://www.facebook.com/home.php?). Strategies used by these online communities to engage members include: • Providing mechanisms for individuals to share resources such as photos, news and descriptions of websites, with identified people, groups and networks; 32 • • Providing mechanisms for individuals to manipulate/modify/develop/enhance and organise things (e.g. photos, videos, favourites); and Providing notifications, suggestions and alerts of items that may be of interest (e.g. videos that match or align to previous video search queries; the recent activities of others within the network) Access permissions categories used by the Rights and Rewards in Blended Institutional Repositories project (http://rightsandrewards.lboro.ac.uk/index.php?section=1): include • • • • Private (just for own use); logged on user access; restricted open access; and public access. Key References Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan, and P. MacGillis (Eds.), The guidebook for performance improvement: Working with individuals and organizations. San Francisco: Pfeiffer. Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating participation: From EML web site to the learning network for learning design. Interactive Learning Environments, 13(1-2), 55–69. Margaryan, A., Milligan, C. & Douglas, P. (2007). CDLOR Deliverable 9: Structured Guidelines for Setting up LORs. Retrieved September 2, 2007, from http://academy.gcal.ac.uk/cdlor/documents/CD-LOR_Structured_Guidelines_v1p0_001.pdf Rogers, E. (2003). Diffusion of innovations. New York: Free Press Recommendations relating to the management of engagement with the Carrick Exchange 1.3.1. The Carrick Exchange policies and practices should be aligned strategically with institutional policies and practices. The Carrick Exchange is being developed for the Australian higher education context. The governing policies and operational practices of this broad target group are not currently aligned. However, as a national online service the Carrick Exchange policies and practices will need to align to those of institutions thereby alleviating obstacles to engagement. It is recommended that the Carrick Exchange policies and practices: • • • are developed in consultation with institutions to ensure alignment of policies and practices particularly in regard to practices of sharing, ownership of resources, intellectual property, moral rights, and copyright; are aligned to national, higher education and institutional initiatives; and offer suggestions for consistency and alignment in the ways that universities engage with the Carrick Exchange. 33 Engagement with the Carrick Exchange should be recognised by Australian institutions as an important part of core teaching and learning activities. 1.3.2. The Carrick Exchange should engage with the National Authentication Framework to ensure integration. The Carrick Exchange, as a national initiative, should be integrated and authenticated with existing national infrastructure and technical systems, for example: • • • The Australian National Data Service (ANDS), part of National Collaborative Research Infrastructure Strategy (NCRIS) project, the Australian Access Federation Project, and the Australian Research Repositories Online to the World (ARROW) project. Research participants supported the notion of a trusted, easy to use authentication system, one interviewee explaining: [There is a need for] spaces outside their own institutions that are easily accessible and not restricted by institutional firewalls, administrative practices and other technical constraints. People use what is easy, especially as there are so many people out there who do not use technology to a great degree. (Interviewee) Linking the Carrick Exchange to existing university databases is also important. Seamless connection and the ability to search the Carrick Exchange when also conducting institutional database searches was identified in the ascilite research findings as an important design feature. This integration of authentication and databases will simplify the number of search systems that users have to understand and operate. 1.3.3. A strategy should be developed to support, encourage and manage champions to promote the Carrick Exchange. Promoting a “human face” to the Carrick Exchange will be a key success factor. Developing networks and communities will require people as well as technology. Advocates and champions from within the Carrick Institute and the higher education community should be identified. These champions are critical as leaders, driving and promoting the Carrick Exchange and enlisting help from others within institutions and the sector to encourage and facilitate engagement. Carrick Fellows and grant holders may be potential champions. However, champions should not be just the current Carrick grant holders, or people well-known in the higher education sector. Engagement with the Carrick Exchange has to go deeper and broader than this so choosing people with the right attributes to be advocates and supporters, and resourcing them adequately is important. There should be a good balance across disciplines, gender and role, including faculty based academics, support staff and librarians. Educational developers, staff developers and librarians may be the most common and frequent users of the Carrick Exchange. They are potentially important institutional champions along with senior leaders in teaching and learning. 34 1.3.4. Partnerships with Australian universities should be established to ensure longevity and sustainability of the Carrick Exchange. Managing the issues of sustainability and ongoing support for the Carrick Exchange is critical. Carrick Exchange resources and functions that support communication and collaboration across the higher education sector must be well maintained into the future. The investment of financial and human resources to establish and operate the Carrick Exchange must not be lost as a consequence of limited long-term planning. As one repository representative noted: One thing that has been learnt from the experience of CLOE is that there is a need for long-term stable funding, not reliant for example on grants and project grants but possibly some sort of Government funding. (Interviewee) Strategies for the Carrick Exchange to maintain momentum, currency, quality and relevance after the initial phases of implementation should be negotiated and explicitly established across the sector. A Memorandum of Understanding should be developed to establish the ongoing commitment and a partnership with Australian universities to ensure the longevity and sustainability of the Carrick Exchange. 1.3.5. Collaboration should be encouraged and supported as a means of fostering engagement and contribution to the Carrick Exchange. While institutions are keen to collaborate, there are issues of competition in resource development that can hinder collaborative ventures. The Carrick Institute is already playing a key role in addressing this dilemma through the Carrick Institute Grants Scheme 9 . A clearly articulated goal of the programs in this scheme is “to facilitate innovation and collaboration”. The Grants Scheme is just one way that the Carrick Institute is addressing and establishing a culture of sharing and collaboration across the sector. The Carrick Exchange is also supporting this culture in the provision of a service that supports communication and collaboration. Use of the Carrick Exchange for collaboration and communication in Carrick projects should be promoted and recognised. There was even the suggestion by many of the ascilite research participants that the Carrick Institute should make it conditional that Carrick Institute grant holders contribute to the Carrick Exchange as part of their obligations in return for financial support. 1.3.6. The Carrick Exchange should support the needs of networks and communities of practice. The target audience for the Carrick Exchange will comprise existing networks and communities of practice within the higher education sector. These networks and communities have communication and collaboration needs that the Carrick Exchange may be able to support. The Carrick Exchange should: 1. Identify the specific needs of user groups - networks and communities; and 2. In partnership with the network/community integrate the resources and functions of the Carrick Exchange that address the identified needs of the group. As well as addressing the specific needs of existing networks and communities, the Carrick Exchange should provide: 9 http://www.carrickinstitute.edu.au/carrick/go/home/grants 35 • • • • Details of networks and communities of practice, and the facility to contact members of these groups; A network of contacts within the higher education community that can contribute to and inform areas of interest to individuals; Links to resource creators and their related networks; and Alerts about new community items and shared spaces. 1.3.7. An evaluation plan should be developed to measure levels of engagement including participation, collaboration and contribution. Regular evaluation of the Carrick Exchange, measurement of uptake and engagement to inform ongoing development and maintenance of the repository and its community should be undertaken. There is a need to have processes to report how those extracting content from the repository are using that material. This information would be very useful in demonstrating the importance and value of the repository and community to key stakeholders and potential funders. Mechanisms to track and evaluate how useful the forums and other communication exchange areas are to members would also be advantageous. The project’s evolution should be tracked from the earliest stages. Other repository managers have reported that this is a key element in developing a successful service. If managed through traditional research practices, this information will ensure that the ongoing lessons learned can be properly documented and evaluated. The Carrick Exchange has to be able to demonstrate its value if it is to be regarded a success. The value of the Carrick Exchange has to be seen in relation to the needs of the primary target audience – university staff. Therefore it is important that the evaluation plan gather information specifically detailing the uptake and engagement of these users. Strategies to manage engagement Strategies for the Carrick Institute and the Carrick Exchange 1. Invite universities to identify an institutional representative to be part of an expert reference group to inform ongoing aspects and activities of the Carrick Exchange, including the development of Carrick Exchange policies and practices and to encourage and support institutional “buy in”, collaboration and partnership with the Carrick Exchange. 2. Promote engagement by using Carrick Exchange staff, institutional champions or project coordinators who “travel around” and work with academics and support staff, within institutions. 3. Support secondments for individuals to take on specific Carrick Exchange promotional and operational roles, e.g. peer reviewing, holding events, publicising resources, putting groups and individuals in touch with one another, providing regular updates and newsletters. Thereby presenting a “human face” to the Carrick Exchange for enhanced engagement. 4. Ensure that rigorous data collection of the project’s evolution is facilitated from the start. Managed through traditional research practices, this information should be properly documented and a critical research lens applied to the processes, resources, development schema and workflows to be evaluated. Strategies for the higher education sector 1. Support the development, promotion and management of the Carrick Exchange in partnership with the Carrick Institute 2. Enable institutional champions to engage in one on one conversation and dialogue with intending users to enhance engagement, through a “just-in-time” approach. 36 3. Enable uptake of secondments from the institutional point of view, for individuals to take on specific Carrick Exchange promotional and operational roles. 4. Support Champions to develop an institutional network to promote the value and worth of the Carrick Exchange. Technical strategies 1. Develop mechanisms for reporting how those extracting content and resources of any kind from the Carrick Exchange are using that material. The information collected should be able to demonstrate the importance and value of the Carrick Exchange and community to key stakeholders and potential funding bodies. 2. Ensure users’ contact details are registered when downloading a resource from the Carrick Exchange. 3. Develop mechanisms to enable an annual audit of resources that provides concrete measures and details of the various resources. Exemplars LORN, Learning Object Repository Network Coordinators: These coordinators in the VET sector, as part of the Australian Flexible Learning Framework, act as champions in their state jurisdictions. These champions encourage and support members of the VET community to access high-quality teaching and learning resources that support flexible delivery from the learning object repositories. http://lorn.flexiblelearning.net.au/Home.aspx Partnerships with current institutional repository owners: Institutional learning object repository owners such as Deakin, RMIT, Griffith, Monash etc., should be approached to establish links with the Carrick Exchange, along with any research repositories associated with the ARROW project. http://arrow.edu.au/repositories Key References Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities. D-Lib Magazine, 13(5/6). Retrieved September 27, 2007, from http://www.dlib.org/dlib/may07/henty/05henty.html Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating participation: From EML web site to the learning network for learning design. Interactive Learning Environments, 13(1-2), 55–69. Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible Learning Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian Flexible Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from http://flexiblelearning.net.au/aboutus/keydocuments.htm Wenger, E., McDermott, R. & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowledge. Boston MA: Harvard Business School. 37 Recommendations related to ease of use 1.4.1. The Carrick Exchange should devise strategies of engagement for the varying levels of technical capability of users. Promoting the benefits and affordances of the Carrick Exchange must be supported by the positive experiences of users. If the engagement of users is not positive they will not return. This was a view firmly held by one interviewee who stated: One of the dangers for engagement is that if the promotion of the repository commences before the system is working efficiently and effectively, users are not just put off for that time, they're put off always. This applies to both accessing and contributing to the repository. (Interviewee) The fundamental requirement for ensuring positive engagement in the Carrick Exchange is ease of use of the system, as one interviewee, in echoing the sentiments of numerous research participants explained: “[The Carrick Exchange] needs to be as quick and easy and simple as possible to use” (Interviewee). There will be varying levels of technological skill, ability and experience across the target audience. Therefore, the experience and expertise of all users should be addressed in the functionality of the system and the additional support provided not only through documentation but through personal assistance. 1.4.2. The Carrick Exchange should be easily accessible and intuitive to use. The Carrick Exchange should be easily accessible and not restricted by institutional firewalls, administrative practices and other technical constraints. Research participants were adamant that unless these barriers to engagement were removed people would not use the Carrick Exchange. As one interviewee noted: One of the other hindrances to using a repository, and ultimately for engagement is that if the process of finding something, grabbing it and reusing it is too difficult, if it's not intuitive, users will be put off. (Interviewee) People use what is easy. Many of the target audience may be those who do not use technology often, consequently the mechanism for conducting a search and using the functions of the Carrick Exchange must be intuitive. The Carrick Exchange needs to have an intuitive common sense approach to the search engines and architecture of the site. (Interviewee). In the early stages, all searches should return something useful, and collections that are low in resources should not be advertised or made available until there is a critical mass of materials or commentary. Ease of use should also extend to seamless authentication and straightforward metatagging procedures. Strategies to ensure ease of use Strategies for the Carrick Institute and the Carrick Exchange 1. Provide training and support with respect to: technical capability; the use and operation of the Exchange; and development of teaching and learning expertise and practices to integrate the interactive and innovative resources available through the Carrick Exchange. 38 2. Work in partnership with universities to provide collaborative spaces that are easily accessible and not restricted by institutional firewalls or administrative practices. Strategies for the higher education sector 1. Institutional champions should provide and complement training and support offered by the Carrick Exchange with respect to: technical capability; the use and operation of the Carrick Exchange; and development of teaching and learning expertise and practices. 2. Assist the Carrick Exchange to provide users easy access to the system that is not restricted by institutional firewalls or unnecessary administrative practices. 3. Provide details of networks and communities of practice to be advertised in the Carrick Exchange. 4. Provide details of a network of contacts within the higher education community that can contribute to and inform areas of interest. Technical strategies 1. Ensure that the Carrick Exchange is easily accessible and not restricted by technical constraints. 2. Provide the facility to enable users to contact members of communities through functions within the Carrick Exchange. 3. Maintain an up-to-date tour of the Carrick Exchange (e.g. tutorial, overview) with examples of specific actions, searches and how people are using the Carrick Exchange. 4. Include an online help function. Exemplars Facebook: An online social networking environment that promotes engagement and interaction amongst members through resources and functions including: • • • • a space in which items such as photos can be stored and shared with friends; networks for users to join, and communicate and share with; web-based games and social interaction activities; and a support system for external parties to create plug-ins to enhance the system. Facebook originally developed in the college sector in North America. It signs up more than a million new members every month (AFP, 2007). Facebook reports that it currently has 200,000 people sign up to it every day and has amassed 42 million users in the three years since it was created in 2004. Facebook is an intuitive, easy to use environment that caters to the varying technological abilities of members. It offers users control over privacy settings for their profiles, photos etc. http://www.facebook.com Intute ease of use: The Intute repository is easy to navigate and locate content. http://www.intute.ac.uk/ Librarything.com: Cataloguing of books online. Works like an international book club. Uses social networking software to create groups, blogs, personal profiles, tags and links to others with similar interests http://www.librarything.com/ Wikipedia: Free online collaborative encyclopaedia. Members create the content which is all fully editable. Their code of conduct assumes that users will behave well. When they do not the community tends to work to improve matters. Generally their code assumes respect for other members even when you may not agree with them; civility and openness amongst members; avoidance of personal attacks and use of sweeping generalisations. Some of Wikipedia’s 39 services include help and reference desks, volunteers to answer questions, a community portal and a “village pump” for discussions about the site itself, site news and a “local embassy” for communication in languages other than English. http://en.wikipedia.org/ Key References AFP. (2007). Facebook to post members profiles on online search engines. Retrieved September 27, 2007, from http://afp.google.com/article/ALeqM5iWsMIytjvcGtnN9Q6ob6X3YHC3BQ Hand, T., Gosper, M., Woo, K., Gibbs, D., Kerr, S. & Rich, D. (2004). Learning objects: User perspectives on the conditions surrounding their use. Proceedings of the Ed-Media 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications, 21-26 June, Lugano, Switzerland, 66–72. Margaryan, A., Currier, S., Littlejohn, A. & Nicol, D. (2006). Community dimensions of learning object repositories, CD-LOR deliverable 1: Learning communities and repositories. Retrieved August 30, 2007, from http://academy.gcal.ac.uk/cdlor/CDLORdeliverable1_learningcommunitiesreport.doc. Margaryan, A. & Littlejohn, A. (2007). Cultural dimensions of learning object repositories. Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and Telecommunications, Vancouver, Canada, 25 -29 June, 4335-4343. Mckenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, adoption and adaptation of project innovations in higher education. A report for the Carrick Institute for Learning and Teaching in Higher Education. Sydney: University of Technology. Recommendations related to promoting engagement with the Carrick Exchange 1.5.1. There should be a strategic plan and adequate support for the promotion and publicity of the Carrick Exchange to ensure successful engagement by the sector. Promotion of the Carrick Exchange is critical to successful engagement by the sector. The Carrick Exchange needs to be proactively promoted and publicised within universities to ensure members of the sector engage with it, know where it is and how to use it. Effective engagement strategies and adequate resourcing are required in support of this. Promotion and publicity of the Carrick Exchange should be strategic and include: • • Communication of information about the benefits and affordances of the Carrick Exchange; and Promotion of the rewards that ensue from engagement with the Carrick Exchange. Many of the research participants recommended that promotion of the Carrick Exchange was imperative for successful engagement with comments from interviewees such as: 40 It is important to promote the repository especially in face-to-face forums. Taking the time to go out and tell people about the repository and explain the details about the repository is very important. (Interviewee) A strategic plan for short- and long-term promotion and publicity of the Carrick Exchange should be developed. Once developed it is critical that this plan is adequately resourced (both financially and with personnel) to ensure that the strategic goals and objectives of the plan are met. 1.5.2. There should be regular opportunities to disseminate and encourage engagement of the higher education sector with the Carrick Exchange. The ascilite research findings identified “push” technology (e.g. alerts to notify when new items have been submitted in a particular area of interest or by a particular person and regular updates, such as electronic newsletters which inform members of recent developments and contributions) as an important design feature for the Carrick Exchange. One interviewee commented: A lot of good things get created that people don’t ever know about and unless individuals go and look at the Exchange they will not know what is available to them through it. So the Exchange needs to have some sort of push mechanism that brings to the attention of users new and worthwhile resources. (Interviewee) As a strategy for promoting engagement “push” technology can facilitate and sustain discussions or remind and inform members and potential users about the Carrick Exchange, its resources and functions. Members should be able to adjust the level of engagement and amount of publicity they receive. 1.5.3. Web 2.0 technologies should be promoted and supported as a mechanism for enhancing engagement with the Carrick Exchange. The Carrick Exchange should provide access to communication technology (e.g. web conferencing for virtual meetings, technological support tools with automated feedback on learning), and Web 2.0 tools for collaboration and social networking. Through harnessing the affordances of the technology and engaging elements that may not be available in other repositories and collaborative spaces, the Carrick Exchange will become an exemplary model for the higher education sector; as such the provision and promotion of these technologies as a supported feature of the Carrick Exchange is important. According to Evans (2007) the critical elements for successful use of social software are: 1. “Authenticity” – a real and established need to use the software; 2. “Relevancy” – its use is relevant to the need, and appropriateness for the client; and 3. “Support” – for the software within an “enabling culture”. These elements for successful use of social software should inform and be addressed in the overall Carrick Exchange strategic plan for promoting and enhancing engagement. 41 Strategies to promote engagement Strategies for the Carrick Institute and the Carrick Exchange 1. Facilitate an annual Carrick Exchange conference to promote and publicise the resources and functions available to potential users to support communication and collaboration across the sector. 2. Promote the Carrick Exchange at all face-to-face forums. 3. Support champions who are driving and promoting use of the Carrick Exchange in institutions. 4. Promote the benefits of contributing to the Carrick Exchange such as contributing to: the advancement and sustainability of teaching and learning; the sharing of resources and ideas with colleagues within and across disciplines; project dissemination at national and international levels; promotion of achievements and scholarship in resource development; the sharing of ideas about teaching and learning; making more accessible and archiving publicly funded project output; and networks and communities of practice, both disciplinary and cross-disciplinary based. Strategies for the higher education sector 1. Promote the Carrick Exchange at all face-to-face forums. 2. Build use of the Carrick Exchange and its resources into existing institutional programs for staff (e.g. Foundation teaching and higher education programs, and Graduate Certificates). 3. Enable champions who are driving and promoting use of the Carrick Exchange in institutions. 4. Provide professional development funding to promote the Carrick Exchange in local and institutional forums. 5. Showcase achievements, expertise and effective use of resources as an important means of promotion and publicity. 6. Disseminate information about the Carrick Exchange via regular online newsletters with links to stories and updates about projects. 7. Follow the strategies suggested by Margaryan, Currier, Littlejohn and Nicol, (2006, pp.4-5) (see Literature Review section of this report) to address socio-cultural issues, pedagogic, organisational and information management, and technological issues. Technical strategies 1. Ensure the Carrick Exchange is continuously refreshed by establishing mechanisms and processes to keep resources, contact details, network and communities’ information and system functions current. 2. Identify on the Carrick Exchange ‘”Resources of the Week/Month” with a rating, based on resources that are most often accessed or the newest peer reviewed resources. 3. Provide user scenarios detailing how the Carrick Exchange has been used in different contexts. Scenarios could include details about design, contexts of use, implementation exactly what and how people interact with the system and each other. Exemplars Amazon.com: This e-commerce site has a system that greets registered users by name and delivers recommendations and updates about available products. The user can collate chosen resources from the site, and develop their own profile of recommendations, ratings and preferences. The system monitors users’ preferences, locates resources based on previous searches and feeds back information that connects users with other like-minded members. A mix of these features may be of value to the Carrick Exchange. http://amazon.com 42 MERLOT: Uses similar techniques to support its educational community, with a strong focus on disciplinary communities. MERLOT also holds regular conferences. http://www.merlot.org CLOE stories: Examples of learning object development and case studies. http://cloe.on.ca/stories.html Key References Evans, V. (2007). Networks, connections and community: Learning with social software. Australian flexible Learning Framework, Australian National Training Authority and Commonwealth of Australia. Retrieved September 27, 2007, from http://www.flexiblelearning.net.au/flx/go/pid/377 Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating participation: From EML web site to the learning network for learning design. Interactive Learning Environments, 13(1-2), 55–69. Recommendations for engagement policies Policies will need to be developed in the following areas. Barton and Waters (2004) in the LEADIRS Workbook also have some useful tips about policies. 1. Policy for national integration with Australian universities seeking executive support for all activities and functions of the Carrick Exchange, linking contribution with recognition and career advancement. 2. Membership policy: a. Who can be a member? e.g. the primary members are those who teach, manage and lead learning and teaching in Australian higher education, and includes individuals, institutions, consultants, professional associations and organisations. Other target members may be international guests, and possibly students. b. Preferably management of membership via trusted authentication, not self-asserting systems. 3. Terms and conditions of use policy covering: a. b. c. d. e. f. Vision statement giving guidance and direction overall; Acceptable use; Appeals policy; Use of tools within the site; Submission policy; The standard of the content (digital assets and commentary) that may be contributed or expected to be found on the Exchange; g. Copyright, intellectual property, and moral rights; h. Privacy and permissions; i. Accessibility. 4. A code of conduct similar to the Wikipedia code should be adopted (http://en.wikipedia.org/wiki/Wikipedia:Policies_and_guidelines), and an appeals policy for disputes is required. 43 a. Respect other members even when you may not agree with them. b. Be civil and act in good faith. Be open and welcoming. c. Avoid making personal attacks or sweeping generalisations. 5. A reporting policy, stating how the Carrick Exchange: a. b. c. d. e. Tracks usage; Reports on uptake; Provides members with feedback on usage; Protects privacy; Evaluates development of the system over time. 6. Policy for engaging different stakeholders and discipline bases: a. A funding and management model to manage initial engagement, allowing for future scalability and sustainability. b. Procedures to target special groups or individuals to establish an initial critical mass of users of the system. c. Management of Carrick Champions and champions within the sector. d. Multiple strategies to manage staff engagement at all levels of universities. e. Procedures to manage and generate activities, forums, workshops, conferences with discipline and cross-discipline focus. 7. Support policies to encourage uptake, integration into everyday praxis within the higher education sector, and management of educational and cultural change targeting: a. Educational and staff support b. Rights management, searching and metadata support, i.e. librarians. 8. Active dissemination policies that acknowledge: a. Excellent contributions; b. The latest contributions to the Carrick Exchange; c. Award of the month and an annual award that is presented with Carrick citations or integrated with the criteria. 44 Resource identification and contribution guiding principles and recommendations ______________________________________________ Introduction As part of the development of the Carrick Exchange ascilite was engaged to conduct research into: • What resource contribution and identification methods will engage and encourage users to contribute and collaborate within the Carrick Exchange? o o o o What are the key factors to support resource contribution, identification and reuse? What types of resources will be contributed? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? The following protocols and mechanisms for resource identification and contribution for the Carrick Exchange have been synthesised from knowledge and expertise existing in: the literature, current practice, exemplary community development to foster engagement with online services, and key practitioners and networks in both national and international settings. As well as the communities, networks and workspaces expected to establish around the Carrick Exchange, a repository of quality learning and teaching materials will be developed. Carrick Exchange resources will either be deposited on the site or linked from websites and databases elsewhere. The term “resources” is taken to include a broad range of formal and informal items in digital format, e.g. products, processes, learning objects and designs, activities, teaching strategies and tips, reports, assessment criteria, reviews, discussions on issues relating to leading and managing teaching. In addition, people are very important to the Carrick Exchange and considered “resources” themselves. In general, defining and maintaining the collection as a quality resource that meets the needs of the sector, overcoming barriers to contribution, and finding the right incentives that will reward contribution of digital resources and commentary will be major challenges for the Carrick Exchange. Recommendations in this section are structured around the following key themes that emerged from the ascilite research findings: 1. 2. 3. 4. Resource identification The culture of sharing in higher education Rewards, incentives and recognition for resource contribution Intellectual property (IP) and rights management 5. Management of resource contribution 45 Principles related to resource identification 2.1.1. The Carrick Exchange should strategically target resource contributors. By identifying key contributors to the Carrick Exchange the likelihood and chance of resources being submitted to the collection is enhanced. These key contributors can themselves be regarded as ‘resources’. Specific groups or individuals who can make valued contributions should be targeted including e.g. discipline groups; Carrick Institute grant holders; and; educational and staff developers. A key finding of the interviews with practitioners in the field was that the Carrick Exchange could have an important role providing shared workspaces for collaborative and cross-institutional groups. There is also an emerging and important role for librarians regarding resource discovery and identification. Librarians are well placed to identify and target special contributors and disseminate knowledge about the system. Support and professional development staff within universities are also likely to identify where crossdisciplinary connections can be made that link with Carrick Exchange initiatives. Further, Carrick Exchange Champions could network across institutions, seek out key academics and support staff within institutions and professional associations to act as mentors and initiate contribution to the Carrick Exchange. Carrick Fellowships may be one way in which particular resource development could be commissioned and promoted. The notion of identifying resources worthy of publishing to the Carrick Exchange and sharing across the sector should be carefully and strategically introduced, as this is not yet common practice. It might be advantageous to start small and target cross-institutional projects with a need for collaborative workspaces. Also, sharing may be more likely to occur in programs where there is team teaching and large student cohorts. Finally, one of the key objectives of the Carrick Institute is to ‘raise the profile and encourage recognition of the fundamental importance 10 of teaching in higher education institutions and in the general community’. This objective could be achieved through the engagement and commitment of the higher education sector, to the Carrick Exchange. A fundamental factor for this is to ensure that those who contribute towards resource identification are adequately recognised and rewarded. 2.1.2. Existing resources that could be contributed or linked to the Carrick Exchange should be identified. One of the keys to the success of the Carrick Exchange is the range, extent and quality of the contributed resources. If the resources meet the needs of users and are easy to access, then the Carrick Exchange will become known as a central ‘hub’ for teaching and learning resources, and the associated supportive communities and technologies. Locating resources that give insight into exemplary pedagogical practice is important: It’s not the stuff per se that matters … what’s really important is the creation of the design, the approach, the tool - with good examples of application where other people can come in and share their view and their examples. (Focus group participant) Initially the Carrick Exchange should link to other existing repositories rather than store large numbers of resources on the site. The identification of resources for the Carrick Exchange should align with the needs of members and the Carrick Exchange should continue to evaluate the relevance and currency of the collection. A discipline focus is important from the outset: For the Carrick Exchange to have as wide an impact and accessibility as possible it needs to have disciplinary presences so that users can see that the resources within the Carrick Exchange have relevance to them and their needs. (Interviewee) 10 http://www.carrickinstitute.edu.au/carrick/go/home/about 46 Resource content types that the Carrick Exchange could aim to identify as a priority are: • • • • • Professional development and teaching support documents that are the outcomes of Carrick Institute projects and can be made publicly available; Reports and updates on Carrick Institute funded projects in their development stages - prior to completion, as well as final project reports; Resources characterised as ‘works in progress’, i.e. developing resources, which may be linked to and open for comment from select members to assist and inform development; Secondary supporting documentation – this documentation supporting specific resources could outline design, development, use, and contexts of use and re-use and function as a guiding framework to accompany some resources contributed to the Carrick Exchange; and Generic teaching resources e.g. tips and strategies, learning designs, assessment ideas, problem-based and case-based learning and group management resources, laboratory activities, and research-style projects. Strategies for advancing resource identification Strategies for the Carrick Institute and the Carrick Exchange 1. Facilitate conference days or workshops with the aim of identifying resources and potential contributors (using the Camp CLOE model for inspiration). Different user groups should be targeted for different days and events. 2. Identify a range of high quality resources from existing institutional and higher education repositories and collections to provide a critical mass of resources in the initial phases of the Carrick Exchange. 3. Provide seed funding for the development of resources for the Carrick Exchange, targeting contributors to produce resources likely to be reused. 4. Target Carrick Grant holders to produce resources for the sector as part of the ‘exchange process’, in return for the investment granted to them by the Carrick Institute. 5. Target collaborative teams and programs, e.g. discipline based initiatives, academics teaching large classes, and cross-institutional projects for contributions to the Carrick Exchange. 6. Adopt the MERLOT Teaching Commons model whereby champions and task forces work with discipline-based teams to create a critical mass of resources for publication as a collection. Partner status with MERLOT allows for reciprocal services between the institution and MERLOT regarding implementation. Strategies for the higher education sector 1. Identify existing high quality resources in repositories and collections for contribution to the Carrick Exchange. 2. Target institutional collaborative teams and programs, e.g. discipline based initiatives, academics teaching large classes, and cross-institutional projects for contributions to the Carrick Exchange. Technical strategies 1. Provide shared workspaces that support collaborative and cross-institutional teams. 2. Provide mechanisms for individuals and groups to nominate resources they have identified for possible contribution to the Carrick Exchange. 47 Exemplars California State University (CSU) Teaching Commons (USA), which links to MERLOT. http://www.cdl.edu/cdl_projects/teachingcommons_home CLOE (Canada): Co-operative Learning Object Exchange: Camp CLOE is an intensive collaborative training and resource development event for partner institutions (for faculty developers, instructional designers, faculty, learning centre personnel, summer student workers etc.) to assist participants in the design of learning objects. Collaboration on research and learning object creation is the focus. http://tlc.uwaterloo.ca/projects/cloe/CaseStory/ Intute (UK): See their collection and development framework (quality consistency and interoperability). Intute employs a network of subject specialists to find, evaluate and catalogue the best resources of the Web. http://www.intute.ac.uk/policy.html See also the guidelines for linking the repository and searches of the content to personal web pages. http://www.intute.ac.uk/integration/ MERLOT Teaching Commons (USA): Exemplary discipline based method for identifying and sharing content and ideas. http://www.merlot.org/merlot/materials.htm See also MERLOT collections: http://taste.merlot.org/merlotcollection.html Key References Australian Partnerships in Sustainable Repositories APSR. Populating a repository – selected references (bibliography). http://apsr.anu.edu.au/apsrfw/rr/index.php?pid=94 Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119). Macquarie University E-Learning Centre of Excellence: Sydney, NSW. Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities. D-Lib Magazine, 13(5/6). Retrieved September 27, 2007, from http://www.dlib.org/dlib/may07/henty/05henty.html Margaryan, A., Milligan, C. & Douglas, P. (2007). Community dimensions of learning object repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object repositories. Retrieved September 10, 2007, from http://academy.gcal.ac.uk/cdlor/documents/CD-LOR_Structured_Guidelines_v1p0.pdf 48 Principles related to addressing the culture around sharing in higher education 2.2.1. A strategic plan for establishing and maintaining institutional partnerships for sharing between institutions and the Carrick Exchange should be developed. The Carrick Exchange should implement institutional partnerships with Australian universities binding both in a relationship of shared responsibility. This should assist in the recognition and management of issues around sharing, engagement of the sector, rights management and resource maintenance. Universities should be encouraged to understand their mutual role supporting the Carrick Exchange and being supported by it. There should be a commitment from universities that a proportion of teaching and learning resources developed using university funds will be made available through the Carrick Exchange. As one interviewee said about the Carrick Institute: The scholarship of teaching brings academics together. The Carrick Institute has brought the concept of learning and teaching more status since its inception in 2004, so the Carrick Institute should use its credibility, status and power in the sector to initiate change in the culture. (Interviewee) 2.2.2. The Carrick Exchange should systemically address the culture surrounding sharing in higher education. A key finding of the interviews was that to engage the sector, effort and resources should be put towards managing cultural and pedagogical change around the issues of sharing. Those already familiar with the management of repositories said that addressing the culture of universities was more time consuming than training people in the use of the system. Sharing learning and teaching resources is not common practice within higher education and competition amongst academics and institutions is a perceived barrier to widespread sharing. However, there is a culture of sharing for the greater good that some interviewees acknowledged should be promoted to individuals and their institutions. There is also evidence of sharing in some of the science-based disciplines (engineering, chemistry, astronomy) and quite often in management. Those more cautious in the sector will align themselves with this key practitioner: People in an academic environment are so used to thinking around the idea of IP and are so nervous about how polished their material is and whether it should go out if it’s not polished. So there needs to be a staged process in the movement to a sharing culture. (Interviewee) However, others may see the following correlations with scholarly publication: If like journal articles, teaching resources are developed and given away through ‘repository sharing’, and assessed via peer review, this will reflect positively on the individual for tenure and promotion, the quality of the university and the quality of teaching at that university. (Interviewee) The growth of institutional repositories (e.g. Deakin, RMIT, Monash and Griffith), and the development of scholarly research repositories to meet Research Quality Framework (RQF) requirements will raise awareness about sharing within the sector. Key to changing the culture will be strategies that focus on promoting the benefits of sharing and clarifying the rewards. The Carrick Institute carries prestige and has been urged by interviewees to use its place in leading teaching and learning to effect cultural change. Change management strategies must be 49 adopted progressively and consistently. “We need to align technology with existing practice in order to facilitate this change and not hope that the technology will change existing practise” (Bates, Loddington, Manuel & Oppenheim, 2006, p.25). While there is a foundation of practice in sharing resources across institutions via discipline based repositories, some institutions will see sharing courseware which could be reused by competitors as problematic. These same institutions may be more likely, however, to share generic resources that do not raise the same concerns. There is also a skill set around sharing that must be acquired before resources can be effectively reused (Littlejohn, 2003). This includes skills in locating and contextualising relevant resources, and designing resources for reuse. Skill development and pedagogical support should be addressed at the institutional level. 2.2.3. Networks and partnerships should be nurtured as a means of fostering contribution to the Carrick Exchange. Effort should be directed towards proactively facilitating the development and continuity of networks and communities of practice engaged in sharing and developing high quality resources around the Carrick Exchange. Networks developed elsewhere may not at first see advantages in relocating to the Carrick Exchange. Professional development and support is likely to be required to assist the higher education community in making best use of the Carrick Exchange for individual and specific group contexts and purposes. Strategies to address culture change Strategies for the Carrick Institute and the Carrick Exchange 1. Develop communicate the benefits of the Carrick Exchange to institutions and the sector, and to support new pedagogical processes and relevant strategies. Follow the LORN proactive methods of engagement (see Exemplars). 2. Clearly communicate information about availability, stages of development and future plans of the Carrick Exchange and its sharing and reuse polices with key stakeholders and interested parties . 3. This approach including resource development support, is regarded as a successful formula for implementation (Nicol et al., 2004 in Weedon, Bricheno & Chidwick, 2004). 4. Begin with the notion of resource management and/or commentary on resources, then move to the newer notion of sharing (RMIT and Loughborough Universities’ models). 5. Provide and support training in the Carrick Exchange with respect to: • • • Technical capability; The use and operation of the repository; and Development of teaching and learning expertise and practices to integrate the interactive and innovative resources available through the repository. 6. Target strategies to address cultural and pedagogical change for sharing, across the diverse range of members, by considering the various roles, groups and levels of awareness: • • Roles – e.g. academics, educational and staff developers, librarians, senior and middle managers, sessional staff, research officers, support staff, staff new to teaching; Groups – communities, communities of practice, networks, disciplines, organisations, and professional associations; and 50 • Levels of awareness – aware, curious, envisioning, trying out, using (Dormant, 1997) (see Table 2 in Engagement guiding principles and recommendations section of this report). Strategies for the higher education sector 1. Implement consistent national policies across institutions to support equitable contribution of resources to the Carrick Exchange. 2. Support cultural change from the top providing leadership and credibility, as well as from the bottom up, supporting those willing to take risks, and providing resource development support. 3. Support activities which promote mentoring and skills development in reuse of digital resources. 4. Support new and beginning teachers, associate lecturers and demonstrators, who may be more inclined to reuse resources that they have not created. Exemplars Learning Object Repository Network project (LORN): Flexible Learning Coordinators within the VET sector act as champions for their state jurisdiction, encouraging and supporting contribution and use of the repositories. Project representatives work actively to integrate and network the repositories across four states of Australia: DET NSW, the NSW Centre for Learning Innovation (CLI), TAFE NSW, TAFE South Australia, TAFE Victoria, TAFE Tasmania and the Flexible Learning Toolboxes Project. LORN’s mission is to link repositories with teachers. There is also a technical link across the repositories via the LORN search facility. MIT Open Courseware (OCW): Open courseware initiative began in 2001. Anyone can download materials (students or teachers). 1700 courses shared. http://ocw.mit.edu/OcwWeb/web/home/home/index.htm RUBRIC: A project for smaller and regional universities. The Toolkit provides a structured framework for partner institutions to evaluate, trial and implement an Institutional Repository solution. http://www.rubric.edu.au/ The University of Ballarat (TAFE): Example of an institution which places all resources developed within the institution on the intranet for all teaching staff to use. VET Learning Object Repository Network (LORN): Proactive work by coordinators to promote a culture of sharing by seeking out teachers in each state and helping them engage with the VET repositories. http://pre2005.flexiblelearning.net.au/projects/resources/2005/vlorn/ Key References Cohere Group. (2002) The learning object economy: Implications for developing faculty expertise. Canadian Journal of Learning Technology. 28(3), 121-134. Ehrmann, S., Gilbert, S. & McMartin, F. (2007). Factors affecting the adoption of facultydeveloped academic software: A study of five icampus projects. Retrieved September 19, 2007, from http://www.tltgroup.org/iCampus/iCampus_Assessment_Full.pdf 51 Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119). Macquarie University E-Learning Centre of Excellence: Sydney, NSW. Kenny, J. (2003). A research-based model for managing strategic educational change and innovation projects. Annual Conference Proceedings of HERDSA (the Higher Education Research and Development Society of Australasia), University of Canterbury, Christchurch, NZ, 6–9 July 2003, Retrieved September 27, 2007, from http://surveys.canterbury.ac.nz/herdsa03/pdfsref/Y1102.pdf Margaryan, A., Milligan, C., Douglas, P., Littlejohn, A. & Nicol, D. (2007). Community dimensions of learning object repositories, CD-LOR deliverable 10: Recommendations to JISC for future research and development. Retrieved September 27, 2007, from http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Final_Recommendations_v1p0.pdf Stuckey, B. & Arkell, R. (2006). Development of an e-learning knowledge sharing model: 2005 Knowledge Sharing Services Project. Australian Flexible Learning Framework project report. Funded by Australian National Training Authority (ANTA) and DEST. Principles related to rewards, incentives and recognition for contribution 2.3.1. The Carrick Exchange should establish a system of rewards, incentives and recognition that is valued, appropriate and manageable, and aligned to institutional policies and practices. If user needs are addressed, there is an inherent incentive to contribute to the repository. Additional incentives need to be at both the institutional and national levels to be worthwhile. Rewards and recognition from the Carrick Institute should be linked to institutional promotional systems and acknowledged. The Carrick Institute Awards provide a good model which is influencing internal institutional promotional systems. Provision of expertise on the Carrick Exchange by individuals should be recognised, and the resulting collaborations and connections rewarded in some way. Evidence of use and reuse will be important and mechanisms which support this should be automated where possible. Tracking the number of accesses and downloads has relevance but is limited in that the information does not describe how the resources are being used once they have left the repository, or e.g. how useful engagement in shared commentary was, in informing and changing teaching praxis. The following are possible rewards and incentives for various forms of contribution to the Carrick Exchange. The findings from the research indicate that these were probably of most importance: • • • • • • Recognition from the home institution for contributions to the Carrick Exchange by providing time and/or funds, in the same way that time and/or funds are allocated for scholarly research. Reports about resources submitted to the Carrick Exchange that could be used for promotion and tenure applications. Acknowledgement of the resource creator each time a resource is reused. Grants to develop and share resources to the Carrick Exchange. Awards to formally recognise submissions made to the Carrick Exchange. Acknowledgement for contributions in various forums such as teaching and learning conferences and forums, and in online newsletters. 52 • Funds for time release or project development are probably considered preferable to many staff as a reward, rather than direct salary supplementation. Financial remuneration of this kind is considered a low priority. This is probably because many staff are engaged in related activity as part of their brief regarding teaching, but want some recognition and reward for that effort, not just more work. Other suggested incentives and rewards: • • • A trusted system of rights management and IP protection for individuals and their institutions for shared resources, covering deposit and distribution. A system that is equally open for consumers and contributors. Staff are unlikely to participate in a system that requires contribution as a prerequisite before resources, forums and tools can be accessed. The Carrick Exchange could act as a broker between individual academics and commercial customers, but generally the ascilite research indicates that selling resources via the Carrick Exchange is a low priority. 2.3.2. The Carrick Institute should ensure that the established system of rewards, incentives and recognition is adequately resourced and supported into the future. To be effective, the reward and recognition system should be maintained through stable funding systems. The funding model for rewards should be ongoing and not project based. Incentives, rewards and recognition for the effort put towards contributing in any way to the Carrick Exchange need to be strongly communicated and promoted to the sector. Some of the exemplars below provide models of awards and award systems that could be adopted. 2.3.3. A strategy for impact and process evaluation should be developed to assess the relevance and value to the sector of the rewards, incentives and the system of recognition. As one of the objectives of the Carrick Institute is to ‘promote and support strategic change in higher education institutions for the enhancement of learning and teaching’ 11 it will be important to measure, across the sector, the impact of rewards and incentives developed to recognise contributions to the Carrick Exchange. The value and impact of these rewards and incentives for the user, institutions and the sector will need to be assessed. Rewards for scholarly research output are well established, but the same is not true for teaching and learning output. A considered change management strategy is required to address this anomaly between scholarly research and teaching and learning products. Strategies for valuing, rewarding and recognising contribution Strategies for the Carrick Institute and the Carrick Exchange 1. Initiate policies from the Carrick Institute that contribute to changes in university policy so as to link contribution to the Carrick Exchange to promotion and career advancement. 2. Recognise contribution to the Carrick Exchange as a mechanism to support applications for Carrick and institutional awards. 11 See ‘About the Carrick Institute’: http://www.carrickinstitute.edu.au/carrick/go/home 53 3. Establish awards from the Carrick Institute for excellence in resource development, contribution and sharing. 4. Create awards for resource and commentary contribution such as ‘Resource of the Month’. Criteria for this could be based on ratings (comments and star ratings), and the number of times a resource is accessed. A more prestigious award could be best (formally) peer reviewed item. 5. Raise awareness amongst university staff involved in promotions and appointments, through a communication campaign about the Carrick Exchange and what contributing to it means. 6. Preserve original attribution for the creator of a resource through licensing agreements. 7. Conduct research and evaluation to track reuse. This could be implemented through targeted Carrick Institute grants, for example. Statistics should be compiled about how the resources are being used, preferably backed up by periodic qualitative evaluation. This record allows for follow up so users can be contacted on how they have used a downloaded resource etc. and provides a measurable record for use in promotion, awards etc. Strategies for the higher education sector 1. Certificates noting contribution to the Carrick Exchange and the extent of re-use of resources should receive recognition and contribute towards teaching portfolios for promotion and tenure. Technical strategies 1. Design and implement automated systems within the Carrick Exchange that provide acknowledgements of the number of resource contributions (including commentary) per member, along with access details of submitted resources. The system should automatically track resource contribution, downloads and commentary. 2. Where a contributed resource has been reused and repurposed, the original contributor could receive acknowledgement for their provision of the initial resource. Perhaps they could be sent a link to the repurposed resource. 3. Provide access for individuals to be able to retrospectively review commentary they have initiated and responded to over the long term. 4. Periodically conduct polls asking members about resources they have used or reused. Exemplars ascilite awards: The awards recognise the exemplary use of technologies in teaching and learning in tertiary education. Awards are open to current financial members of this professional organisation. http://www.ascilite.org.au/index.php?option=com_content&task=view&id=20&Itemid=36 BetterPhoto monthly competition: BetterPhoto can be regarded as a hobby related learning community which includes both amateurs and professionals. The community aims to support sharing of photos, skill sharing and development. Networking opportunities and courses are available. Small prizes are offered as incentives but recognition and personal sales from professional sites are the key incentives to contribute. http://www.betterphoto.com/contest.asp Carrick Institute Awards: Awards for teaching and learning in higher education. The model could be extended to the Carrick Exchange. http://www.carrickinstitute.edu.au/carrick/go/home/awards Carrick Institute Teaching Quality Indicators Project: Research into rewarding and recognising quality teaching. Outcomes from the project have relevance to the Carrick Exchange. http://www.carrickinstitute.edu.au/carrick/go/home/pid/370 54 CLOE: Awards for contribution and reuse. http://cloe.on.ca/awards.html MERLOT: ‘Jump start CV’. MERLOT members can export a portfolio of their profile to use for CV and promotions. http://www.merlot.org/merlot/members.htm MERLOT also offers the following awards: • • • • Exemplary Online Materials Award (peer reviewed); Volunteer of the Year Award (contribution to the peer review Editorial Boards); Distinguished Service Award (individual who exemplifies the vision that brings MERLOT members together creating the MERLOT Community and its services); and Application of MERLOT awards. See http://taste.merlot.org/awardsoverview.html University of Queensland: Investigation into bringing parity between scholarly research and effort towards teaching. http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesRe portApr07.doc Key references Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project, Academic survey: Final Report. Loughborough University & JISC. Retrieved September 27, 2007, from http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/11374231 50_SurveyReport.pdf Mckenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, adoption and adaptation of project innovations in higher education. A report for the Carrick Institute for Learning and Teaching in Higher Education. Sydney: University of Technology. University of Queensland. (2007). Report of the Working Party on the Diversity of Academic Roles. Internal review. Retrieved September 27, 2007, from http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesRe portApr07.doc Principles related to Intellectual Property and Digital Rights Management 2.4.1. The Carrick Exchange should establish IP and Rights Management policies and practices that align to institutional policies and practices. The whole sector must become much more aware of the policies and procedures that are in place regarding resource ownership and sharing. There is . . . a sense of proprietary ownership of intellectual property . . . Generally academics are keen to look at the resources of others but more reluctant when it comes to sharing their own work (Interviewee). 55 Academics are frequently unclear about their rights over resources created for teaching purposes. In many institutions (e.g. RMIT), the institution holds copyright on material produced by staff, but staff retain the moral rights. The Carrick Institute has a key role in aligning these policies and practices across the sector and in providing leadership for this alignment through the Carrick Exchange. The Carrick Exchange could promote the approach taken by some institutions (e.g. Loughborough University) that the university owns the copyright but allows its employees the right to share resources created during their employment. 2.4.2. IP and Rights Management should address the concerns and needs of contributors Some academics fear that credit for creating a resource will be “inaccurately or unevenly” given away. Limited understanding of copyright issues can lead to concerns over misuse of reused materials. The practice of sharing will be affected users levels of trust in the system and the management of intellectual property rights. The concerns of university administrators should also be acknowledged as one interviewee commented: There will be resources and intellectual property that will require specific sign off from the University before broad sharing can be enabled because of the compromise to the ‘competitive advantage’ that sharing may evoke. (Interviewee) However, one practitioner interviewed from the VET sector acknowledged that teachers are growing more circumspect about issues of IP and sharing, and teachers in that sector are learning that resources become obsolete very quickly: For teachers to cope with this they are realising that they have got to collaborate and share resources [and therefore find solutions to issues such as IP]. (Interviewee) 2.4.3. The Carrick Exchange should develop systems and structures to ensure that IP and Rights Management policies and practices are implemented and appropriate. The Carrick Exchange should draw from lessons learned in the international higher education sector where issues of IP and rights management have been investigated and strategies adopted for use in other repository systems, such MERLOT, CLOE, the Rights and Rewards project (Bates et al, 2006), and draw on the findings of the OAK Law report. The greater use of open source software in the future may also bring changes to approaches to licensing systems that should be monitored. Strategies to address intellectual property and rights management Strategies for the Carrick Institute and the Carrick Exchange 1. Clearly communicate to institutions the policies and protocols for the Carrick Exchange, being explicit about the need for clarity on institutional positions on IP, copyright, moral rights and academic exchange and sharing. 2. Devise a communication plan that will reach all staff within the sector, new and continuing, about the IP, copyright and moral rights that pertain to individuals and institutions regarding ownership and sharing of learning and teaching resources. 56 3. Develop and disseminate policies for access and rights, management of resources, and intellectual property (IP) for the Carrick Exchange. 4. Use the forums of the Carrick Exchange to discuss rights management and ensure the systems put in place are those that suit the sector best and reflect changing technologies and contexts of use. (Following LORN project strategy 12.) 5. Implement the Creative Commons licensing system as a simple and fair method. 6. Have a clear statement of copyright as it affects reuse (distribution) within the licensing agreement. Loughborough University research (Bates et al, 2006) suggests that this clarity is essential otherwise users are unclear how materials can be legally and fairly modified within the guidelines. Strategies for the higher education sector 1. A possible framework for sharing is to adopt a model where ownership of resources created by academics and support staff in the course of their employment is claimed by the home institution; however, the institution grants its employees the right to share that material. 2. Have designated staff such as librarians to support and possibly manage the copyright and intellectual property requirements for resources submitted to the Carrick Exchange (e.g. Jorum staff help with metadata input). Technical strategies 1. Provide templates for copyright and IP that can be used when submitting resources to make the process uncomplicated. 2. Provide the means to clearly brand resources shared through the Carrick Exchange with the name of the originating institution as well as the contributor/author. Many journals automatically insert a title page (e.g. Routledge) but this will vary according to the type of resource. Exemplars CLOE and Creative Commons: CLOE recommends the most recent Canadian version of the Creative Commons Attribution-NonCommercial-ShareAlike (BY-NC-SA) licence, as the default copyright licence for the CLOE repository. http://cloe.on.ca/creative_commons.html Loughborough University Repository: Manages licensing and rights for sharing by claiming copyright on resources created by its employees, but giving them the right to share these. http://rightsandrewards.lboro.ac.uk/index.php?section=1 MITOpencourseWare (OCW): Initiative uses Creative Commons licensing. http://ocw.mit.edu/OcwWeb/web/home/home/index.htm OAK Law Project: Aims to develop legal protocols for managing copyright issues within an open access framework. Its focus is research repositories, but its policies have implications for the Carrick Exchange project (as noted in the digital rights Discussion Paper, commissioned by Education.Au by Mason, Macnamara and Galatis, 2007). http://www.oaklaw.qut.edu.au/ RMIT and Loughborough University: Both universities are developing workflow models regarding contribution and other aspects of their repositories which could be of interest to the Carrick Exchange. 12 Interviewee communication. 57 RUBRIC project metadata: Support for repository managers, http://www.rubric.edu.au/RUBRIC_Toolkit/docs/Metadata.htm Second Life (SL) digital rights policy example alternative: A multi-user 3D online world created by its residents. Members move around the virtual world using an avatar and interact with other members much as people do in real life. Communities self establish around social interests, education and business etc. Universities and other educational institutions have begun, e.g. holding classes in SL and staffing virtual libraries with real world volunteers. Many blogs and places for commentary exist not only inside the virtual world but outside as well. The owners of SL act as service providers and enforce few controls over residents. It is generally selfregulating within the set terms of service. However, due to pressure from community members and commentators, for example, gambling was banned. Residents retain the right to full intellectual property protection for any digital content they create in Second Life, including avatar characters, scripts, objects, designs etc. Residents however must grant the owners of SL, Linden Lab a license to, amongst other conditions, use, reproduce and distribute their content in any media for marketing and/or promotional purposes. http://secondlife.com/ Key References Barton, M. & Waters, M. (2004). Creating an institutional repository, LEarning About Digital Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 27, 2007, from http://www.dspace.org//implement/leadirs.pdf Pappalardo, K., Fitzgerald, A., Fitzgerald, B., Kiel-Chisholm, S., O'Brien, D. & Auston, A. (2007). A guide to developing open access through your digital repository. DEST funded Open Access to Knowledge Law Project OAK Law Project. Retrieved September 27, 2007, http://eprints.qut.edu.au/archive/00009671/01/9671.pdf Mason, J. Macnamara, D. & Galatis, P. (2007). An investigation into digital rights management issues in the Australian higher education sector: A discussion paper. Commissioned by Education.Au. Internal report. Principles related to management of resource contribution 2.5.1. The Carrick Exchange should develop clear policies, guidelines and procedures for resource contribution. A national approach regarding the submission policy for the Carrick Exchange should be implemented. There should be a stratified system of classification, with peer reviewed contributions at the top level of approved or endorsed content, and allowing for other categories of submission that are not formally peer reviewed or may be under review. Clarity about the status of resources will help selection of items and commentary by users. As peer review may be a disincentive for some to contribute to the system, a tiered system of quality assurance and review should be implemented (see Peer Review and Commentary section of this report). The system should allow for the relative merits of a resource to be assessed by the user before an item is downloaded. Clear labeling will assist in this. Contribution policies should include guidelines and criteria: 58 • • • Encouraging productive feedback to authors, and commentary around problematic issues of teaching and learning; Ensuring the quality of submitted resources meets the expectations of the sector. This includes systems to ensure currency of resources through auditing and archiving procedures. Providing useful secondary metadata, such as details of the context of use, changes required when adopted in other disciplines. 2.5.2. The Strategic Plan should include guidance on resource management, archiving, sustainability, and support and encouragement for resource contribution. The Strategic Plan should define what will be included in the Carrick Exchange collection, including the more formally contributed digital materials and the informally contributed commentary. Some resources will need to be preserved well into the future; others will be more ephemeral. Currency may diminish with some resources, and these should be removed. However, there may still be a need to archive a selection or resources for perpetuity. Information specialists should be consulted on how this should best be managed. The Carrick Exchange should determine whether all commentary will be stored indefinitely or for a defined period. Auditing, archiving and version control will need to be managed through quality assurance mechanisms (see Peer Review and Commentary section of this report). Contribution tools and templates should be simple, accessible and intuitive. So as not to become a barrier to contribution, user defined metadata must be easy to assign and supported by as much system assigned metadata as possible. Some focus groups suggested that resources that are examples of ‘poor’ teaching practice or poor technology use are of value in certain contexts, just as high quality resources are valuable; but each resource needs appropriate tagging to help its discovery for multiple purposes and users. User defined metadata and tagging is often poorly and inconsistently applied. Many Web 2.0 technologies, however, are predicated on this folksonomy approach. This affects search and retrieval of resources and should be factored into user engagement and contribution strategies for particular content types. Each discipline will have different views about what is appropriate, what taxonomy best suits retrieval of resources, the benefits of contributing teaching materials versus the merits of conversations about learning and teaching etc. The relative value of various content types, the granularity of resources added, their ability to be customised, and ease of use will all be assessed differently by different individuals and disciplines. 2.5.3. An evaluation plan should be developed to assess and report on resource contribution to the Carrick Exchange. Evaluations should be used as a basis for identifying funding needs for the maintenance and further development of the Carrick Exchange and as justification for ongoing investment in the Carrick Exchange. Evaluation of the evolving networks and communities, the value of commentary and contribution should be investigated. Impact evaluation for example, could assess the use of the Carrick Exchange as: • • • • an appropriate workspace for team collaboration and facilitator of group processes; an impetus for increased effectiveness in the use of information and communication resources; a promoter of communities of practice; a space for facilitating the sharing of teaching and learning materials; and 59 • an enabler of greater reflection on pedagogy within higher education. Strategies for managing resource contribution Strategies for the Carrick Institute and the Carrick Exchange 1. Provide long-term, stable funding and not rely solely on short-term grant funding. However, additional funding options should be explored, such as advertising, sponsorship and income generation through innovative project work and staff development initiatives. The Carrick Exchange could also act as a broker in the future between individual academics and commercial customers. 2. Through the Strategic Plan, outline the funding models for supporting resource development for the Carrick Exchange within institutions, e.g. grants for Carrick Exchange resource development, commissioned resources required by the sector. Also especially provide assistance for resource contribution to institutions whose staff contribute resources to the Carrick Exchange. 3. Within a strategic plan that outlines a phased roll out approach beyond the initial implementation, devise activities across institutions that bring different stakeholder groups including academics and support staff together, sharing resources and expertise. As the novices, i.e. users new to the Carrick Exchange, develop expertise and skills, they in turn become mentors to others within their own institutions, and in cross-institutional projects. 4. Promote the contribution benefits of the Carrick Exchange that are pertinent to the potential contributor, e.g. The Carrick Exchange could be promoted to some academics as a space where they can store and develop ‘works in progress’ used in collaborative projects, and invite others (probably a small selected network of colleagues) to give feedback to assist in further development. Tools such as wikis may aid in this type of collaborative contribution. 5. Model the contribution policy documents for the Carrick Exchange, on those found on the Edna and MERLOT sites, with input from the Jorum guidelines. (Note: Jorum operates on an institutional model of contribution, whereas CLOE and MERLOT offer open individual membership.) 6. Develop an evaluation plan to track evolution of the system and its efficacy over time. Technical strategies 1. Adopt a combined approach to metadata contribution, to meet the needs of different users of the system. Metadata options should include standardised subject headings that users can select from and assign to the shared resource, plus a keyword search function and a browse option. Resource contributors could assign metadata about, e.g. what they think the item is and what it could be used for; other members could add metadata about how useful they found the resource or contribution; and peer reviewers (for some contributions) could add tags around formal review categories. 2. Allow contributors to have the option to elect whether their resource should be peer reviewed, as formal peer review may be seen as a barrier to contribution for some academics. 3. Provide a dual track publishing system where all resources can be published immediately but informally through the forums, at the same time as a resource may be undergoing quality assurance and/or peer review. This allows for immediate publication (‘use as is’ but ‘user beware’), but also allowing time for quality and peer review processes. (see Peer Review and Commentary section of this report). 4. Contribution tools and templates should be simple, accessible and intuitive. 5. Establish mechanisms and systems to enable version control and unique identifiers to manage and promote reuse and repurposing of content and the ability to improve materials that have been contributed to the Carrick Exchange. 6. Resource management at the personal level will be enhanced by tools on the Carrick Exchange which mirror systems already in use on sites such as Amazon.com or Librarything.com, where users can create and manage ‘baskets’ or personal lists that store links to resources of interest tagged by the user, and to create lists of other members with similar interests. 60 7. Consult information specialists about an appropriate preservation and archiving policy for short-term and long-term purposes. Exemplars CLOE archiving policy: Contributors agree to share a resource for at least two years. The resource is then reviewed for continuing inclusion in the collection after a five year period. If the resourcee has not been accessed for two years it is archived. http://cloe.on.ca/ VET Learning Object Repository Network project (LORN): Working on version control and reuse. http://lorn.flexiblelearning.net.au/Home.aspx Contribution policies for digital resources – see the following models and the policy recommendations at the end of this section: Edna: http://www.edna.edu.au/edna/go/about/policies MERLOT: http://taste.MERLOT.org/policies.html Jorum: http://www.jorum.ac.uk/contributors/chelp/guidelines.html The Jorum Workflow Report provides some clarity on processes, although it is based on resource contribution that is initially institutionally based rather than individually based, and there is no formal peer review process (only informal rating and commentary). Also Jorum resources are published before cataloguing. The process is: Contribute, Publish, Catalogue, Review. See Jorum Publications/Jorum R&D reports currently available: http://www.jorum.ac.uk/publications/ Intute guidelines and the Desire Information Gateways Handbook (section 2): Relevant for ideas about formulating policy. Intute Collection Development Framework and Policy http://www.intute.ac.uk/policy.html Desire Information Gateways Handbook http//www.desire.org/handbook/2-1.html MITOpenCouseWare (OCW): Business model - open sharing of resources funded by Massachusetts Institute of Technology and donations from around the world. https://giving.mit.edu/givenow/ocw/MakeGift.dyn Contribution policies for commentary resources – see the following models and the policy recommendations at the end of this section: MERLOT acceptable use policy: http://taste.MERLOT.org/acceptableuserpolicy.html Edna content standards: http://www.edna.edu.au/edna/go/about/policies/pid/119 Yahoo data storage policy: http://info.yahoo.com/privacy/us/yahoo/datastorage/details.html 61 Key References Barton, M & Waters, M. (2004). Creating an institutional repository, LEarning About Digital Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 29, 2007, from http://www.dspace.org//implement/leadirs.pdf Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities. D-Lib Magazine, 13(5/6). Retrieved May 16, 2007, from http://www.dlib.org/dlib/may07/henty/05henty.html Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning. London and Sterling: Kogan Page. McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and implementation. Journal of Digital Information, 5(3), Retrieved September 21, 2007, from http://jodi.tamu.edu/?vol=5&iss=3 Recommendations for resource identification and contribution policies Policies will need to be developed in the following areas. Note: Barton and Waters (2004), provide sound advice in their LEADIRS Workbook, Creating an Institutional Repository. Resource identification policies 1. Collections policy a. State how the Carrick Exchange differentiates itself from other collections, databases, portals and communities. b. Identify who manages and promotes the collections. c. Specify the selection criteria and methods used to ensure quality, e.g. quality of content, usability (ease of use, consistency, interoperability), educational effectiveness. 2. Classification of resources policy a. Specify what resource types are collected by the Carrick Exchange e.g. Carrick grant project resources; learning, curriculum and teacher resources; technological resources; peer reviews and social resources; policy resources; and archived resources. b. Identify the process used for different classifications: resource under development or completed, resource for/not for quality assurance, for/not for formal peer review, for/not for informal peer review. c. Carrick award resources; highly rated resources; resources by individuals; resources by institution. d. Methods for version control. e. Metadata management. f. Licensing system. (Suggest beginning with Creative Commons.) 3. Registration policy a. Guidelines or steps for registration and contribution (objects and commentary), including templates to assist resource contribution and getting started. 62 4. Policies for archiving resources: a. Policy on data storage; b. Checks every 3- 5 years if authors wish their resource retained on the Carrick (issues of currency and relevance); c. Carrick Exchange selective archiving for the long-term – some resources should be archived for historical purposes or for reference even when apparently past their perceived currency. d. Annual audit of resources based on downloads and access records. 5. Policies to manage resource identification and contribution through: a. b. discipline-based initiatives or Editorial Boards; and annual conferences or workshops where resources are generated or reviewed. Resource contribution policies 1. Resource contribution guidelines and policies: a. Determine who can contribute, what resources, when, for how long (2-3 years?), quality assurance and peer review procedures. b. Workflow charts to show how resources are ‘triaged’, according to classification: Category 1: For full, formal peer review and quality assurance before submission to the public repository, following a system of “triage”. Concurrently made available (marked as “under peer review”) for additional commentary by members of the Carrick Exchange (dual track publication). Category 2: Not for peer review, quality assurance only – but concurrently made available for additional commentary by members of Carrick Exchange. Marked as ‘under quality assurance testing’. Category 3: Non-reviewed resources – items submitted “as is”, i.e. with no quality assurance, informal commentary or formal peer review attached, contributed at owners’ own risk - subject to commentary by members of Carrick Exchange and marked as such. 2. Policy regarding Carrick Institute grant holders: Mandatory contribution of project development reports, project outcomes and deliverables to the Carrick Exchange as a condition of receipt of a grant. 3. Metadata policy: This should ensure that metadata is easy to apply, complete and maintained, and aids retrieval. Should address: a. Classification of each resource and b. Secondary metadata about use of the resource, context of use where possible. 4. Rights management/licensing policy and guidelines should specify: a. For individuals – how their IP and moral rights are protected, and the implications for sharing. b. For institutions – the implications of sharing and contributing resources to the Carrick Exchange by staff who created the resources in the course of their employment. The policy and guidelines should address issues of competition and ownership. c. Policy type 63 o o Creative Commons or standard copyright; Indicate if any likelihood of e-commerce or brokering in the future. d. Three specific Guidelines (derived from CLOE): o o o o Statement of Copyright (including contact details of copyright holder) Terms of Use (note that ‘fair use’ is insufficient for terminology) Modification Rights (grant, or contact details to request permission) Should cover deposit rights and distribution rights. 5. Management of shared workspaces, rights and responsibilities policies and guidelines. 6. Rewards and incentives: a. Carrick Exchange Memorandum of Understanding with all Australian universities, recognising the value of Carrick Exchange contributions and the role they play in the core business of teaching and learning within higher education, and their importance for promotion and career advancement. This should facilitate alignment of institutional policy with Carrick Institute aims and vision to transform teaching and learning. b. Within the Strategic Plan outline the methods for rewarding individuals and institutions who contribute and use the Carrick Exchange such as: o o o Awards for voluntary work, resource contribution excellence, contribution to the peer review process, contribution to the community etc. (see Strategies 3.3.1, 3.3.2 and 3.3.3); Downloadable certificates and/or e-portfolio providing evidence of contributions and reuse that can be used for promotion and tenure; Acknowledgement for authors as the original creator of a resource. 64 Peer review and commentary guiding principles and recommendations ______________________________________________ Introduction As part of the development of the Carrick Exchange ascilite was engaged to conduct research into: • What Peer Review and Commentary protocols and mechanisms can be derived from the higher education community? o What new and current methods exist for peer review and commentary of resources that can be adapted for use for the reviewing of teaching and learning resources? − − − − o o What gets peer reviewed? What methodologies apply to different types of resources? What process for peer review is used? Who are the reviewers? What recommended policies and procedures are currently in use? What are the policy implications for the Carrick Exchange? The following protocols and mechanisms for peer review and commentary for resource contribution to the Carrick Exchange have been synthesised from knowledge and expertise existing in: the literature, current practice, exemplary community development to foster engagement with online services, and key practitioners and networks in both national and international settings. Peer review for this research refers to the evaluation of teaching resources and commentary contributed to the Carrick Exchange. Review is conducted by equals, those with qualifications and standing in the higher education sector who are capable of assessing the worth and value of ICT-based teaching resources (artifacts and discussions). Peer review may be formal and conducted by teams of experts, it may be limited and more of a quality assurance process, or it may be informal and include discussion, feedback and comments made on resources and ideas contributed to the Carrick Exchange. Peer review as quality assurance may include assessment of the currency, educational design and construction of resources; compliance with copyright, intellectual property and digital rights management policies; and technical accuracy and reliability. Formal peer review, a lengthier and more demanding review process, might replicate the scholarly peer review process which leads to publication in the higher education sector. Peer review may also be an informal process whereby members of the community voluntarily respond to others’ contributed resources, or resources stored elsewhere but linked to via the Carrick Exchange (known as “commentary” in this research). These resources could be finalised products which the authors publish and share, or resources under development. This informal sharing of ideas could be an important element of the Carrick Exchange and the basis for various communities of practice. Informal commentary can be seen as another useful mechanism for peer review and academic recognition. It can be recognised as a support mechanism for peers around best practice in learning and teaching and essential to the development of communities of practice. It can also be a mechanism by which individuals show leadership in the field. Peer review can be conducted against specified criteria, and/or as a means of giving and receiving structured feedback to improve teaching resources. Resources identified as formally 65 peer reviewed are likely to be perceived as high quality examples of best practice. The implementation of rigorous peer review processes is key to maintaining high standards. As one focus group participant stated: Rigorous peer review is important for the outcomes and status of the review process. Having peer review processes in place supports the perception of quality resources and communication of best practice. (Focus group participant) The findings of the ascilite research indicate that a formal peer review system for the Carrick Exchange is required. It is recommended that the system chosen mirror the current system of scholarly research review. Peer review of learning and teaching resources does not currently have the same status as peer review of scholarly research. This is a key issue that should be addressed to recompense staff for the effort and resources put towards this activity and raise the status of scholarly review of educational resources. For formal peer review, the traditional system of “blind” peer review where reviewers are not identified versus a system where reviewers are known was discussed in the interviews and focus groups. Some participants favoured the latter system because of the accountability demanded; others favoured the traditionally anonymous system because it is intended to assist with objective and democratic evaluation. Others suggested that where an author was able to engage in a dialogue with the reviewers this would be beneficial for developing a full understanding of the design dimensions and intended and actual learning outcomes. Some in the reference groups pointed to the developing open peer review systems trialled by a number of scholarly journals such as Nature (see Literature Review section of this report). Open publication of a paper before it has been through the formal peer review process allows for speedier dissemination of innovation. In the case of Nature magazine, once a research article has passed an initial quality check, the author may post their paper on the journal’s website, and anyone, provided they supply their name and email address, can comment on the research; the traditional blind peer-review process continues in the background. This option has positive implications for submission of resources to the Carrick Exchange where it will often be important to expedite publication to maintain currency. However, although publication is hastened, the process of soliciting informal peer review comments, while received positively by journal readers, has had limited success and comments have not been substantive. This may be different in smaller communities that gather around the Carrick Exchange but in the more public areas it may follow the results of the Nature research (see Literature Review section of this report). Regarding informal peer review in this research, participants indicated that it should be an open process where opinions were identified with an author, not anonymous contributions. The credibility of the person contributing the comments was an important factor for members considering the worth and value of others’ commentary. Further, it was though that “named contributions” would promote scholarly, thoughtful review of resources and lessen the likelihood of work being insensitively treated or “rubbished”. As one reference group member said: Web 2.0 has sparked a growing number of "free" and "open" movements that challenge current publishing and peer review models — including the Free and 13 14 Open Source Software movements, the Open Access Movement , Open 15 16 Source Journalism , and Creative Commons ought to be considered in the 13 Software which the user can use for any purpose, study the source code of, adapt to their needs, and redistribute - modified or unmodified. 14 Immediate and unrestricted online access to digital scholarly material, primarily peer-reviewed research articles in journals. 15 commonly used to describe forms of innovative publishing of online journalism, rather than the sourcing of news stories by a professional journalist 16 The Creative Commons (CC) is a non-profit organization devoted to expanding the range of creative work available for others legally to build upon and share. The organization has released several copyright licenses known as Creative Commons licenses. These licenses, depending on the one chosen, restrict only certain rights (or none) of the work 66 new era of publishing as each contributes different perspectives and concerns on the emerging practices surrounding digital repositories and scholarly publishing. (Reference group member) Peer review and commentary were seen to have the following goals: • • • • • Support the delivery of high quality, reviewed resources; Enhance resources submitted to the Carrick Exchange which are “under development”; Foster professional development around the development and review of teaching and learning resources; Encourage collaboration and the development of networks and communities of practice; and Reward and recognise resource creators and reviewers. Recommendations in this section are structured around the following key themes that emerged from the ascilite research findings: 1. 2. 3. 4. Peer Review Goals Peer Review Processes and Procedures Informal Commentary Rewards and Incentives for Formal and Informal Peer Review Protocols and mechanisms related to the goals of peer review 3.1.1. Formal peer review policies and procedures of the Carrick Exchange should align with institutional policies and practices, and promote the scholarship of teaching and learning. The products of formal peer review (reports, ratings, documentation providing acknowledgement of peer review etc.) must have currency and credibility. To ensure this there should be alignment nationally with institutional recognition and reward processes and systems, providing incentives to both reviewers and contributors. The peer review scheme adopted should promote the scholarship and advancement of teaching for those who have resources reviewed and those who undertake those reviews. Recommended policies and procedures for peer review are provided below (see Strategies for Carrick Institute and Carrick Exchange). 3.1.2. The goals of formal peer review should be established and inform related policies and procedures. The purpose and expected outcomes of the formal peer review process should be established and outlined in the strategic plan. It is suggested that these goals might be: • • • to assure the quality of resources; formalise the process of peer review; provide a review process that is recognised as transparent, fair, valid and reliable; and 67 • foster recognition of contributors whose resources are peer reviewed, and reviewers who conduct the evaluations. The identified goals should then inform the development of related policies and procedures. 3.1.3. Peer review should enable judgements about the quality and usefulness of a resource. As a national repository the Carrick Exchange is in a position to implement a formal review process – a facility that national repositories and their communities may find too resource intensive to support. One interviewee suggested that the success of the Carrick Exchange was dependent on the standard and effectiveness of the peer review process. Peer review of educational resources is likely to be a more complex undertaking than review of, scholarly research output which is usually presented as a single medium, text document. The difficulty of assigning evaluation criteria, the context of use, and the variables that impact on expected learning outcomes, add to this complexity. The issue of assessing the quality of resources in Carrick is different to assessing the quality of journal articles because of the practical bent of the things that are in Carrick. (Interviewee) In preparing policies and procedures for the formal peer review process criteria should be established to address evaluation standards. MERLOT provides a useful set (see: http://taste.merlot.org/evaluationcriteria.html). CLOE also uses a similar standard. MERLOT uses three categories for its evaluation standards: 1. Quality of content a) Does the software present valid (correct) concepts, models and skills? b) Does the software present educationally significant concepts, models, and skills for the discipline? 2. Potential effectiveness as a teaching-learning tool (the most difficult to evaluate, especially where modules are taken out of context – assessing the purpose of the resource is very important – the review should be contextualised). 3. Ease of Use – particularly for first time users (teachers or students). Other criteria and standards against which the teaching resources can be evaluated can be found on the AUTC Learning Design Project website, the CLOE and MERLOT websites, and in documents by HEAL, the Health Education Assets Library (2006), the University of Queensland et al (2004), and Souza and Persily (2005). Other questions around the peer review process put forward by participants in the research include: • • Should feedback from learners be included in the review of resources where learning is the focus? Should the review process be fully online and integrated with the Carrick Exchange? Finding a balance between quality review (particularly in informal contexts) and democratic commentary open to all members of the Carrick Exchange will be a challenge. As one reference group member suggested: 68 There is a need to develop models and processes for peer review alongside issues of copyright, development of acceptable use policy and innovative use of technologies that model open source, participatory and interactive knowledge creation. (Reference group participant) 3.1.4. Resources contributed to the Carrick Exchange should be classified according to the level of peer review received. Formal peer review is not suited to all resources, therefore the Carrick Exchange should allow for a number of levels of peer review. To speed up sharing and publication procedures, resources placed in a peer review workflow should be made available to the community prior to and during the peer review process, and labeled as “under review”. This allows for a dual track system of publication and submission of resources (see Figure 3). Resources could be concurrently submitted for informal review to either all members of the Carrick Exchange or a restricted group of selected members, and at the same time submitted as a resource to receive formal peer review. This would be a choice each contributor could make at the time of submission of any resource, along with a nomination about the level of peer review that the resource should receive. To ensure that users of the system know the status of any resource retrieved through a search of the Carrick Exchange, the following classifications could be adopted: Category 1: For full, formal peer review and quality assurance before submission to the public repository, following a system of “triage”. Concurrently made available (marked as ‘under peer review’) for additional commentary by members of the Carrick Exchange (dual track publication). Category 2: Limited review resources, quality assurance only – but concurrently made available for additional commentary by members of Carrick Exchange. Marked as ‘under quality assurance testing’ (dual track publication). Category 3: Non-reviewed resources – items submitted “as is”, i.e. with no quality assurance, informal commentary or formal peer review attached, contributed at owners’ own risk - subject to commentary by members of Carrick Exchange and marked as such. Other sub-classifications could include: • • • • “Expert” reviewed resources; User (or general member) reviewed resources (receiving informal commentary as review); Carrick funded projects – which receive a certain amount of review through the project development process; and Status of completion – “completed” or “under development”. The limited review or quality assurance of resources could include an assessment of: • • • technical accuracy, reliability and interoperability; compliance with copyright, intellectual property and digital rights management policies; and the relevance, currency, and construction of resources. Formal review processes should follow the MERLOT criteria (see 3.1.3 above, and Figure 3). 69 3.1.5. The formal peer review process should be appropriately resourced and adequately rewarded The peer review process could be a resource intensive process, so only those resources which are of sufficient quality to warrant this assessment should be submitted for full peer review. The issue of adequate resourcing of the peer review process was a concern raised by research participants in both Cycle 1 interviews and Cycle 2 focus groups. Practitioners in both cohorts suggested that the resourcing of the peer review process could ultimately have a significant impact on its success. This sentiment was echoed by the project team in the collaborative Peer Review of Learning Materials Report (University of Queensland et al, 2004). The report also suggested that peer review processes be integrated into other existing institutional processes. Feedback from the sector and the experience of institutional repositories suggests that the Carrick Exchange may need to employ individuals to manage the peer review processes. The Carrick Exchange should not rely solely on volunteers. Further, it was proposed that not only teaching academics be involved in the processes, but that educational designers and IT personnel may need to be employed to assist with quality assurance and provide feedback on the design and potential effectiveness of the resource. The Carrick Exchange is advised to engage a team of experts with a mix of expertise (e.g. disciplinary, educational and technical specialists) to undertake the formal peer review process. Once a resource has been submitted to the Carrick Exchange staff employed by the Carrick Exchange could complete the initial quality assurance checks, then pass the resource to a review panel. This panel would be convened and chaired by a person nominated by the Carrick Institute. Participants in one of the focus groups suggested this position could be occupied by a Carrick Institute Fellow. As well as managing the peer review process, the Carrick Institute should make provision for a linked system of reward and recognition (see also Resource Identification and Contribution section of this report and principles related to rewards, incentives and recognition for contribution) for authors and reviewers. Focus group participants strongly urged that membership of a peer reference group or formal review panel be viewed as prestigious and that peer review responsibilities valued as part of an academic’s professional development. The British Academy, (Shepherd, 2007) urge those responsible for the management of universities and research institutes to encourage and reward peer review activity suggesting that such a practice might stop high calibre academics, already overburdened with work, from declining peer review duties, and may even attract reviewers. Strategies to ensure the goals of the peer review process Strategies for Carrick Institute and Carrick Exchange 1. Develop policies and procedures for peer review that: • • • • • • • • stipulate parity of formal peer review process with scholarly peer review of research; ensure alignment with institutional promotion policies; identify formal peer review roles and responsibilities and differentiate voluntary and paid roles; provide quality assurance; determine the processes and procedures, criteria and standards of formal peer review; foster review from multiple perspectives (e.g. pedagogical, technical and disciplinary); identify strategies for recognising and rewarding authors and peer reviewers; and include grievance and appeals protocols, guidelines and procedures. 70 2. Lobby university senior executives to acknowledge the formal peer review of resources submitted to the Carrick Exchange in institutional recognition and reward processes and schemes. 3. Establish the goals of the formal peer review process. 4. Prescribe and resource peer review processes and procedures for the levels of peer review (see Figures 3 and 4). Recognise and address in the development of these peer review processes and procedures: • • • • issues relating to effort and resources, time and staffing; that peer review is perceived as both an incentive and disincentive for resource contribution, so members should be able to opt for or against any kind of peer review or quality assurance; workflow alternatives; and supporting materials (e.g. guidelines, protocols). 5. Provide templates, for both the reviewer and the contributor, outlining the criteria for assessment of resources for both quality assurance and formal peer review. 6. Devise a strategy for rewarding and recognising peer reviewers (e.g. provide reward “in kind” through reimbursement for attendance costs for reviewers at an annual Carrick Exchange Conference). 7. Identify a pool of reviewers and potential review panel chairs (e.g. editors of journal and conference proceedings, Carrick Institute Fellows and citation recipients, invite contributors to identify potential reviewers, target specific communities of practice and members of these communities to be reviewers). Strategies for the higher education sector 1. Promote the view that resource sharing is not dissimilar to publication of articles and to this end peer review (including quality assurance) of learning and teaching resources provides a means for achieving this change. Technical strategies 2. Ensure the level of review that contributed resources have undergone is clearly communicated and easily identifiable, this could possibly be an automated feature of the system. 3. Automate as many quality assurance processes as possible throughout the Carrick Exchange: e.g. alerts to notify of broken links, automated metadata tagging etc. Exemplars Appendix G: An overview of formal peer review processes adopted by various repositories. ACELL (Chemistry database): An example of a discipline based repository implementing peer review. Resources on the site are not published until they have been through a thorough evaluation by staff and students. See the workflow model as well. http://acell.chem.usyd.edu.au/homepage.cfm AUTC Learning Designs website: Products of the AUTC project on ICT-based learning designs. http://www.learningdesigns.uow.edu.au/index.html CLOE: Proforma for peer review evaluation of learning objects. http://cloe.on.ca/peerreview.html 71 MERLOT peer reviewers: MERLOT utilises individuals engaged in other institutional initiatives as peer reviewers. http://www.merlot.org/merlot/index.htm The methods adopted by MERLOT to choose or find peer reviewers are: a) Staff volunteer, receive training in the peer review system and then move through a series of steps before becoming a full reviewer, an associate member of the editorial board or an editor. With experience, as qualifications and the quality of work is established, individuals progress up the ladder. b) Institutions nominate peer reviewers who go through a training process and prove themselves to their colleagues. As these individuals are noticed and the quality of their work acknowledged they are asked to take on more responsibility. Key References Health Education Assets Library (HEAL) (2006). Editorial Policy. Retrieved September 22, 2007, from http://www.healcentral.org/services/policies/HEAL_Editorial_Policy_v2.0.pdf Shepherd J. (2007). Help Wanted. The Guardian, 4 September. Retrieved September 22, 207, from http://education.guardian.co.uk/higher/research/story/0,,2161680,00.html Souza, K.H. & Persily, G. (2005). National Peer Review of Digital Learning Materials in the Health Sciences. Retrieved September 22, 2007 from http://cit.ucsf.edu/events/event_mar05.php University of Queensland, Griffith University & Queensland University of Technology (2004). Peer review of learning materials report: A collaborative project involving the University of Queensland, Griffith University and Queensland University of Technology. 30 August 2004. Protocols and mechanisms related to the process and procedures for peer review 3.2.1. The Carrick Exchange should establish processes, criteria and standards that enable consistent, equitable and fair peer review. Peer review should be viewed as “institutionally neutral” to promote its value within the higher education community. As noted in the Literature Review, the MERLOT system of formal peer review is considered one of the best models. This system is based on the academic peer review practices for scholarship and publication in higher education, an “expertise-orientated” approach (Worthen et al., 1997). Building on the MERLOT model, Nesbit et al. (2002) created a convergent participation model for evaluation of learning objects where resources undergo a two cycle process: two individual experts assess the resource then a combined group assessment is made, amalgamating the feedback from both assessments. A model for formal peer review for the Carrick Exchange, based on the MERLOT and CLOE systems, and Nesbit et al (2002), has been provided in Figure 3 and a model for limited review has been provided in Figure 4. 72 Bruce (1997) and Smith (2005) identify a number of important features for a formal peer review process. It is recommended that these be reflected in the design of the Carrick Exchange formal peer review process: 1. The resource contributor should agree to, or nominate the resource for formal peer review. 2. Peer review should occur according to an agreed timeframe, with specific goals and outcomes and clear identification of exactly what is to be reviewed. Peer review protocols, mechanisms and processes should address: What is being reviewed; who the reviewers are; and the criteria for assessment. 3. The existence of a phase in the peer review process dedicated to the review of the process itself may be warranted. It may also be worthwhile for the resource contributor to have tools to conduct their own self-evaluation before the formal peer review process. 4. Individual outcomes of the review should be known only to the peer reviewers and should remain confidential. However, a summarised version of the review could be made public, following the MERLOT model. 5. The peer review process should lead to decisions about action to be taken to ensure improvements. What to review: resources that could be reviewed by peers, include, but are not limited to: • • • • • • • • • • • • • Course outlines, Course materials, Current curriculum, Curriculum development documents, Course syllabi, Statements of teaching philosophy, Assessment programs, Marking schemes, Learning objects, Simulations, Case studies, Role plays, and Learning designs. Who should review: As discussed earlier a group of experts should undertake the review – discipline, educational and technical/media experts, plus an Editorial Chair. (see recommendation 3.1.5.) Peer review should not be conducted by those in a line-management relationship. An effective means of establishing how useful a resource is may be gained from the feedback of practitioners who have used the teaching resource. The collaborative Peer Review of Learning Materials Report (University of Queensland et al, 2004), suggest that staff for whom a significant part of their core business involves developing learning resources may be appropriate peer reviewers. Criteria for review: See recommendation 3.1.3 above. Proformas and protocols: A proforma, developed and informed by the research, is suggested in Appendix H: Carrick Exchange Peer Review Criteria and Standards. Suggested protocols for peer review are provided in Appendix I. Models for the peer review process: See Figures 3 and 4. Roles: The HEAL (2006) Editorial Policy provides some useful detail on roles in the peer review process (see Exemplars below) as does Barton and Warton (2004), in the LEADIRS Workbook. See also Appendix F – Roles identified for the Carrick Exchange. Administrator or Quality Assurance Editor: For both models it is suggested that the quality assurance technical check be conducted by a special Carrick Exchange 73 administrator or editor. Checks on the resource could include accessibility, functionality – checking that hyperlinks work, plug-ins are available, platform and browser compatibility are identified, the resource meets criteria for inclusion in the Carrick Exchange collection, adheres to copyright restrictions and that adequate metadata has been provided. While housed in the non-peer reviewed collection contributors could stipulate whether the resource is simultaneously available for informal commentary, and who may contribute the informal commentary (e.g. everyone, a particular community of practice, trusted colleagues, invited critical friends). Reviewers: Reviewers conduct and record their reviews independently, are then convened by the Panel Chair (probably online – synchronously or a synchronously) to discuss their ratings (numerical and written comments). Convening of the panel to discuss the assessments helps to mitigate against bias by the Chair and gives a more holistic evaluation, taking into account technical, educational and discipline specific issues. Panel Chair: May be appointed by the Carrick Institute. Oversees the process and facilitates the discussion amongst the reviewers, and the final decision which is communicated to the author as a numerical rating (perhaps for quality of content, ease of use and potential educational effectiveness) and comments. Author: May be contacted during the review process by the Panel Chair for further clarification. 74 Figure 3: Formal peer review process model for resource contributions to the Carrick Exchange (Based on MERLOT and CLOE models, and Nesbit, Belfer & Vargo, 2002) CE = Carrick Exchange 75 Figure 4: Quality assurance only (limited peer review) model for resource contributions to the Carrick Exchange Key: CE = Carrick Exchange 3.2.2. The peer review process should meet the expectations of the sector and the needs of members. Quality, accuracy, and currency of resources were elements identified by focus group participants as very important in relation to peer review. The Carrick Exchange should be: A high quality space, housing high quality resources, with peer review the key to this perception. (Focus group participant) Other expectations of the sector identified from the research include: • • timely publication – the time between resource contribution and publication should be as short as possible); Allowing resources to be published informally and be open to comment at the same time as the resource undergoes peer review is a partial solution to this problem (this follows the procedure trialled by the scientific journal Nature); and quality metadata about the peer review outcome so that members can assess the utility of resources with ease, particularly in large collections. Resources that have been through the full peer review process will be associated with scholarly information, (e.g. about the context of use for which the resource was designed, theoretical underpinnings) and assessments by a number of experts. This additional information will provide members of the Carrick Exchange with added value over and above items not subjected to this process. Therefore it is important that the Carrick Exchange regularly checks with members whether the process is meeting their needs. This could be 76 conducted, for example, through online forums on the Carrick Exchange and during conferences. Strategies related to the process and procedures for peer review Strategies for the Carrick Institute and the Carrick Exchange 1. Establish a formal peer review process that meets the expectations of the sector (Figure 3 is provided as a possible process). 2. Identify the roles and responsibilities required to implement the designed formal peer review process. (see recommendation 3.2.1 above and Appendix F.) 3. Develop a strategic plan to resource roles and responsibilities, e.g. generate a databank of invited or potential personnel, including panel chairs (such as Carrick Institute Fellows and citation winners), and peer reviewers (with a mix of expertise), quality assurance administrative or editorial staff, reviewers (some of whom may suggested by contributors). 4. Ensure those undertaking the formal peer reviews are adequately qualified and trained; consider mandatory training or professional development for peer reviewers. 5. Develop standards and criteria for assessing educational resources, submitted to the Carrick Exchange for formal peer review. 6. Check with members that the peer review process is meeting needs. This could be facilitated via online forums and during conferences. 7. Ensure all resources are clearly identified as to their status regarding peer review, quality assurance etc. 8. Enable contributors to identify their preference for the resources they submit, e.g.: • • • • For/not for formal peer review; For/not for limited review, i.e. quality assurance; For/not for commentary and feedback; and For public release/for limited (small group) release. Strategies for the higher education sector 1. Position membership of a Carrick Exchange peer reference panel as a role with prestige, valued as part of an academic’s or professional staff member’s professional development. Technical strategies 1. Automate the initial quality assurance process as much as possible (e.g. alerts to notify of broken links, automated metadata tagging etc.). Exemplars AUTC The Learning Designs project: http://www.learningdesigns.uow.edu.au/index.html The project used a formal peer review process comprising three main phases: 1. Prior to resource submission, contributors completed a “learning design submission form” providing details of their learning design and themselves. 2. Two evaluators individually completed an assessment of the learning design (instructions and a proforma for recording this evaluation were provided to evaluators). 77 3. The two evaluators confer with each other on their individual assessments of the learning design and reach a consensus about the evaluation generating and submitting one evaluation report. CLOE, Cooperative Learning Object Exchange: Peer review processes. http://cloe.on.ca/ HEAL editorial policy v2 (2006): Details about the roles in the peer review process. http://www.healcentral.org/services/servicesPeerReview.jsp MERLOT peer review model: This model is widely considered to be a good example of a peer review process . http://taste.merlot.org/aboutpeerreivews.html Nature magazine “Precedings”: Pre-print publication of scientific research open for general peer review by anyone who supplied their name and email address. This allows for dual track peer review. http://blogs.nature.com/peer-to-peer/2007/06/nature_precedings.html Key References Bruce, C. (1997). Peer Review: A Handbook, Queensland University of Technology. Retrieved September 23, 2007, from http://sky.fit.qut.edu.au/~bruce/peerreviewhanbook/. Health Education Assets Library (HEAL) (2006). Peer Review of Digital Scholarship. Retrieved September 22, 2007, from http://www.healcentral.org/services/servicesPeerReview.jsp. McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and implementation. Journal of Digital Information, 5(3), http://jodi.tamu.edu/?vol=5&iss=3 Nesbit, J. Belfour, K. & Vargo, J. (2002). A convergent participant model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3). http://www.cjlt.ca/content/vol28.3/nesbit_etal.html Smith, C. (2005). Guidebook for individuals embarking on peer review of teaching, The Teaching and Educational Development Institute, University of Queensland. Retrieved September 23, 2007, from http://www.tedi.uq.edu.au/downloads/evaluations/Guidebook_PeerReview.pdf. Taylor, P. & Richardson, A. (2001). Validating scholarship in university teaching. Evaluations and Investigations Programme. Canberra, ACT: DETYA, Commonwealth of Australia. The University of Queensland, Griffith University & Queensland University of Technology (2004). Peer review of learning materials report: A collaborative project involving the University of Queensland, Griffith University and Queensland University of Technology. 30 August 2004. Worthen, B., Sanders, J. & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd Ed.). New York: Longman. 78 Protocols and mechanisms related to informal commentary 3.3.1. Guidelines and protocols for informal commentary should be established to ensure that the needs and expectations of contributors and users are met. Peer review may be an informal process whereby members of the community voluntarily respond to others’ contributed resources, or resources stored elsewhere but linked to via the Carrick Exchange. Informal commentary might include discussion, feedback or comments made on resources and ideas contributed to the Carrick Exchange. Commentary is seen to be beneficial because it can provide informed users’ ratings, perspectives and assessment. Research participants indicated that it would be preferable if all informal comments were attributable to identified members. It was believed that this would help foster trust and credibility within the community. Guidelines for informal peer review or commentary will enhance the likelihood of submitted feedback being a valuable form of information especially with regard to further and subsequent resource development. Guidelines and procedures should ensure that commentary and feedback is managed via established protocols and processes, reducing the incidence of derogatory or negative comments. Guidelines should provide a model for commentary and develop members’ skills in providing constructive feedback. In addition, members’ feedback should be aligned with Carrick Exchange processes and policies to provide a desired standard of commentary. Members of the Carrick Exchange will have diverse needs and varying disciplinary backgrounds. Individuals may use submitted resources in ways the creator might not necessarily have imagined, and it will be valuable to record these diverse contextual uses. The issue is how to encourage those who have tried out a resource downloaded from the Carrick Exchange to return and report on their reuse and/or modification, and record their peer review. Communities such as the LAMS Community have found that only a very low proportion of items in the repository get rated. Research participants saw spaces within the Carrick Exchange where contributors can house resources and invite trusted colleagues to review and make comments about their resource as worthwhile. As one focus group participant explained: You are more likely to make comments about a resource, and your use of a resource, where you feel secure in knowing who the audience is, as opposed to making a comment in a forum of strangers, who may pounce and ridicule your comments. (Focus group participant) As with formal peer review, there should be mechanisms to enable the contributor to indicate if they want informal commentary or feedback in any form, when submitting a resource to the Carrick Exchange. A ratings system was seen to be both desirable and problematic. While setting criteria for numerical ratings (e.g. one to five stars) was considered an asset, it was often seen as difficult for users to know how consistently any criteria might have been be applied in an informal ratings context. A resource may be useful to one member but not to another, depending on e.g. context, target audience and experience. Questions about a ratings system raised by research participants that the Carrick Exchange should address include the following: • • • • Would the system be an anonymous rating system, and therefore lack credibility? How would you compare one rating of 5 stars with another rating of 5 stars? What is being measured? What are the rating criteria? 79 • Who will return to the website and rate others’ contributions once they have downloaded a resource? What incentive is there to go back? What if a resource gets a poor rating? Is this detrimental for the contributor as well as the Carrick Exchange? Or does the idea of “accountability when public”, assume a certain standard and avoid this issue? • Solutions should also be found for how the experience of the person providing the commentary or review might be judged. They may need to complete a profile of their experience and rate the relevance of their expertise in reviewing a resource. Strategies for informal commentary Strategies for Carrick Institute and the Carrick Exchange 1. Develop guidelines and procedure/s for submission of informal commentary. These guidelines and procedures should ensure that: • • • • • commentary and feedback is moderated (e.g. by Carrick Exchange personnel) so that negative comments are managed appropriately; (users could check a box to say “This comment is inappropriate”); Carrick Exchange appoint staff and volunteers to model constructive feedback approaches; informal comments should be attributable to identified members; the implemented (star) rating system has guiding criteria to assist those providing feedback; it may be advantageous to allow this rating system to evolve over a period of time (e.g. 6 months) as a consequence of the submissions and suggestions of users; and contributors are able to nominate, when submitting a resource, if they do not want feedback in any form. 2. Develop an appeals process to manage members’ identification of an inappropriate resource or unacceptable comments. This could be an icon that can be clicked beside every comment or resource (e.g. following the YouTube model). Strategies for the higher education sector 1. Provide a registry of staff willing to provide feedback and act as community reference group members, e.g. name and times when available. Technical strategies 1. Provide mechanisms to enable users to rate and provide discussion, feedback and comments about resources submitted to the Carrick Exchange. 2. Ensure that a requirement of submitting informal commentary is identification of the person submitting the feedback (i.e. not anonymous). 3. Provide functionality to allow informal commentary on developing work, with mechanisms to enable contributors to restrict or prohibit access to work under development. Exemplars Amazon.com: An eCommerce company which sells goods over the internet advertises details of available products listed in various merchandise categories. Each product has a star rating 80 and a link to existing customer reviews and the facility for users to submit their own review. The publicised star rating is an average of all the submitted ratings from reviewers. Submitted comments can be rated and commentary on the comments can also be submitted generating in some instances an asynchronous discussion in regard to the initial submitted comment. Reviewers are ranked according to the number of comments they have contributed across the site. For example at the time of writing this report the top reviewer had submitted 14,794 reviews. http://www.amazon.com The Carnegie Foundation for the Advancement of Teaching, Teaching and Learning Commons: An community space provided to enrich and encourage exchange of knowledge about teaching and learning. This repository invites users to read, understand, and comment on others’ work. The exchange provides resources, suggestions and a means to comment on work in the Commons (e.g. suggestion of ways to annotate). http://commons.carnegiefoundation.org/ EducaNext: A service supporting the creation and sharing of knowledge for higher education. It is open to any member of the academic or research community. Informal peer review is encouraged from those using the learning resources (faculty members and students) inviting them to describe their perceptions and experiences of a learning resource after they have used it. Descriptions are based on a tripartite inquiry on the technical, pedagogical, and organisational aspects of the resource. http://www.educanext.org/ubp Jorum: A JISC 17 -funded collaborative venture in UK higher and further education sector. It is a national repository developed to collect and share learning and teaching materials, allowing their reuse and repurposing. Jorum has a star rating and comments service where users are encouraged to add ratings and evaluative comments to resources. In addition, Jorum will be establishing a web-based forum facility for comments and discussions held at the Jorum website. Contributors and members of the Jorum team are notified when comments are added. http://www.jorum.ac.uk LAMS Community: Members can contribute and download learning designs and contribute to discussion forums. Learning designs can be rated with a five star rating and qualitative comments can be added. http://lamscommunity.org MERLOT: Invites users to rate linked resources from one to five stars and to indicate if the resource has been used in a classroom. A comment requires remarks as well as the star rating. Materials that have member comments posted can be found through an advanced search. MERLOT also offers “From the Author Snapshots” which are short explanations from authors of learning materials about the development and use of their learning materials. http://www.merlot.org MySpace: A social networking website offering an interactive, user-submitted network of friends, personal profiles, blogs, groups, photos, music and videos. MySpace enables user identified comments to be added to submissions such as videos and music. A percentage rating based on the positive (“booyah”) or negative (“no way”) votes a submission receives is also provided. The profiles of users providing the informal commentary can be accessed by clicking on their user name listed next to the comment. The number of times a resource has been accessed and the number of comments that have been submitted in relation to a resource are also listed with the resource details. http://www.myspace.com/ Slashdot: A science, science fiction, and technology-related news website that features usersubmitted and editor-evaluated current affairs news stories with a "nerdy" slant. Stories on the 17 The Joint Information Systems Committee (JISC) supports education and research by promoting innovation in new technologies and by the central support of ICT services (http://www.jisc.ac.uk/). 81 site have linked discussions with comments generally submitted by site visitors which Slashdot's editors accept or reject for general posting. Threading and moderation of submitted comments are features of the slashdot discussions and contributors will also categorise the nature of their comments/discussion (e.g. insightful, funny, informative, interesting, offtopic, flamebait, Troll) http://slashdot.org/ Warwick University Experts directory: A directory of “experts” - individuals who have offered to help others in their department, faculty or the whole university by providing advice in their area of expertise. Warwick University advertise that an expert may be able to: • • give technical support. help with choosing tools and techniques, and understanding how to best use them. http://www2.warwick.ac.uk/services/elearning/experts/ YouTube: A video sharing website where users can upload, view and share video clips. Unregistered users can watch most videos on the site, while registered users are permitted to upload an unlimited number of videos. Details on the number of times a video has been viewed, rated as a user’s favourite video, rated by users or had a comment posted in relation to it are provided. Comments that have been provided can also be rated using a “thumbs up” or “thumbs down”, denoting good or poor comments, and there is also the option to reply to a posted comment. Responses to a video can be in the form of text comments or a video response. Each posted video has a star rating that is an average of the collective submission of ratings. Details on when the video was submitted, meta tags, and the contributor are also provided. http://www.youtube.com Key References Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. JISC Technology and Standards Watch. Retrieved May 1, 207 from www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf Protocols and mechanisms related to rewards and incentives for formal peer review and informal commentary 3.4.1. The Carrick Exchange must implement rewards and incentives for participation in either formal peer review or informal commentary Rewards and incentives for participation in either formal peer review or informal commentary underpin the ultimate success of the entire review process. For contributors and users to engage in the review process there needs to be a perception of added value. This was firmly supported by research participants with comments such as: Given the level of high quality review that is being sought it may be necessary to pay the reviewer. (Focus group participant) Some sort of recognition needs to be given to peer reviewers. (Focus group participant) 82 Recognition of effort in providing reviews. (Focus group participant) What is the incentive for those doing the reviewing? (Interviewee) Why should I review something? I’m already reviewing journal papers, grant applications, conference papers – this is one more. (Focus group participant) Research participants have identified the following as rewards and incentives for participation in either formal or informal peer review: • • • • Members are more likely to contribute commentary and engage in discussions if they trust the community. This community may also be an incentive, as one focus group participant explained: “There is an aspect of community that acts as an incentive for engaging in commentary”. Positive, productive feedback is more likely to encourage contributors to submit resources for peer review and commentary. Acknowledgment and recognition by the Carrick Exchange and institution for involvement in formal peer review (either as a contributor of a resource for formal peer review or as a peer reviewer). Links to institutional promotion, as focus group participant explained: As far as recognition of people’s contribution to teaching and learning goes, promotion is the epicenter, and probably the next closest is actually applying for a job … If Carrick can produce changes in that area . . . so that certain forms of feedback . . .can be included in a teaching portfolio or equivalent, that would be a huge incentive to contribute. (Interviewee) Strategies for rewards and incentives for formal peer review and informal commentary Strategies for Carrick Institute and the Carrick Exchange 1. Establish protocols with Australian universities for encouraging the use of formally peer reviewed resources as evidence for promotion and career advancement. 2. Establish criteria and processes for rewards and incentives for contributing resources that are peer reviewed, and for being a peer reviewer (e.g. the ‘No 1 reviewer’ - see the amazon.com example; reimbursement for attendance costs for reviewers at an annual Carrick Exchange Conference for peer review). 3. Recognise the effort in providing commentary within an annual report. 4. Ensure that the products of peer review (reports etc.) have currency and credibility in relation to established recognition and reward processes and contexts (especially promotion). Strategies for the higher education sector 1. Recognise formally peer reviewed resources as evidence for promotion and career advancement. Technical strategies 1. System facility that provides a “Monthly spotlight on No 1 reviewer”. 2. Automated system provided tally of contributions per person. 83 Exemplars Amazon.com rewards: Informal reviewers rewarded by giving them a rank that is assigned according to the number of informal reviews and comments they have submitted. http://www.amazon.com Key References Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project, Academic survey: Final report. Loughborough University & JISC. Retrieved July 16, 2007, from http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/1137423 150_SurveyReport.pdf Recommendations for peer review and commentary policies Policies will need to be developed in the following areas: 1. Guidelines and policies to inform national policy on parity of formal peer review of educational resources compared with scholarly peer review of research. 2. Policy for reporting on peer review processes (communication with institutions) to link with institutional promotion policies. 3. Quality assurance practices and procedures: a. Triage resources identified for this process. b. Timeline indicating time between resource contribution and publication, depending on classification. c. Dual track processing if resource submitted for quality assurance and if submitted concurrently as a resource for informal peer review. 4. Policy and procedures for attribution and identification of reviewers: a. Identify source of all informal commentary. b. Identify reviewers in the formal peer review system. c. Rated resources will be anonymous, but all commentary identified. d. Identify what work is voluntary and what is paid work in the peer review system. e. Composition of review teams (educational, discipline and technical/media experts) 5. Policy and procedures for formal peer review. 6. Criteria used for formal peer review, and for informal rating, e.g. quality of content, ease of use and potential effectiveness as a teaching tool (Merlot criteria). 7. Guidelines for submitting resources for peer review or informal commentary, including protocols for submitting student feedback as part of the review process. 8. Guidelines and policies for the editorial board. 9. Appeals policy. 10. Publication policy. 84 11. Commentary guidelines to ensure thoughtful review, not damaging commentary (terms of use). 12. Policies for implementing training in peer review. 13. Management policy for peer review conference day. 85 Conclusion ______________________________________________ The response from the sector through interviews, focus groups and reference groups, to the development of the Carrick Exchange has been measured. Generally there is positive anticipation, tempered by knowledge that the issues to be addressed during implementation are not insignificant. A list of the benefits of the Carrick Exchange and challenges to adoption that have emerged through the research are identified. The lists tend to show that the sociocultural challenges are greater than the pedagogic issues. In contrast, the pedagogic benefits were seen to be considerable. If the Carrick Exchange can address the socio-cultural challenges it is likely that it will have an initiative that meets the pedagogical needs of its target audience. Benefits The potential benefits of the Carrick Exchange were seen to be: Socio-cultural issues • • • Activities and personnel to support and encourage use; Engagement strategies that target user groups with well-tailored strategies and incentives; and Acknowledgement as the author when other members reuse a resource contributed to the Carrick Exchange. Pedagogic issues • • • • • • • • • • • • Provision of a database for the sharing and reuse of high quality professional development and teaching resources (digital artefacts and communications), linked to other similar databases and networks within and outside of Australia; Access to Carrick Institute project reports, documents and output; A well classified and maintained collection of teaching and learning resources which is readily searchable and of sufficient substance to warrant further investigation; A collection of resources with a discipline focus; Links to discipline and cross-disciplinary communities and networks; A formal peer review system based on scholarly research peer review processes; A well maintained registry of national and international experts in teaching and learning; Shared workspaces that support private and group projects; Informal and timely feedback from peers on developing resources under development; Personnel within institutions to support engagement and pedagogical processes associated with use and reuse of resources sourced from or linked to the Carrick Exchange (e.g. educational and staff developers); Training in the use of the Carrick Exchange and the tools it provides; and An annual Carrick Exchange conference for the purposes of sharing ideas and knowledge, networking, distribution of awards, identifying resources to be contributed to the Carrick Exchange, peer review of resources, training in peer review, and professional development generally. Organisational and information management issues • Champions placed within the Carrick Institute and universities to lead and promote the Carrick Exchange; 86 • • • • • Support personnel within institutions to help with information literacy, metadata creation, and copyright and licensing issues (librarians); The ability of the Carrick Institute to lobby senior management and achieve commitment from key stakeholders; Rewards and incentives for contribution, peer review and leadership relating to the Carrick Exchange, that link to promotion and career advancement; Policies and strategies that link home institutions with the mission of the Carrick Institute and the Carrick Exchange; and Active promotion and marketing of the system for its pedagogic benefits not only its technical capabilities. Technological issues • • • Incorporation of Web 2.0 technologies that foster connection and member creation of resources; Easy access to new software and tools for trial purposes; and Seamless and intuitive integration with home institution and other databases. Challenges Using the organising framework of Margaryan, Currier, Littlejohn and Nicol (2006), the clustering of perceived or actual challenges to use and engagement with the Carrick Exchange were seen to be: Socio-cultural issues • • • • • • • • • Initiating and maintaining interest in the innovation; The real costs of contribution or membership; Maintenance issues for the originating author after submission of the resource; Time-poor academics whose workloads prohibit engagement in activities other than immediate priorities; Perceptions about increased workloads as a result of contribution to the Carrick Exchange (contribution of artefacts or communications); Compulsory contribution procedures (voluntary contribution preferred); Cross-institutional rivalry and competition; The impact of formal and informal peer review and commentary processes regarding consumption of time and resources, cronyism and poorly moderated commentary; and Issues of trust in: the system; the credibility of informal peer review processes; the management of intellectual property rights by individuals and their institutions; and the maintenance of rewards and incentives to support contribution to the system. Pedagogic issues • • • Perceptions that the Carrick Exchange provides no additional benefits over and above the systems and communities that are already in existence; Currency and quality of the resources in the repository; and Timely publication of peer reviewed resources (long timeframes compromising access and currency of resources). Organisational and information management issues • • Inconsistent application of rights management, fears of losing ownership of resources, intellectual and moral rights; Lack of alignment between institutional policies on ownership of resources, intellectual property rights and copyright, and individuals’ understanding of the policies; 87 • • • • The resource intensive nature of formal peer review; Lack of parity between scholarship and effort in teaching, and scholarship in research; Lack of benefits and rewards for contributing to the Carrick Exchange; and Perceptions that the Carrick Exchange will be another interesting concept that is not sustained into the future due to inadequate provision of management, policy and financial resources. Technological issues • Lack of ease of use of the system, including authentication and metadata procedures. Initiating uptake in the early stages and maintaining engagement into the future, consistent application and understanding of licensing and intellectual property policies, and cultural, pedagogical and technical issues, all emerge as dimensions of development and implementation that need clarity. The tension between the desire to share effective pedagogy and resources, and the inherent competition that exists within and across the sector is of considerable concern. There seems to be a general desire to promote best practice within the sector, but academics and professional staff are not naive and realise they must also consider how these practices can be factored into busy workloads and balanced against other competing career options. Rewards and incentives will be crucial to address these issues and encourage peer review, sharing and reuse. Further, there are examples of sharing of learning objects and resources within and across institutions from which lessons can be learned. At the institutional level, the new Research Quality Framework (RQF) requirements that mandate the deposit of research quantum into institutional repositories may effect some cultural change within the sector regarding knowledge management and sharing. This could have positive ramifications for the Carrick Exchange in terms of the practices of academics using repositories. However, the only mandatory deposit that is likely to be sustained in the Carrick Exchange is the deposit of Carrick Institute related documents and project outcomes as an obligatory responsibility of acceptance of grant funding. This will, nonetheless, seed the Carrick Exchange collection and help the establishment of a critical mass or resources. The introduction of peer review processes will be an interesting new initiative to track. The sector seems to regard formal peer review as an attractive option, and as a means of rewarding and documenting good teaching and learning. However, the disincentives of the formality itself, and the time taken to complete thorough peer review will be a barrier to some, as will the risks of informal peer review (negative feedback and impact on competitive advantage). Despite these problematic issues, there are a number of exciting new aspects to the Carrick Exchange that could have significant benefits for the higher education sector, such as: aggregation of a major collection of Australian professional development materials relating to teaching and learning; collaborative workspaces for project development; formal and informal peer review; networks and communities that support sharing and reuse of resources; and a registry of Australian and international experts in diverse fields of teaching and learning. The Carrick Institute has considerable prestige within the sector, and is in a position to lead effective change nationally and possibly internationally. Representatives of the sector in this study have confirmed that by providing champions, acting as a sponsor, promoting engagement, providing pedagogical support, addressing real needs and rewarding and recognising effort for contribution and review of others’ work, the Carrick Exchange can find a place in Australian higher education. As with any issue that requires change management, effective communication will be the key. 88 Future research The findings of this research have been commissioned to inform the development of the Carrick Exchange. The next step after delivering this report to the Carrick Exchange management team is to present the findings at a symposium of ascilite members at the 2007 annual conference. After receiving feedback from the members and developing the ideas about engagement, sharing and reuse of resources, rewards and recognition and peer review, the findings from the final cycle of research will also be delivered to the Carrick Exchange management team, further informing the development process. The findings so far indicate that further research needs to be undertaken particularly in the area of rewards and recognition, and in refinement of the peer review model for evaluation of educational resources. Evaluation of the initiative as a whole is another important area for future research. The Design-Based approach used in this research proved to be a robust methodology. Each new data set was gathered and analysed, its significance and relevance was tested against the literature and the views of a new group of informed practitioners. This iterative and collaborative process strengthens critical evaluation and the final outcomes of the research. A set of eight general design principles have been derived from the research. 1. Designing a community and a space for the sharing and development of teaching and learning materials and processes for the higher education sector requires recognition of the diversity of potential users and their needs. The development of community is complex. Existing communities will not necessarily shift from their current homes to relocate in the new space. There must be a significant advantage that will encourage networks and communities to relocate, re-form or establish in the new space. This must be well articulated to potential members. 2. Seed the repository with resources closely related to the mission of the sponsoring body for the innovation. For example, a national collection of high quality professional development and teaching resources (digital and human), linked to other similar databases and networks of high quality resources, will be of considerable benefit to new and experienced academics and educational support staff within the higher education sector. 3. There is a tension between the formal and informal processes of a network and repository such as the Carrick Exchange. The design and dynamics of the informal sharing and review processes must be balanced against the formal requirements of standards, formal peer review, rights management and preservation of materials. 4. Development of a community, built around a national repository with international linkages, requires funding and management plans and strategies which address development in a staged approach. Initial design and implementation resource requirements and strategies will differ from those which sustain the initiative into the future and embed it into the everyday work practices of the target audience. 5. To create a critical mass of resources (digital and human) from which the initiative can build and grow, existing discipline communities, special interest groups and networks should be leveraged. Members from these groups can provide leadership and models of engagement in the identification and contribution of resources from which the collection in all its forms can grow. 6. To be effective at the national level the community and its repository need to build on and develop seamless integration with organisational and information management systems that already exist within the sector. This includes aligning in particular with university organisational strategies and policies, and systems of reward and recognition. 89 7. Change management strategies are difficult to implement, requiring time and systemic support to be effective. Change cannot be effected through a ‘face-less” website. Key change agents must be identified and supported using the affordances of the new emerging community. 8. Peer review of learning and teaching resources is both an incentive and a disincentive for contributors to a repository and community such as the Carrick Exchange. Rewards and incentives to support contribution and peer review processes is required to surmount the barriers. 90 References ______________________________________________ AFP. (2007). Facebook to post members profiles on online search engines. Retrieved September 27, 2007, from http://afp.google.com/article/ALeqM5iWsMIytjvcGtnN9Q6ob6X3YHC3BQ Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. Joint Information Systems Committee (JISC) Technology and Standards Watch, February. Retrieved August 15, 2007 from http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf Australian Partnerships in Sustainable Repositories APSR. Populating a repository – selected references (bibliography). Retrieved September 27, 2007, from http://apsr.anu.edu.au/apsrfw/rr/index.php?pid=94 Barton, M. & Waters, M. (2004). Creating an institutional repository, LEarning About Digital Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 29, 2007, from http://www.dspace.org//implement/leadirs.pdf Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project, Academic survey: Final report. Loughborough University & JISC. Retrieved July 16, 2007, from http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/1137423 150_SurveyReport.pdf Bradley, C. & Boyle, T. (2004). The design, development and use of multimedia learning objects. Journal of Educational Multimedia and Hypermedia, 13(4), 371-389. Breslin, C., Nicol, D., Grierson, H., Wodehouse, A., Juster, N. & Ion, W. (2007). Embedding an integrated learning environment and digital repository in design engineering education: Lessons learned for sustainability. British Journal of Educational Technology, 38(5), 805-816. Bruce, C. (1997). Peer Review: A Handbook, Queensland University of Technology. Retrieved September 23, 2007, from http://sky.fit.qut.edu.au/~bruce/peerreviewhanbook/ Dalziel, J. (2002). Reflections on the COLIS (Collaborative Online Learning and Information Systems) Demonstrator project and the “Learning Object Lifecycle”. In A. Williamson, C. Gunn, A. Young & T. Clear (Eds.), Winds of Change in the Sea of Learning: Proceedings of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Auckland, New Zealand: UNITEC Institute of Technology. Dalziel, J. (2006). Building communities of designers, In H. Beetham and Sharpe, R. (Eds.), Design for Learning: Rethinking Pedagogy for a Digital Age. RoutledgeFalmer: London. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Desire (2000). Desire Information Gateways Handbook. Retrieved March 31, 2007, from http://www.desire.org/handbook/2-1.html Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan, and P. MacGillis (Eds.), The guidebook for performance improvement: Working with individuals and organizations. San Francisco: Pfeiffer. 91 Ehrmann, S., Gilbert, S., & McMartin, F. (2007). Factors affecting the adoption of facultydeveloped academic software: A study of five icampus projects. Retrieved September 19, 2007, from http://www.tltgroup.org/iCampus/iCampus_Assessment_Full.pdf Ely, D. (1999). Conditions that facilitate the implementation of educational technology innovations. Educational Technology, 39(6), 23—27. Evans, V. (2007). Networks, connections and community: Learning with social software. Australian flexible Learning Framework, Australian National Training Authority and Commonwealth of Australia. Retrieved July 10, 2007, from http://www.flexiblelearning.net.au/flx/go/pid/377 Foster, N.F. & Gibbons, S. (2005). Understanding faculty to improve content recruitment for institutional repositories. D-Lib Magazine. 11(1). Retrieved September 21, 2007, from http://www.dlib.org//dlib/january05/foster/01foster.html Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119). Macquarie University E-Learning Centre of Excellence: Sydney, NSW. Gunn, C., Woodgate, S. & O’Grady, W. (2005). Repurposing learning objects: An alternative to reusability? ALT-J, 13(3), 189-200. Hand, T., Gosper, M., Woo, K., Gibbs, D., Kerr, S. & Rich, D. (2004). Learning objects: User perspectives on the conditions surrounding their use. Proceedings of the Ed-Media 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications, 21-26 June, Lugano, Switzerland, 66–72. Health Education Assets Library (HEAL) (2006). Editorial Policy. Retrieved September 22, 2007, from http://www.healcentral.org/services/policies/HEAL_Editorial_Policy_v2.0.pdf Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities. D-Lib Magazine, 13(5/6). Retrieved May 16, 2007, from http://www.dlib.org/dlib/may07/henty/05henty.html Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. and Koper, R. (2005). Facilitating participation: From EML web site to the learning network for learning design. Interactive Learning Environments, 13(1-2), 55–69. Jorum. (2007). Contributing to Jorum. Contributing to Jorum for Beginners. Retrieved September 5, 2007, from http://www.jorum.ac.uk/contributors/chelp/guidelines.html Kenny, J. (2003). A research-based model for managing strategic educational change and innovation projects. Annual Conference Proceedings of HERDSA (the Higher Education Research and Development Society of Australasia), University of Canterbury, Christchurch, NZ, 6–9 July 2003, Retrieved August 11, 2007, from http://surveys.canterbury.ac.nz/herdsa03/pdfsref/Y1102.pdf Koppi, T. & Lavitt, N. (2003) Future directions. In E. Duval, W. Hodgins, D. Rehak and R. Robson (Eds.) Learning Objects 2003 Symposium: Lessons learned, questions asked. EDMEDIA 2003 World Conference on Educational Multimedia, Hypermedia & Telecommunications, Honolulu, Hawaii, USA, 24 June 2003, 39-43. Retrieved May 2, 2007, from http://www.aace.org/conf/edmedia/LO2003Symposium.pdf Learning Technology Standards Committee (2002). IEEE Standard for Learning Object Metadata. IEEE Standard 1484.12.1, Institute of Electrical and Electronics Engineers, New 92 York. Retrieved September 18, 2007, from http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf Lefoe, G., O'Reilly, M., Parrish, D., Bennett, S., Keppell, M., & Gunn, C. (2007). The Carrick Exchange: Not just another repository. Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and Telecommunications, 25-29 June, Vancouver, Canada, 108-113. Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning. London and Sterling: Kogan Page. Margaryan, A., Currier, S., Littlejohn, A. & Nicol, D. (2006). Community dimensions of learning object repositories, Retrieved CD-LOR deliverable 1: Learning communities and repositories. Retrieved August 30, 2007, from http://academy.gcal.ac.uk/cdlor/CDLORdeliverable1_learningcommunitiesreport.doc. Margaryan, A. & Littlejohn, A. (2007). Cultural dimensions of learning object repositories. Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and Telecommunications, Vancouver, Canada, 25 -29 June, 4335-4343. Margaryan, A., Milligan, C., & Douglas, P. (2007). Community dimensions of learning object repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object repositories. Retrieved July 23, 2007, from http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Structured_Guidelines_v1p0.pdf Mason, J., Macnamara, D. & Galatis, P. (2007). An investigation into digital rights management issues in the Australian higher education sector: A discussion paper. Commissioned by Education.au. Internal report. McDonald,J. & Star, C. (2006). Designing the future of learning through a community of practice of teachers of first year courses at an Australian university. In R. Philip, A Voerman & J. Dalziel (Eds), Proceedings of the First International LAMS Conference 2006: Designing the Future of Learning (pp.65-76). 6-8 December 2006, Sydney: LAMS Foundation. http://lamsfoundation.org/lams2006/papers.htm McKenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, Adoption and Adaptation of Project Innovations in Higher Education. A report for the Carrick Institute for Learning and Teaching in Higher Education. Sydney: University of Technology. McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and implementation. Journal of Digital Information, 5(3), Retrieved September 21, 2007, from http://jodi.tamu.edu/?vol=5&iss=3 Monahan, C. & Owies, D. (2005). The DOMS at Deakin. Proceedings of the Educause Conference of Australasia, Auckland, April 5-8, 2005. Retrieved September 10, 2007, from www.educause2005.auckland.ac.nz/interactive/papers/A20.pdf Najjar, J., Ternier, S. & Duval, E. (2004). User behaviour in learning objects repositories: An empirical analysis. Proceedings of the Ed-Media 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications, 21-26 June, Lugano, Switzerland, 43734378. Nature (2006). Nature's peer review trial. December. doi:10.1038/nature05535. Retrieved September 20, 2007, from http://www.nature.com/nature/peerreview/debate/nature05535.html Nature (2007). Nature Precedings is live. Retrieved September 28, 2007, from http://blogs.nature.com/peer-to-peer/2007/06/nature_precedings.html 93 Nesbit, J., Belfour, K. & Vargo, J. (2002). A convergent participant model for evaluation of learning objects. Canadian Journal of Learning and Technology, 28(3). Retrieved August 3, 2007, from http://www.cjlt.ca/content/vol28.3/nesbit_etal.html Pappalardo, K., Fitzgerald, A., Fitzgerald, B., Kiel-Chisholm, S., O'Brien, D. & Auston, A. (2007). A guide to developing open access through your digital repository. DEST funded Open Access to Knowledge Law Project OAK Law Project. Retrieved September 27, 2007, http://eprints.qut.edu.au/archive/00009671/01/9671.pdf Philip, R. (2007). Adaptable and reusable learning designs: Will they be shared? Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and Telecommunications, Vancouver, Canada, 25 - 29 June 2007, 2980-2985. Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible Learning Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian Flexible Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from http://flexiblelearning.net.au/aboutus/keydocuments.htm Phillips, R., Rai, R., Sudweeks, F., Gururajan, R., Jones, M., Shiers, D. & O’Neil, R. (2005). Use and usability of learning objects within the COLIS Demonstrator framework. In J. Dalziel, R. Philip, & J. Clare (Eds.) The COLIS Project: Collaborative Online Learning and Information Services. Macquarie University E-Learning Centre of Excellence (MELCOE), Macquarie University: Sydney. Phillips, R., Orrell, J. & Millea, J. (2007). Creativity in the Envisioning of a Collaborative Learning Space: First stages in the development of the Carrick Exchange. Proceedings of AusWeb 07, Coffs Harbour, NSW, 30 June - 4 July. Retrieved July 30, 2007, from http://ausweb.scu.edu.au/aw07/papers/refereed/phillips/ Reeves, T. (2000). Enhancing the worth of instructional technology research through ‘design experiments’ and other development research strategies. Paper presented at the Annual Meeting of the American Educational Research Association, 27 April, New Orleans, LA. Ringan, N., Corley, L. & Campbell, L. (2005). Evaluation of the COLIS Framework form a UK perspective. In J. Dalziel, R. Philip & J. Clare (Eds.), The COLIS Project: Collaborative Online Learning and Information Services, Macquarie University E-Learning Centre of Excellence: Sydney, NSW. Ch. 11. Rogers, A. (2006). Get Wiki with it. Wired. 14 September, pp.30-32. Retrieved September 18, 2007, from www.wired.com/wired/archive/14.09/start.html?pg=3 - 33k Rogers, E. (2003). Diffusion of innovations. New York: Free Press. Scott, G. (1999). Change matters: Making a difference in education and training. St Leonards NSW: Allen and Unwin. Shepherd J. (2007). Help Wanted. The Guardian, 4 September. Retrieved September 22, 207, from http://education.guardian.co.uk/higher/research/story/0,,2161680,00.html Siemens, G. (2007). Scholarship in an age of participation. Elearnspace, March 27. Retrieved September 29, 2007, from http://www.elearnspace.org/Articles/journal.htm Smith, C. (2005). Guidebook for individuals embarking on peer review of teaching, The Teaching and Educational Development Institute, University of Queensland. Retrieved September 23, 2007, from http://www.tedi.uq.edu.au/downloads/evaluations/Guidebook_PeerReview.pdf 94 Souza, K.H. & Persily, G. (2005). National Peer Review of Digital Learning Materials in the Health Sciences. Retrieved September 22, 2007 from http://cit.ucsf.edu/events/event_mar05.php Stuckey, B. & Arkell, R. (2006). Development of an e-learning knowledge sharing model. Knowledge Sharing Services Project. Australian Flexible Learning Framework. Retrieved July 23, 2007, from http://www.flexiblelearning.net.au/…/kjohnson/public/Development_of_an_%20elearning_knowledge_sharing_model.pdf Taylor, P. & Richardson, A. (2001). Validating scholarship in university teaching. Evaluations and Investigations Programme. Canberra, ACT: DETYA, Commonwealth of Australia. University of Queensland. (2007). Report of the Working Party on the Diversity of Academic Roles. Internal review. Retrieved September 27, 2007, from http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesR eportApr07.doc University of Queensland, Griffith University & Queensland University of Technology (2004). Peer review of learning materials report: A collaborative project involving the University of Queensland, Griffith University and Queensland University of Technology. 30 August 2004. van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, N. Nieveen, R.M. Branch, K.L. Gustafson, & T. Plomp, (Eds.), Design methodology and developmental research in education and training (pp. 1- 14). The Netherlands: Kluwer. Weedon, E., Bricheno, P. & Chidwick, N. (2004). The Impact of Large-scale Network Learning on Further and Higher Education Institutions. JISC report, December 2004. Retrieved July 12, 2007, from http://www.sfeuprojects.org.uk/inlei/project_summary.html Wenger, E., McDermott, R. & Snyder, W. (2002). Cultivating communities of practice. Boston MA: Harvard Business School Press. Wetterling, J., & Collis, B. (2003). Sharing and reuse of learning resources across a transnational network. In A. Littlejohn (Ed.), Reusing online resources. London: Kogan Page. Wiley, D.A. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D.A. Wiley (Ed.), The Instructional Use of Learning Objects. Retrieved March, 26, 2007, from http://reusability.org/read/chapters/wiley.doc. Worthen, B., Sanders, J. & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd Ed.). New York: Longman. 95 Appendices ______________________________________________ Appendix A: Glossary of terms Term Definition for the purposes of this research Commentary Informally contributed feedback or narrative. This may be contributed, e.g. to an online forum as part of a discussion on issues around teaching, or as part of an assessment of a resource within the Carrick Exchange, contributing to informal peer review processes. Networks, communities, and Communities of Practice (CoP) Network: A system of interconnected people (Macquarie Dictionary) Contribution Learning object (LO) Community: A group of people with a common background or shared interest. Community of Practice: A more specific term than community. “A group of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise . . . by interacting on an ongoing basis” (Wenger et al. 2002, p.4). CoPs are characterised by a domain of knowledge, a community and shared practice. Contribution to the Carrick Exchange includes: a) digital artefacts (resources such as learning objects, teaching support materials, curriculum outlines, project reports, etc.); or b) a communication of some kind, such as a contribution to a discussion or a blog, or as a comment on another resource in the Carrick Exchange. The term is often used very broadly, but also has a very specific meaning. Some of the various definitions are as follows. The broad definition given by Wiley is probably the version most relevant to the Carrick Exchange context. “Any digital resource that can be reused to support learning.” (Wiley 2000, p.7) “An aggregation of one or more digital assets, incorporating metadata, which represents an educationally meaningful, stand-alone unit” (Dalziel, 2002) “Any entity, digital or non-digital, which can be used, re-used or referenced during technology supported learning” (The Learning Technology Standards Committee, IEEE definition, 2002) Learning object repository (LOR) A database of learning objects. Often used broadly to refer to any digital collection of learning and teaching resources. The Carrick Exchange can be regarded as a community of those who lead, manage and teach in higher education. The community is supported by a LOR. Peer review The evaluation of teaching resources and commentary contributed to the Carrick Exchange. Review is conducted by “peers” or equals. It may be: a) formal and conducted by teams of experts; b) for quality assurance purposes only: or c) informal and include discussion, feedback and comments made on resources and ideas contributed to the Carrick Exchange. 96 Term Definition for the purposes of this research Resources A broad term for any digital artefact relating to teaching and learning that can be stored electronically, contributed and shared through the Carrick Exchange. For the purposes of this research, the words “resources” and “materials” have been used interchangeably. a) It may, e.g. include formal and informal items, products, processes, learning objects, learning designs, activities, assessments, reviews, criteria, reports, plans. b) In this context resources also include communications – comments made about items contributed to the Carrick Exchange, or discussion sessions. Once contributed this “commentary” becomes “resources” for further comment or development by others. c) People can also be considered to be important resources for the Carrick Exchange, as they share, network, provide expertise, peer review and comment on matters relating to teaching and learning. 97 Appendix B: Repositories, associations and projects referred to in the report Repository, association or project ACELL Advancing Chemistry by Enhancing Learning in the Laboratory Description ACELL is a peer reviewed database of resources for undergraduate chemistry. ACELL aims to provide educationally sound chemistry experiments, evaluated by students and academic staff; professional development opportunities for chemistry academic staff; and facilitate the development of a community of practice in chemistry education. http://acell.chem.usyd.edu.au/homepage.cfm AESharenet AEShareNet's role is to help create a comprehensive, efficient system that is needed to streamline reciprocal copyright licensing practices in education. http://www.aesharenet.com.au/ ANDS ANDS has outreach services which bring in data that needs to be archived for the nation; it has stewardship services, metadata, and federation services. the Australian National Data Service APSR Australian Partnership for Sustainable Repositories ARROW Australian Research Repositories Online to the World APSR aims to establish a centre of excellence for the management of scholarly assets in digital format. The partnership includes major research universities, the National Library of Australia, and APAC (the Australian Partnership for Advanced Computing). http://www.apsr.edu.au/ The ARROW project identifies and tests software or solutions to support best practice institutional digital repositories comprising e-prints, electronic theses, e-research and electronic publishing. The project is funded by the Australian Commonwealth Department of Education, Science and Training, under the Research Information Infrastructure Framework for Australian Higher Education. http://www.arrow.edu.au/ The Australasian Society for Computers in Learning in Tertiary Education A society for those involved in tertiary computer-based education and training, including educational interactive multimedia. It provides a forum to stimulate discussion in the educational use of technology as well as promoting research and evaluation. http://www.ascilite.org.au/ Australian Access Federation Project National infrastructure and authentication project. http://www.aaf.edu.au/) Australasian Association for Engineering Education AAEE is a professional association committed to fostering excellence and innovation in engineering education, bringing together people across Australia and New Zealand. http://www.aaee.com.au/ ascilite AAEE (A2E2) Australian Flexible Learning Framework The Australian Flexible Learning Framework provides the vocational education and training (VET) system with e-learning skills, professional development opportunities, products, resources and support networks to meet today's increasingly technology-driven learning environment. The Framework is a national strategy collaboratively funded by the Australian Government and all states and territories. http://www.flexiblelearning.net.au/flx/go/home 98 Repository, association or project Description Carrick Exchange The Carrick Exchange is an online service created by the Carrick Institute of Australia that will provide learning and teaching resources and support communication and collaboration across the higher education sector in Australia and internationally. It will contain collaborative workspaces, tools and communication facilities. http://www.carrickexchange.edu.au/index.html Carrick Institute The mission of the Carrick Institute for Learning and Teaching in Higher Education is to promote and advance learning and teaching in Australian higher education. http://www.carrickinstitute.edu.au/ CD-LOR The CD-LOR project in the United Kingdom aims to identify and analyse the factors that influence practical uptake and implementation of learning object repositories within a range of different learning communities. The aim is to benefit UK higher and further education. http://www.academy.gcal.ac.uk/cd-lor/ Community Dimensions of Learning Object Repositories CLOE the Co-operative Learning Object Exchange A collaboration between Ontario universities and colleges in Canada for the development, sharing, and reuse of multimedia-rich learning resources. This occurs through the CLOE Learning Object repository. http://cloe.on.ca/index.html Cornell University Physics eprint archive Cornell University Physics eprint archive. http://www.arxiv.org Council for the Humanities, Arts and Social Sciences (CHASS) As a peak body for the humanities arts and social sciences, CHASS aims to represent the interests of the sector; to promote the contribution of the sector to government, industry and the public; to provide a forum for discussion between the humanities, arts and social sciences sectors in Australia; and to build the innovative capacity of Australia, through better linkages between the sector and industry, science and technology. http://www.chass.org.au/ DIDET The DIDET Project is led by the University of Strathclyde, Stanford University and Olin College. The aim is to enhance learning through students’ participation in global team-based design engineering projects. http://www.didet.ac.uk/ Edna Free online network for educators housing online resources and offering a collaborative network for the education and training community. http://www.edna.edu.au/edna/go Education Network Australia HERDSA, Higher Education Research and Development Society of Australasia Scholarly society for improvement of teaching and learning in higher and tertiary education. http://www.herdsa.org.au/ Intute The Intute service is created by a network of UK universities and partners. Subject specialists select and evaluate websites in the database and write high quality descriptions of the resources. http://www.intute.ac.uk/ LORN project Part of the Australian Flexible Learning Framework, LORN is an initiative supporting the Australian VET community, providing access to high-quality learning and teaching resources across a number of repositories. http://lorn.flexiblelearning.net.au/lorn/go/home/pid/119 Learning Object Repository Network 99 Repository, association or project MAMS Meta Access Management Project MERLOT Multimedia Educational Resource for Learning and Online Teaching Description The MAMS project is concerned with the infrastructure and integration of multiple solutions to managing authentication, authorisation and identities across systems, together with common services for digital rights, search services and metadata management. http://www.melcoe.mq.edu.au/projects/MAMS/ A user-centred, searchable collection of peer reviewed, higher education, online learning materials created by registered members, and a set of faculty development support services. where learning materials and pedagogy are shared by the members. http://www.merlot.org/merlot/index.htm Minister of Communications Digital Strategy Advisory Group (NZ) The NZ Digital Strategy is a government action plan for ensuring New Zealand is a world leader in using information and technology. http://www.digitalstrategy.govt.nz/ MIT open courseware model Massachusetts Institute of Technology http://ocw.mit.edu/index.html NCRIS Through NCRIS, the Australian Government provides funding to researchers for major research facilities, supporting infrastructure and networks necessary for world-class research. http://www.ncris.dest.gov.au/ National Collaborative Research Infrastructure Strategy OAK Law Project Open Access to Knowledge ODLAA Open and Distance Learning Association of Australia RRBIR Rights and Rewards in Blended Institutional Repositories RUBRIC Regional Universities Building Research Infrastructure Collaboratively SNUSE, Sydney Basin Network of University Science Educators The project aims to make sharing knowledge across domains, and the world, both legal and efficient. The project will develop legal protocols for managing copyright issues in an open access environment. At a technical level it will investigate provision and implementation of a rights expression language. The project will integrate with existing open access repositories at both legal and technical levels. http://www.oaklaw.qut.edu.au/about ODLAA is a professional association that aims to advance the practice and study of distance education in Australia; foster communication between distance educators; and maintain and extend links with other national and international associations with related aims and objectives. http://odlaa.une.edu.au/ University of Loughborough institutional repository, UK http://rightsandrewards.lboro.ac.uk/index.php?section=1 This DEST funded project aims to: build capability across smaller research universities in the IRUA group and country areas; enable the research output of those institutions to be available trans-nationally; and contribute to the research mission of higher education in Australia and internationally through collaboration with New Zealand partners. Partners are: the University of Southern Queensland (USQ), the University of New England, the University of the Sunshine Coast, the University of Newcastle, and Massey University in New Zealand. http://www.rubric.edu.au/ Network of Science Educators http://www.science.unsw.edu.au/guide/slatig/snuse.html 100 Appendix C: Key practitioners interviewed in Cycle 1 Category Stakeholders Anticipated contribution to project Project focus area Comparisons with international initiatives Models of use and peer review Comparisons with national initiatives Resource identification and contribution Comparisons with national initiatives Models of use, peer review Institutional repositories perspective Peer review, models of use, resource contribution MERLOT (USA) Intute (UK) CLOE (Canada) International connections (6) MOD4L Project (UK) NZ E-learning Collaborative Development Fund & Open Educational Resources (NZ) EduForge (NZ) Digital Repository Team, Griffith University National initiatives (3) RUBRIC project (regional initiative) MAMS – Meta Access Management Systems National Research and Policy Advice Project Australian Flexible Learning Framework Technical and Further Education (TAFE ) sector (4) Learning Object Repository Network Project (LORN) - Australian Flexible Learning Framework Teaching and Learning Exchange TAFE (general) Rights and Rewards in Blended Institutional Repositories (UK) Institutional repositories (2) Deakin University RMIT University Associations (discipline and professionally based) (10) Open and Distance Learning Association of Australia (ODLAA) Council for the Humanities, Arts and Social Sciences (CHASS) Association’s needs for communicating and resource sharing Resource identification and contribution, peer review, models of use, engagement Sydney-Basin Network of University Science Educators (SNUSE) Australian Council of Deans of Science Australian Council of University Art and Design Schools Engineering Australia Australian Mathematical Society 101 Category Stakeholders Anticipated contribution to project Project focus area Higher Education Research & Development Association (HERDSA) Australian Council of Deans of Science Council of Australasian Directors of Academic Development User perspectives on the development, utilisation and review of digital repositories Resource identification and contribution, models of use, engagement Deputy/Pro Vice-Chancellor (1) User perspectives on the development, utilisation and review of digital repositories Models of use, peer review Carrick grant holders and/or Carrick Fellows (1) Project needs for communication and resource sharing Resource identification and contribution Australian Tertiary Institutional Users (educational developers) (2) University of Queensland University of Wollongong 102 Appendix D: Focus group participants – Cycle 2 Universities represented and participants’ details Universities represented (22) Melbourne (12) Brisbane (4) Sydney (6) Deakin University, Vic Southern Cross University, NSW Macquarie University, NSW La Trobe University, Vic Central Queensland University, QLD The University of Sydney, NSW Monash University, Vic RMIT University, Vic The University of Melbourne, Vic Griffith University, QLD University of New South Wales, NSW University of Southern Queensland, QLD University of Technology, Sydney, NSW Victoria University, Vic Flinders University, SA University of Western Sydney, NSW The University of Adelaide, SA University of Wollongong, NSW Curtin University of Technology, WA Edith Cowan University, WA Murdoch University, WA The University of WA, WA Participants’ details (Note: some participants represented more than one category) Frequency Staff Development 8 Members of cross-institutional teams (e.g. Carrick Institute or other collaborative project) 7 Educational Development 6 Librarians 5 Repository representatives 3 ‘Early adopter’ in the use of technology 2 Academics interested in learning and teaching but with limited experience of technology 2 Females 13 Males 9 103 Appendix E: Engagement table specific example – Carrick Grant Holders The following table has been adapted from Dormant’s 18 (1997, p144) stages of awareness framework. This example has been developed to illustrate how one specific user group (Carrick Grant Holders) might be engaged with the Carrick Exchange, depending on their level of awareness. Stage of awareness General strategy Carrick Exchange strategy Awareness Advertise Advertise • Passive regarding the change • Little/no information about change • Little/no opinion about change Be an advertising agent Provide information prior to and throughout the grant application process. Provide information on the website, email & phone details for 1st contact Appeal to his or her needs and wants Support roles & facilitating personnel Carrick Institute Grant Scheme coordinators, Provide reasons for need for the CE and benefits such as: CE Managers & Champions, a) central place for communication, such as project meetings; CE promotional staff b) project content management, within private space; make publicly available progress of grants & outcomes; Local Institution Champions c) broaden network - point of contact for other projects. Be credible and positive Curiosity Inform Inform • More active regarding change Identify specific concerns Identifying what CE offers grant holders such as public dissemination space for projects, private and public project workspaces, shared resources, links to literature and other previous and current related projects (either Carrick or not), and networks of experts. Provide clear information about concerns Provide information as website documentation and at short f2f sessions or as part of other Carrick Institute related activities on receipt of grant (include in grant criteria). Emphasise pluses, acknowledge minuses Have evidence of national & international benefits for using the CE. Local Carrick Grant coordinators Have evidence of a quality database of professional development & teaching resources derived from other Carrick Institute projects Members of the Grant team • Express personal job concerns • Asks questions about own work and change CE discipline-based Champions & collections managers Carrick Institute Grant Scheme coordinators, 18 Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan, and P. MacGillis (eds.), The guidebook for performance improvement: working with individuals and organizations. San Francisco: Pfeiffer. 104 Stage of awareness General strategy Carrick Exchange strategy Support roles & facilitating personnel Envisioning Demonstrate Demonstrate • Active regarding change Give success images Develop training materials accessible from the CE. CE & Local institutional Champions, Provide demonstrations Provide 1-2 hour f2f demonstrations (f2f and/or online). Staff & Educational Developers • Expresses workrelated job concerns Provide digital stories, showcasing & case studies on the CE demonstrating use of the CE by other grant holder. • Asks questions about how change works Present a template or exemplar about how groups could engage with the system for public and private spaces. Connect with peer users Grant holders and others with experience in the use of the CE Use ‘push’ technology, customisable to individuals’ needs, to connect groups and members (e.g. email, RSS feeds, newsletters and alerts). Identify related project material, other experts, links to the literature & related projects, previous workshop material. Encourage registration so as to view activities of available communities & networks. Tryout Train Train • Active regarding change Provide effective training Support & provide local f2f training initiatives CE Trainers Offer training in peer review and evaluation. Help desk supporters • Has opinions about change • Interested in learning how-to Provide job aids, checklists Promise technical follow-up Phone support and ‘just-in-time’ mentoring for establishing project space and using the tools. Provide documentation, fact sheets, Frequently Asked Questions (FAQs), guidelines, tutorial animations contact details of institutional Champions. Librarians Educational Developers Technical Writers Institutional Champions Provide forums on CE for getting started, for Q&A, pedagogical and technical Provide a database of members available to support others in specialist areas 105 Stage of awareness General strategy Carrick Exchange strategy Use Support Support • Active regarding change Provide necessary technical help Develop documentation and resources to support engagement. • Uses change on the job • Asks detailed questions about use Provide pedagogical help Provide help desk and centralised support for engagement. Provide support for embedding reuse and sharing into the curriculum e.g. activities, forums, conferences and guidance at the personal and group level For ease of use provide support clearing copyright, metadata entry and checking, rights management and version control Modelling good practice in the use of communication and collaboration tools (through hosting and facilitating exchange). Engage people f2f by conducting conferences and workshops which focus on resource identification & submission, and training to undertake peer review of submitted material. Provide recognition Provide reinforcement Support roles & facilitating personnel CE and institutional information specialists Educational and staff developers, Carrick and local Champions, CE community members Librarians, copyright managers Carrick Institute managers, Institutional managers Provide recognition and awards for contribution (e.g. commentary and participation) and excellence which support career advancement; link with promotional policies and procedures, e.g. certificates, Carrick awards. Source: Adapted from Dormant 1997; Key: CE = Carrick Exchange May need different strategies for individuals and groups 106 Appendix F: Roles identified for the Carrick Exchange A number of roles have been identified in the ascilite research. Some are identified below. Role Focus Activity Carrick Exchange Manager Overview Promote and manage the Carrick Exchange so that it continues to grow the collection, providing relevant, accurate and sustainable resources (human and digital). Carrick Exchange Champion Engagement Promotion of the Carrick Exchange amongst the academic community. Engages with local Carrick Exchange Champions, research and education focus. High level engagement at the national level with Executive in each university. Contributes to the annual conferences on Peer Review and Resource Identification and Contribution. Institutional Champions Engagement Located at each Australian university. Promotion and education about the Carrick Exchange at the institutional level. Liaison with the Carrick Exchange Champions and staff within own institution. Technical Writer Engagement Generate promotional and informational material for the website. Suggest various media for presentation. Work with discipline specialist and Carrick Exchange Manager. Contributions Editor (digital resources) Resource contribution and identification Sort and select or reject digital resources submitted to the Carrick Exchange for deposit. Send selected items for the quality assurance process and/or peer review. Liaise with authors and contributors. Promote contribution to the Carrick Exchange and identification of resources. Contributes to the annual conferences on Resource Identification and Contribution. Contributions Editor (informal commentary) Resource contribution and identification Monitor specified, but not all, discussions. Mediate and monitor disputes. Generate activities on certain forums that contribute to engagement. Support and encourage group participation in the private workspaces. Promote contribution to the Carrick Exchange and identification of resources (human and digital). Contributes to the annual conferences on Resource Identification and Contribution. Collections Manager Resource contribution and identification Probably discipline based, but may be generalist in early stages of the Carrick Exchange. Identification of resources to maintain and sustain the collection. Promote contribution to the Carrick Exchange. Contributes to the annual conferences on Resource Identification and Contribution. Quality Assurance Manager and/or Administrative Officer Peer Review and commentary Manages process of quality assurance (technical, copyright & basic educational integrity checks). Liaison with Peer Review Chair. Contributes to the annual conferences on Resource Identification and Contribution. Copyright officer Resource contribution and identification Manages copyright checks on deposited items, promotes understanding about digital rights issues, contributes to forums on digital rights, intellectual property, moral rights etc. Contributes to the annual conferences on Resource Identification and Contribution. Liaises with individuals and institutional representatives on copyright and licensing matters. 107 Role Focus Activity Editorial Board Members Peer Review and commentary Members provide guidance and direction for the peer review processes and about the collection and its integrity. Contribute to the annual conference which supports training in peer review and deliberation on the issues of peer review of technology based educational resources. Peer Review Editorial Board Chair Peer Review and commentary Manage and lead the Editorial Board’s activities, supervising and overseeing the peer review processes, inviting reviewers to participating and selecting reviewers as required. Contributes to the annual conferences on Peer Review. Reviewers for formal peer review Peer Review and commentary Specialists in discipline areas, education, or media/information who comprise the review panel for the purpose of evaluating resources accepted for the peer review process. These positions may be a voluntary unless the amount of work becomes sufficient to warrant payment for regular work. Each review panel may consist of a different set of members (other than the Editor), according to the nature of the resource to be peer reviewed. Staff and Educational Developers/Designers and Librarians Engagement, Resource contribution and identification, and University based. Specialists in curriculum, educational technology, professional development and information management located within institutions who have a key role in promoting the Carrick Exchange to other staff within their institutions. Conduct workshops and work one-to-one with interested personnel. Liaison with institutional Carrick Exchange Champions. Peer Review and commentary Technical Helpdesk Training Support Officer Provides just-in-time support for personnel using the Carrick Exchange, using email, chat, and phone. May also conduct web conferencing training sessions. 108 Appendix G: Examples of formal peer review processes for learning and teaching resources CLOE: Cooperative Learning Object Exchange, http://cloe.on.ca/ About: CLOE is a collaboration between Ontario universities and colleges for the development, sharing, and reuse of multimedia-rich learning resources through the CLOE Learning Object repository. Each institution develops multimedia learning resources to address instructional challenges shared by the other partners, or uses/adapts learning objects created at another partner institution to use at their own institution. Considerations: Peer review process for learning objects submitted to CLOE has the following characteristics- Clear (consistent with author guidelines); Recognised (being a reviewer is institutionally valued); Automated (neither onerous nor a disincentive for reviewer or reviewee; automation of certain tasks should occur); Familiar (like other forms of scholarly peer review in the disciplines); Thorough (capacity to evaluate the form and the content of a learning object). Evaluation Standards: 1. Quality of Content: The content must be professional, clear and accurate. The use of technology must be appropriate for the content and the learning object should be provided with academic references and credits to the creators. 2. Effectiveness as a teaching/learning tool: The learning object should be able to function as a learning tool in different types of learning environments. The pre-requisite knowledge or skills have to be identified and the author must provide evidence that the learning object enhances student learning. The learning object must have clear learning objectives and the target learners must be identified. There must be clear instructions on how to use the learning model. It must also provide an opportunity for students to get feedback within or outside the learning object. 3. Ease of use: The learning object must have easy to use navigation tools and user control. The author must indicate if the learning object can be accessed by learners with various needs. Technical requirements to run the learning object must be stated. Protocols: The CLOE peer review process requires the involvement of two kinds of reviewers: instructional design experts and subject matter experts. The CLOE peer review process involves an instructional designer and two subject matter experts. Instructional design experts evaluate the degree to which the LO is likely to meet its instructional goals as described by the author(s) and defined in the CLOE Guidelines for Authors. Subject matter experts are responsible for examining the LO for the validity and quality of the content, for factual information and for the overall contribution of the LO towards student learning. See Appendix 1 & 2 for A model of the CLOE Peer Review process and the Peer Review evaluation form. EducaNext: http://www.educanext.org/ubp About: EducaNext is a service supporting the creation and sharing of knowledge for Higher Education. It is open to any member of the academic or research community. Considerations: EducaNext co-ordinates peer review of online content on two levels – formal and informal. The formal peer review is conducted in a classic way by inviting subject matter experts to assess learning resources in terms of three major aspects: content, usability and instructional design. The informal peer review is conducted with the users of learning resources, be they faculty members or students. They are required to describe their perceptions and experiences of a learning resource right after they have used it. Their descriptions are based on a tripartite inquiry on the technical, pedagogical, and organisational aspects. Protocols: Large-scale resource entities like complete online courses undergo a more comprehensive review process. A team of evaluators with different backgrounds, including experienced instructional designers, media designers, domain experts, programmers, and end users, are invited to take part in a forum to analyse and discuss the strengths and weaknesses of online courses. EducaNext Catalogue Managers are responsible for screening the quality of metadata descriptions of learning resources. In their role of catalogue administrators they are able to remove inappropriate descriptions. The Health Education Assets Library (HEAL): http://www.healcentral.org/index.jsp 109 About: HEAL is a free digital resource repository designed to meet the needs of health sciences educators. HEAL promotes the preservation and exchange of useful educational assets while respecting ownership and privacy. Considerations: All items submitted by authors to HEAL undergo the peer review process HEAL employs a rigorous review process, which follows the traditional scholarly model of peer review. Two unbiased reviewers who are content experts and/or experienced users of instructional technology systematically examine the quality and usefulness of the submitted resource use a standardised instrument to rate each item in terms of a set of review criteria, including: • • • • Preparation and content quality Effectiveness of the material Presentation and ease of use Significance Protocols: Following standard academic practice, resources are either rejected, accepted conditionally, or accepted as-is. Reviewers can also accept outstanding resources “with acclamation.” In addition, reviewers are specifically asked to rate both the quantity and quality of descriptive information (i.e., “metadata”) that authors provide. Reviewers are faculty at accredited institutions of higher education that have expertise in one of the basic or clinical sciences. Notably, they are “peer users” of instructional technology (not necessarily “peer developers”) and have demonstrated excellence in teaching. The resources are not blinded for review and reviewer comments are made available to the author after the review. Upon acceptance, resources are permanently published in the HEAL Reviewed Collection. Authors receive a certificate of publication describing the resource, the review process, and date of acceptance. Resources can be submitted for peer review using an online Submissions Form (login for this is required). Intute: http://www.intute.ac.uk/policy.html About: Intute is a free online service providing access to the very best web resources for education and research. All material is evaluated and selected by a network of subject specialists to create the Intute database Considerations: Intute aims to inform both content users and content suppliers about the nature, extent, and accessibility of the collections. The Intute Collections and Cataloguing Management (CCM) Working Group consist of staff from across the Intute service with experience in collections management and/or cataloguing. A central and over-arching collections policy for the Intute service provides each Intute Subject Group with a basis for their subject-specific collection policy. Intute employs a network of subject specialists to find, evaluate and catalogue the best of the Web. The Intute collection is selective and only online resources that meet the agreed quality selection criteria are included. The selection criteria are outlined in Appendix 6 JIME – Journal of Interactive Media Education http://jime.open.ac.uk/index.html About: JIME is a publication that does not have a chronological concept of "issue". Instead, JIME publishes articles for open peer review as they are received. Final versions of articles are published as soon as they complete the Review Process Considerations: In JIME's review environment: 1. 2. 3. Authors have the right of reply. Reviewers are named, accountable for their comments, and their contribution acknowledged. The wider research community has the chance to shape a submission before publication. Submissions will be critiqued by many more reviewers than is currently possible in the conventional review process, and this will take place in a public rather than private forum. While JIME prefers all comments to be signed, anonymous contributions are also permitted. Willingness of both authors and reviewers to engage in this process underpins its success. 110 Protocols The review process for JIME submissions provided in Appendix 5. This review model has three stages: preprint under private, open peer review, preprint under public, open peer review, and publication. These are explained below. Private open peer review... Articles submitted to JIME are first reviewed by three reviewers who are named, and acknowledged for their contribution to a review. They post their reviews as threaded comments to a private site. Reviewers have the option of posting anonymously, but usually reviewers are happy to be named, and in JIME's conversational review model, it helps to know to whom you are talking, and hence, how better to interpret comments. Authors are encouraged to respond to these comments and reviewers in turn (who may not necessarily agree with each other). This takes place during an agreed period when authors and reviewers are able to respond. Publication as a preprint for public, open peer review... On the basis of the discussion, if the editor assigned to the submission judges it to be of sufficient quality -- that is, broadly acceptable, pending changes based on the review discussion -- the submission will then be published as a preprint for public open peer review, and announced to relevant communities to invite their participation. The author-reviewer discussion provides the 'seed' for this second phase of online review debate. This phase of open review will be closed after one month. The editor will post to the discussion an editorial report summarising the most significant issues, and specifying change requirements to the authors. Following publication... In conventional journals, the point of publication is the beginning of scholarly debate. JIME brings this point forward by making submitted preprints accessible, and continues to support discussion about the revised, published article. In addition, the most interesting review comments/exchanges are published with the final version, providing readers with insight into the issues that arose during review, and enabling them to build on those discussions. Thus, authors can post links to publications to point to subsequent work. Readers can post comments and links to point to work which has not been referenced, or did not exist when the article was written. Authors, reviewers and anyone else who has subscribed to the article will receive email alerts to new postings to its discussion forum. The final publication will be freely accessible on the JIME site. Jorum: http://www.jorum.ac.uk About: Jorum is a JISC-funded collaborative venture in UK Higher and Further Education to collect and share learning and teaching materials, allowing their reuse and repurposing, and standing as a national statement of the importance of creating interoperable, sustainable materials. It is run jointly between the EDINA and MIMAS national data centres. Protocols: Metadata: Contributors are required to complete a few basic metadata fields available in the system. The software system in addition creates some metadata automatically. To provide full quality assurance of metadata, a team of information specialists from the JISC Intute service complete the remainder of the fields for all objects deposited, apart from learning assets such as images, where completion of full educational metadata would be difficult. Technical: A Collection Procedures policy provides the details of certain checks that contributors are required to undertake. The Jorum team reserves the right to withdraw materials from the repository after publication for discussion with contributors, if they receive alerts or queries from other users unable to play their materials and technical checks reveal problems with the materials. The Jorum team check around 10% of materials as they are added to the repository and Intute cataloguers check that all materials catalogued by them run correctly when they add metadata. In addition, project or departmental teams can request that another stage in the Jorum workflow be added for their group, to enable a pedagogical review stage within their own group. The LAMS Community: http://www.lamscommunity.org About: The LAMS Community provides an environment for using and sharing digital lessons under open content licenses referred to as sequences. Considerations: There is an informal peer review process in place, a rating feature for each sequence – including the number of times the sequence has been downloaded, and comments to the author from users 111 MERLOT: http://www.merlot.org About: MERLOT is a user-centred, searchable collection of peer reviewed, higher education, online learning materials created by registered members, and a set of faculty development support services. Considerations: Faculty selected in accordance with MERLOT guidelines perform the peer review of MERLOT learning resources. The peer review process is led by Editors and an Editorial Board. Protocols: The Peer Review Process follows the model of peer review of scholarship, and includes: 1. − − − − Developing Evaluation Standards The MERLOT Editorial Boards provide leadership, tools, and training in developing evaluation standards and processes. The MERLOT Administrative Team provides the Editorial Boards with a framework of evaluation criteria which is based on: Quality of Content; Potential Effectiveness as a Teaching Tool and Ease of Use All Peer Reviewers on each Discipline-specific Editorial Boards share and compare their evaluations following the processes developed and the framework provided to create test cases. These test cases are then used to develop evaluation guidelines/criteria that are applied to all materials in the discipline. Each Editorial Board establishes substantial inter-rater reliability in its evaluations before evaluation procedures are implemented on the remaining materials in its discipline. 2. Conducting Evaluations: − Editorial Boards decide on the process for selecting materials to be peer reviewed. If a collection of materials is not large enough to adequately establish inter-rater reliability, its Editorial Board works to expand the collection sufficiently. Review teams typically use the following two stage review process: − − Stage 1 Cursory Review to Identify Worthy Candidates: • • • • The Editorial Board reviews its collection and "triages" materials as follows: 1. Definitely worth reviewing 2. Possibly worth reviewing 3. Not worth reviewing at this time Peer Reviewers and Associate Editors report their cursory evaluation to their Editorial Boards. Peer Reviewers can post Member Comments based on cursory evaluations as appropriate. The Editorial Board compiles its list of worthy materials. Those deemed "definitely worth reviewing" receive top priority in the review process. Stage 2 Intensive Review of Worthy Candidates: • • • • The Editorial Board assigns "worthy" materials to Editorial Board Members. In some cases, the Editorial Board asks the author(s) of the material(s) for permission to review. Two Peer Reviewers each use their Editorial Board's review procedures, forms, and evaluation standards as they independently review the material. Reviewers write peer review reports using the evaluation criteria for MERLOT learning materials as a guideline and publish these individual reviews in their Editorial Board Workspace. If there is any significant disparity in the two reviews, an Editor or Associate Editor assigns the material to a third reviewer. 3. Reporting Evaluations: • − − − − The Editor or Associate Editor reviews both individual reviews and creates an integrated or Composite Peer Review Report. The Editorial Board sends the Composite Peer Review Report to the author(s) for feedback and permission to post the review on MERLOT. The Editorial Board posts the Composite Peer Review Report on the MERLOT website. Authors may ask the Editor to send two letters, to two individuals of their choice, summarising the peer review process and including the Composite Peer Review Report. 112 Appendix H: Peer review criteria and standards When a resource is submitted it will undergo an initial functionality review. After passing this first level of functionality testing, 19 the resource is reviewed on the following criteria: N.B. Normally a rating of “not at all” on any question by the reviewers will require that the author provide additional information or revision of the resource before it is accepted. Resource and Scope 20 : Reviewer: Quality of Content 1. The content of the resource is accurate and up-to-date 2. The use of technology is appropriate for this content 3. The content is presented clearly and professionally (spelling, grammar, etc.) 4. Appropriate academic references, citations and/or links are provided 5. Credits to creators are provided 6. Multimedia resource quality is good 7. Documentation provides sufficient information to guide users in using the resource, including suggestions for adaptations or extensions Not at all Somewhat Definitely Effectiveness as a Teaching/Learning Tool 8. There are clear learning objectives and/or statement of purpose 9. The resource meets the stated learning objectives/purpose 10. The target learners are clearly identified (academic level addressed/technical ability/demographics) 11. There are clear instructions for using the resource 12. The technology helps learners to engage effectively with the concept/skill/idea 13. The resource provides an opportunity for learners to obtain feedback within or outside the resource 14. The author provides evidence that the resource enhances student learning 21 15. Pre-requisite knowledge/skills, if needed, are identified 16. The resource stands alone and could be used in other learning environments 19 Initial functionality testing will be conducted by the CE gatekeeper and will include checking to ensure that links work, plug-ins are available, platform and browser compatibility are identified, the resource is useful, adheres to copyright restrictions and adequate metadata has been provided, et cetera. 20 Scope of resource may be suitable for one learning session, several learning sessions, entire length of program/course, a full program of study, or other as specified 21 Acceptable evidence could be anecdotal comments, student feedback questionnaires, or more formal learning impact studies. 113 Quality of Content Not at all Somewhat Definitely Significance 17. Resource offers an innovation in learning and teaching 18. Resource contributes to the field Ease of Use/Usability 19. The resource is easy to use (i.e. navigation, user control) 20. The author indicates whether the resource is accessible for learners with diverse needs 21. Technical requirements for the resource are provided 22. Application loads, launches and executes smoothly (appears to have no bugs) 23. Terms and conditions of use are specified 24. The design of resource (or part of the resource) is scalable 25. The costs of implementation in other contexts are identified Overall rating of this resource: π π π π Does not meet minimum standards – REJECT Meets minimum standards, but there are concerns – ACCEPT WITH REVISIONS Meets quality standard – ACCEPT Excellent overall – ACCEPT WITH ACCLAMATION Additional Comments: Strengths of this resource: Modifications required for acceptance: Suggestions for future development: 114 Appendix I: Protocols for peer review (Based on the HEAL peer review policy) The protocols below detail a review process that follows from the technical check to be completed by Carrick Exchange staff to confirm the functionality and integrity of the contribution I.e. links work, plug-ins available and operable, platform compatibility, adheres to copyright restrictions, adequate metadata provided etc. 1. Objectives of Review Process (Formal and Informal) • • • • Assure the quality of CE collection Systematise the appraisal of resource contributions (including experts) Provide a review process that is nationally recognised as transparent, fair, valid and reliable Foster the recognition of CE as a dynamic, useful and accessible collection. 2. Roles of Chair of review panel, Reviewers, Carrick Exchange Staff • Chair of review panel − − − − − − − − • Reviewers − − − − • Notify panel chair within three days if contribution not in area of expertise, or present a conflict of interest 22 Read guidelines prior to review activity Review materials according to guidelines prior to the deadline, using the proforma Forward review details to panel chair with a recommendation regarding acceptance or rejection, provide comments and suggested modifications Carrick Exchange Staff − − − − − − 22 Assume accountability for the peer review process Make final decisions to accept or reject contributions Identify three review panel members (subject specialist, educational specialist and media or collegial specialist) Convene review panel Notify panel members when contribution can be located for review Communicate with contributor/s regarding outcomes of review Communicate with Carrick Exchange staff Contribute to periodic evaluation of peer review process Acknowledge receipt of contribution (this may be automated and CE staff need only monitor the operations of the system) Contact contributor if submission appears to infringe copyright or is unacceptable for other reasons Contact contributor if additional metadata or further information is required prior to review Notify panel chair when contribution is ready for review and identify location for access Issue certificates of acknowledgement (this may also be an automated system requiring only a monitoring role) Maintain a review database that includes prospective panel chairs and reviewers Guidelines yet to be developed 115 3. Review Process and Procedures • Formal − − − − − − − − • Panel chairs and reviewers must have the following qualifications… 23 Prospective reviewers sign online up as members of CE and indicate their qualifications and area of expertise Carrick Exchange staff oversee the maintenance of the reviewers’ database Contributors may recommend reviewers Sets of items submitted for review will be regarded as a set. The review of the set however is based on the systematic examination of each individual item Material can be published previously in other outlets, and arrangements with partner organisations can be made for joint review and acceptance to collections The copyright of materials remains with the original copyright owner The Carrick exchange review process, including review criteria, and names of panel chairs and reviewers are displayed in the Carrick Exchange Informal − − − − − Informal commentary can be guided by prompts to users such as: Which category would you like to add your comment (a) friendly praise; (b) constrictive critique; (c) how did you do that?; (d) how have you used this?; and (e) how did it work for you? “Member spotlights” can allow display of a cumulation of informal acknowledgement Submission of comment can be acknowledged as “Thank you for submitting your comments. You are a true contributor to the Carrick Exchange” Contributor can further reply to the question “how useful was this comment to you?” 4. Review Criteria and Standards (refer to Appendix H) 23 Qualifications to be determined 116