University of Wollongong
Research Online
Academic Services Division - Papers
Academic Services Division
2007
ascilite Report 4 for the Carrick Exchange Project:
Final Report
R. Philip
D. Parrish
Geraldine E. Lefoe
University of Wollongong, glefoe@uow.edu.au
Meg O'Reilly
Southern Cross University
Recommended Citation
Philip, R.; Parrish, D.; Lefoe, Geraldine E.; and O'Reilly, Meg: ascilite Report 4 for the Carrick Exchange Project: Final Report 2007.
http://ro.uow.edu.au/asdpapers/81
Research Online is the open access institutional repository for the
University of Wollongong. For further information contact Manager
Repository Services: morgan@uow.edu.au.
ascilite Report 4 for the Carrick Exchange Project: Final Report
Abstract
Whilst the development of repositories for sharing learning and teaching resources has expanded
considerably in the last few years in Australia, many repositories have a particular institutional or discipline
focus. In Australia there is no dedicated national repository or community available for the higher education
sector that provides a cross disciplinary forum for sharing ideas and resources about learning and teaching,
and for cross-institutional collaboration. The Carrick Institute for Learning and Teaching in Higher
Education is well placed to address this gap by developing the Carrick Exchange (see Phillips, Orrell & Millea,
2007). Utilising the current expansion and availability of Web 2.0 services, the Carrick Exchange can support
the development of a national and international community to achieve these goals and meet the Institute’s
objectives. In particular the Carrick Exchange will provide an avenue to demonstrate the values of inclusivity
by “assisting the development of networks and communities which support higher education staff who have a
direct impact on the advancement of learning and teaching” and through the focus on long-term systemic
change (Carrick Institute website).The Carrick Exchange is seen by many in the sector as a major initiative
with the potential to significantly improve the profile of teaching and learning.
Disciplines
Education
Publication Details
This report was originally published as Philip, R, Parrish, D, Lefoe, G and O'Reilly, M, ascilite Report 4 for the
Carrick Exchange Project: Final Report, Wollongong, NSW, The Australasian Society for Computers in
Learning in Tertiary Education, 2007. Original report available here
This report is available at Research Online: http://ro.uow.edu.au/asdpapers/81
ascilite Report 4 for the Carrick Exchange Project:
Final Report
September 2007
Prepared by:
ascilite
The Australasian Society for Computers
in Learning in Tertiary Education
1
Document details
Document name
ascilite Report 4 for the Carrick Exchange Project:
Final Report
Authors
Robyn Philip, Dominique Parrish, Geraldine Lefoe, and Meg O’Reilly
For ascilite (The Australasian Society for Computers in Learning in
Tertiary Education)
File name
ascilite_Report 4 for CE_v1_0.doc
Other documents to be
read in conjunction
with this document
ascilite/Carrick Exchange Proposal and Project Plan 2007
ascilite Reports 2, 3 and 4 for the Carrick Exchange Project 2007
ascilite Resource Identification Network (RIN) Project, Stage 1 ascilite Component Think Tank Issues Paper, 2006
Version History
Date
04/10/07
Version
1.0
Comment
Final authorised by Geraldine Lefoe
Project team
Project Leaders
Dr Geraldine Lefoe, University of Wollongong
Meg O’Reilly, Southern Cross University
Project Manager
Dominique Parrish, Learning Achievements & Solutions
Research Manager
Robyn Philip, Consultant
Project Team
Dr Cathy Gunn, University of Auckland
Dr Mike Keppell, Hong Kong Institute of Education
Project Steering Group
(ascilite Executive)
Dr Joe Luca, Edith Cowan University,
Kar-Tin Lee, Queensland University of Technology,
Dr Barney Dalgarno, Charles Sturt University,
Dr Bob Corderoy, University of NSW,
Caroline Steel, University of Queensland
Allan Christie, Netspot Pty Ltd
International Adviser
Dr Allison Littlejohn, Glasgow-Caledonian University
Acknowledgement
The research team would like to thank all the interviewees, focus group participants, reference
group members and other advisors who took part in the research and willingly shared their
experience, ideas and time. We also wish to acknowledge the additional support provided by the
ascilite executive team, and the Carrick Exchange Project Team.
Support for this research has been provided by the Carrick Institute for Learning and Teaching in
Higher Education, an initiative of the Australian Government Department of Education, Science
and Training. The views expressed in this report do not necessarily reflect the views of The
Carrick Institute for Learning and Teaching in Higher Education.
2
TABLE OF CONTENTS
Key outcomes .......................................................................................................................... 5
Introduction.............................................................................................................................. 8
Background.............................................................................................................................. 8
Methodology .......................................................................................................................... 10
Aims and outcomes............................................................................................................... 10
The design-based methodology .......................................................................................... 11
Literature review .................................................................................................................... 16
Engagement guiding principles and recommendations ................................................... 23
Recommendations related to the purpose of the carrick exchange........................... 24
Recommendations related to multiple levels of engagement ..................................... 27
Recommendations relating to the management of engagement with the carrick
exchange ........................................................................................................................... 33
Recommendations related to ease of use ..................................................................... 38
Recommendations related to promoting engagement with the carrick exchange ... 40
Recommendations for engagement policies................................................................. 43
Resource identification and contribution guiding principles and recommendations ... 45
Principles related to resource identification ................................................................. 46
Principles related to addressing the culture around sharing in higher education.... 49
Principles related to rewards, incentives and recognition for contribution .............. 52
Principles related to intellectual property and digital rights management ................ 55
Principles related to management of resource contribution ....................................... 58
Recommendations for resource identification and contribution policies.................. 62
Peer review and commentary guiding principles and recommendations ....................... 65
Protocols and mechanisms related to the goals of peer review ................................. 67
Protocols and mechanisms related to the process and procedures for peer review 72
Protocols and mechanisms related to informal commentary...................................... 79
Protocols and mechanisms related to rewards and incentives for formal peer review
and informal commentary ............................................................................................... 82
Recommendations for peer review and commentary policies .................................... 84
Conclusion ............................................................................................................................. 86
Benefits................................................................................................................................... 86
Challenges.............................................................................................................................. 87
Future research...................................................................................................................... 89
References ............................................................................................................................. 91
Appendices ............................................................................................................................ 96
Appendix A: Glossary of terms....................................................................................... 96
Appendix B: Repositories, associations and projects referred to in the report........ 98
Appendix C: Key practitioners interviewed in cycle 1................................................ 101
Appendix D: Focus group participants – cycle 2 ........................................................ 103
Appendix E: Engagement table specific example – carrick grant holders .............. 104
Appendix F: Roles identified for the carrick exchange .............................................. 107
Appendix G: Examples of formal peer review processes for learning and teaching
resources ................................................................................................. 109
Appendix H: Peer review criteria and standards ........................................................ 113
Appendix I: Protocols for peer review.......................................................................... 115
3
Executive Summary
______________________________________________
Whilst the development of repositories for sharing learning and teaching resources has
expanded considerably in the last few years in Australia, many repositories have a particular
institutional or discipline focus. In Australia there is no dedicated national repository or
community available for the higher education sector that provides a cross disciplinary forum
for sharing ideas and resources about learning and teaching, and for cross-institutional
collaboration.
The Carrick Institute for Learning and Teaching in Higher Education is well placed to address
this gap by developing the Carrick Exchange (see Phillips, Orrell & Millea, 2007). Utilising the
current expansion and availability of Web 2.0 services, the Carrick Exchange can support the
development of a national and international community to achieve these goals and meet the
Institute’s objectives. In particular the Carrick Exchange will provide an avenue to
demonstrate the values of inclusivity by “assisting the development of networks and
communities which support higher education staff who have a direct impact on the
advancement of learning and teaching” and through the focus on long-term systemic change
(Carrick Institute website).The Carrick Exchange is seen by many in the sector as a major
initiative with the potential to significantly improve the profile of teaching and learning.
Most importantly, it is about connecting people with people . . . It is a space to
1
contribute, discuss and debate. (Carrick Exchange website)
ascilite, the Australasian Society for Computers in Learning in Tertiary Education, was
engaged to assist with the development of the Carrick Exchange (2006 – 2007) by conducting
research into:
•
•
•
Users’ needs within the higher education sector for collaborative and communication
spaces for learning and teaching;
The conditions and contexts of use of resources available for sharing and reuse within the
higher education sector; and
The policies necessary to facilitate engagement of the higher education sector with the
Carrick Exchange.
The challenge for the Carrick Institute in developing the Carrick Exchange is that the
literature, along with national and international practice, indicates that uptake of repositories
of learning materials has been slower than expected. Poor engagement by target users may
be related to lack of clarity about purpose and lack of strategies and resources directed to
fully engaging the community for which the repository has been designed. The ascilite
research has focussed on identifying the protocols, procedures and strategies that will be
required to address these issues. A design-based methodology was adopted using
interviews, focus and reference groups methods to gather data which was analysed in a
cyclical and iterative way along with the current literature. In this way ascilite was able to
document the experience, opinions and advice of the target users from the Australian higher
education sector. The findings will inform the next iteration of Carrick Exchange development.
1
http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381
4
Key outcomes
The findings derived from the ascilite research in Stage 2 are listed in terms of the
recommendations, principles, protocols and mechanisms.
User engagement
Recommendations related to the purpose of the Carrick Exchange
1.1.1. The purpose of the Carrick Exchange should be well articulated to the sector.
1.1.2. The quality of the processes, resources and functionality of the Carrick Exchange
should meet the expectations of the sector.
Recommendations related to multiple levels of engagement
1.2.1. A strategic plan for the engagement of the sector at multiple levels should be
developed and implemented.
1.2.2. Membership and the conditions of membership for the Carrick Exchange should be
specified.
1.2.3. The Carrick Exchange should meet the diversity of members’ needs and contexts of
use.
Recommendations relating to the management of engagement with the Carrick Exchange
1.3.1. The Carrick Exchange policies and practices should be aligned strategically with
institutional policies and practices.
1.3.2. The Carrick Exchange should engage with the National Authentication Framework to
ensure integration.
1.3.3. A strategy should be developed to support, encourage and manage champions to
promote the Carrick Exchange.
1.3.4. Partnerships with Australian universities should be established to ensure longevity
and sustainability of the Carrick Exchange.
1.3.5. Collaboration should be encouraged and supported as a means of fostering
engagement and contribution to the Carrick Exchange
1.3.6. The Carrick Exchange should support the needs of networks and communities of
practice.
1.3.7. An evaluation plan should be developed to measure levels of engagement including
participation, collaboration and contribution.
Recommendations related to ease of use
1.4.1. The Carrick Exchange should devise strategies of engagement for the varying levels
of technical capability of users.
1.4.2. The Carrick Exchange should be easily accessible and intuitive to use.
Recommendations related to promoting engagement with the Carrick Exchange
1.5.1. There should be a strategic plan and adequate support for the promotion and publicity
of the Carrick Exchange to ensure successful engagement by the sector.
1.5.2. There should be regular opportunities to disseminate and encourage engagement of
the higher education sector with the Carrick Exchange.
1.5.3. Web 2.0 technologies should be promoted and supported as a mechanism for
enhancing engagement with the Carrick Exchange.
5
Resource contribution
Principles related to resource identification
2.1.1. The Carrick Exchange should strategically target resource contributors.
2.1.2. Existing resources that could be contributed or linked to the Carrick Exchange should
be identified.
Principles related to addressing the culture around sharing in higher education
2.2.1. A strategic plan for establishing and maintaining institutional partnerships for sharing
between institutions and the Carrick Exchange should be developed.
2.2.2. The Carrick Exchange should systemically address the culture surrounding sharing in
higher education.
2.2.3. Networks and partnerships should be nurtured as a means of fostering contribution to
the Carrick Exchange.
Principles related to rewards, incentives and recognition for contribution
2.3.1. The Carrick Exchange should establish a system of rewards, incentives and
recognition that is valued, appropriate and manageable, and aligned to institutional
policies and practices.
2.3.2. The Carrick Institute should ensure that the established system of rewards, incentives
and recognition is adequately resourced and supported into the future.
2.3.3. A strategy for impact and process evaluation should be developed to assess the
relevance and value to the sector of the rewards, incentives and the system of
recognition.
Principles related to Intellectual Property and Digital Rights Management
2.4.1. The Carrick Exchange should establish IP and Rights Management policies and
practices that align to institutional policies and practices.
2.4.2. IP and Rights Management should address the concerns and needs of contributors
2.4.3. The Carrick Exchange should develop systems and structures to ensure that IP and
Rights Management policies and practices are implemented and appropriate.
Principles related to management of resource contribution
2.5.1. The Carrick Exchange should develop clear policies, guidelines, and procedures for
resource contribution.
2.5.2. The Strategic Plan should include guidance on resource management, archiving,
sustainability, and support and encouragement for resource contribution.
2.5.3. An evaluation plan should be developed to assess and report on resource
contribution to the Carrick Exchange.
Peer review and commentary
Protocols and mechanisms related to the goals of peer review
3.1.1. Formal peer review policies and procedures of the Carrick Exchange should align
with institutional policies and practices, and promote the scholarship of teaching and
learning.
3.1.2. The goals of formal peer review should be established and inform related policies and
procedures.
6
3.1.3. Peer review should enable judgements about the quality and usefulness of a
resource.
3.1.4. Resources contributed to the Carrick Exchange should be classified according to the
level of peer review received.
3.1.5. The formal peer review process should be appropriately resourced and adequately
rewarded
Protocols and mechanisms related to the process and procedures for peer review:
3.2.1. The Carrick Exchange should establish processes, criteria and standards that enable
consistent, equitable and fair peer review.
3.2.2. The peer review process should meet the expectations of the sector and the needs of
members.
Protocols and mechanisms related to informal commentary
3.3.1. Guidelines and protocols for informal commentary should be established to ensure
that the needs and expectations of contributors and users are met.
Protocols and mechanisms related to rewards and incentives for formal peer review and
informal commentary
3.4.1. The Carrick Exchange must implement rewards and incentives for participation in
either formal peer review or informal commentary
7
Introduction
______________________________________________
Background
The Carrick Exchange is a major initiative within the Australian higher education sector.
The Carrick Exchange is a new online service that will provide learning and teaching
resources and support communication and collaboration across the national and
international higher education sector. The Carrick Exchange is a hub for the
exchange of ideas about teaching practice in the Australian higher education sector.
It is a place to explore, discover and experiment with issues, technologies, processes
and ideas.
Most importantly, it is about connecting people with people . . . It is a space to
contribute, discuss and debate. (Carrick Exchange website) 2
ascilite, the Australasian Society for Computers in Learning in Tertiary Education, was
engaged to assist with the development of the Carrick Exchange (2006 – 2007) by conducting
research into:
•
•
•
Users’ needs within the higher education sector for collaborative and communication
spaces for learning and teaching;
The conditions and contexts of use of resources available for sharing and reuse within the
higher education sector; and
The policies necessary to facilitate engagement of the higher education sector with the
Carrick Exchange.
The ascilite research was funded by the Carrick Institute for Learning and Teaching as a subproject of the Carrick Exchange.
This final report on guidelines and recommendations is one of four documents which present
the findings derived during Stage 2 of the ascilite research (January to September 2007).
Report 1: The Literature Review
Report 2: Themes, Issues and Concerns Emerging from Key Practitioner Interviews
Report 3: Themes, Issues and Concerns Emerging from Focus Groups
Report 4: Final Report
Preliminary work was undertaken in 2006 as a Stage 1 investigation, examining the
background issues which might impact on the development of the Carrick Exchange.
Outcomes of Stage 1 of the project are reported in the Resource Identification Network (RIN)
Project, Stage 1 - ascilite Component Think Tank Issues Paper 2006.
The challenge for the Carrick Institute in developing the Carrick Exchange is that the
literature, along with national and international practice, indicates that uptake of repositories
of learning materials has been slower than expected. Poor engagement by target users may
be related to lack of clarity about purpose, and lack of strategies and resources directed to
fully engaging the community for which the repository has been designed.
2
http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381
8
With this in mind, the key drivers for the ascilite Stage 2 research were:
•
•
•
consolidation of knowledge and current practice;
investigation into engagement of the higher education sector; and
the development of communities of practice.
Consolidation: Stage 2 of the ascilite/Carrick Exchange project built on Stage 1 research
informing the technical development of the Carrick Exchange by reporting on a broad range of
user needs and contexts of use of resources and communication environments.
Engagement: Through an examination of professional groups that have knowledge and
expertise in developing and adopting learning and teaching resources, the ascilite
investigation aimed to find some solutions to the general lack of user engagement in
repositories and online services in education world-wide reports. The research focussed on
methods of engagement with informal repositories of knowledge using Web 2.0 technologies,
and explored alternative methods of capturing information and networks of users in this
context.
Developing communities of practice: Consideration of how networks and communities of
practice are built and sustained is key to the on-going success of the Carrick Exchange.
Communities of practice are not formal arrangements or organisational bodies but are
informal entities that exist due to the members’ shared domain of knowledge, their sense of
community and shared practice. Communities and networks such as those around existing
online services were investigated (MERLOT (North America), CD-LOR (UK), the Minister of
Communications Digital Strategy Advisory Group (NZ), the Australian Partnership for
Sustainable Repositories (APSR), and Australian Research Repositories Online to the World
(ARROW) etc.). Factors impacting on these communities and reported here include
organisational, socio-cultural, information management and technological issues.
Full details of the background to the ascilite project are provided in the ascilite/Carrick
Exchange Proposal and Project Plan 2007. Further analysis of the Carrick Exchange is
provided in reports by Phillips, Orrell and Millea (2007), and Lefoe, O'Reilly, Parrish, Bennett,
Keppell and Gunn (2007).
A glossary of the key terms used in this research is provided as Appendix A.
Presentation of findings, recommendations and guiding
principles
In presenting the consolidated findings as recommendations and guiding principles in this
report in the three key areas (engagement, resource contribution and identification, and peer
review and commentary), the authors note that there may appear to be some repetition. This
is unavoidable as some issues have implications that cross all three areas and manifest in
different ways according to the focus area.
9
Methodology
______________________________________________
Aims and outcomes
Aim
The aim of Stage 2 of the ascilite research was to inform development of the Carrick
Exchange through further research into the three key areas: engagement, resource
identification and contribution, and peer review and commentary. The data and analysis was
to be derived from a coordinated and collaborative series of activities, distilling the available
knowledge and expertise from:
•
•
•
•
The literature
Current practice
Exemplary community development to foster engagement with online services, and
Key practitioners and networks in both national and international settings.
Outcomes
The outcomes of the Stage 2 research are:
1. Recommendations and procedures to foster engagement and uptake of the Carrick
Exchange by the sector.
2. Guiding principles for resource contribution and identification, networking and
engagement that extend beyond the development of technical infrastructure and
standards to consideration of the ease of use and the ability to customise the resources;
3. The development of Peer Review and commentary protocols, and mechanisms for
resource contribution and identification to be incorporated in the Carrick Exchange.
Key terms
See Appendix A for a glossary of key terms used in this research.
10
Table 1: Research questions for Stage 2
Research questions
1. What Peer Review and
commentary protocols and
mechanisms can be derived from
the higher education community?
Sub-questions
•
•
•
2. What resource contribution and
identification methods will engage
and encourage users to contribute
and collaborate within the Carrick
Exchange?
3. What are the successful
methods for establishing and
maintaining engagement (i.e. use,
reuse and recognition) by the
higher education community with
the Carrick Exchange?
•
•
•
•
•
•
•
What new and current methods exist for peer review
and commentary of resources that can be adapted for
use for the reviewing of teaching and learning
resources?
o What gets peer reviewed?
o What methodologies apply to different types
of resources?
o What process for peer review is used?
o Who are the reviewers?
What recommended policies and procedures are
currently in use?
What are the policy implications for the Carrick
Exchange?
What are the key factors to support resource
contribution, identification and reuse?
What types of resources will be contributed?
What recommended policies and procedures are
currently in use?
What are the policy implications for the Carrick
Exchange?
What are the key factors for establishing and
maintaining engagement in academic and nonacademic online communities e.g. online support
groups, interest and resource sharing groups etc?
What recommended policies and procedures are
currently in use?
What are the policy implications for the Carrick
Exchange?
The design-based methodology
A design-based research methodology was adopted for Stage 2 of the ascilite research. This
required an iterative approach to data collection and analysis, similar to action research in its
inclusion of stakeholders during each stage. The difference, however, is that the outcomes of
design-based research should be a series of recommendations and/or principles for design
(Design-Based Research Collective, 2003; van den Akker, 1999). (See Figure 1.)
Figure 1: The design-based research cycle. Adapted from Reeves (2000)
Overall, the design-based research cycle was completed three times during Stage 2 of the
Carrick Exchange project (Figure 2). The final cycle will be completed in December, 2007.
11
Each cycle focused on one type of stakeholder group and each successive cycle was
informed by interpretation of the data from the previous cycle/s.
Cycle 1 Key practitioner interviews
In Cycle 1 the project team approached thirty five key members of the higher education
community, liaising with organisations within Australia and overseas as part of the process of
developing a well grounded understanding of repositories and their communities based on
evidence and experience (See Appendix C). Twenty-nine of these individuals agreed to
interviews. National and international interviewees were chosen from across disciplines,
institutions and fields of interest and identified as those with:
(a) a need for online collaboration and communication services for learning and teaching;
and/or
(b) significant experience in the field.
Interviewees included representatives of international projects (6), national initiatives (3),
vocational education and training initiatives (4), institutional repositories (3), discipline and
professional associations (9), institutional users (2), Deputy/Pro Vice-Chancellors (1), and one
Carrick Institute Fellow.
The research questions underpinning all interviews were:
1. What peer review and commentary protocols and mechanisms can be derived from the
higher education community?
2. What resource contribution and identification methods will engage and encourage users
to contribute and collaborate within the Carrick Exchange?
3. What are the successful methods for establishing and maintaining engagement (i.e. use,
reuse and recognition) by the higher education community with the Carrick Exchange?
Each interview was conducted by a single interviewer (one of the project team members)
either face-to-face or by phone, and recorded with the participant’s consent. Interviewers took
notes during the session and a summary transcript of the interview was made. In five cases
transcripts were created from notes only, due to failure of the technology. Transcripts were
returned to interviewees for validation and then entered in qualitative analysis database
software for identification of emerging themes and analysis. Cross-checking of themes by two
ascilite project team members was undertaken to increase reliability. This assisted in the
validation and preliminary interpretation of findings. A full report of the interviews is included in
ascilite Report 2 for the Carrick Exchange Project: Themes, Issues and Concerns Emerging
from Key Practitioner Interviews
Cycle 2 Focus groups
Cycle 2 was undertaken through three focus groups (Brisbane, Sydney and Melbourne). A
total of twenty-two practitioners participated. The aims of the focus groups were to build on
the outcomes of the data already gathered from the individual interviews in Cycle 1 and:
•
•
•
•
•
Review the data and report from the key practitioner interviews;
Discuss the issues that arose from the data;
Prioritise issues raised and generate possible strategies for addressing these;
Generate ideas for policy development and implementation; and
Identify missing data or where further research was required.
12
Representatives were invited from each Australian university and where possible included:
•
•
•
•
•
•
•
Academics interested in learning and teaching but with little experience of technology;
Members of cross-institutional teams (Carrick grant holders, consortiums etc.);
Educational and staff developers;
Librarians;
‘Early adopters’;
Repository representatives; and
A mix of gender and cultural backgrounds.
Availability of staff meant that it was not always possible to achieve this diverse mix in each
focus group. However we did achieve representation from five of the six States, including
Western Australia and South Australia. There were no representatives from the Northern
Territory, The Australian Capital Territory or Tasmania. Four to twelve participants were in
each group, and each focus group included some representation of educational and staff
developers, and members of cross-institutional collaborative project teams. In at least two of
the focus groups there were librarians and repository representatives. Two participants
represented the vast majority of staff in higher education who have an interest in learning and
teaching but may have only had limited use of the technology. In addition, within the groups
there were varying degrees of expertise from those highly skilled in the use of technology
through to those with a more limited experience. Two participants had also attended the
Carrick Exchange Think Tank forum in Melbourne, September 2006 (Stage 1). Some focus
group participants aligned to more than one representative category.
Prior to the focus groups participants were sent:
•
•
•
the focus group questions, derived from the data in Cycle 1;
promotional material providing a background and context to the Carrick Exchange; and
the Executive Summary from the Report on Cycle 1 – Key Practitioner Interviews.
The findings from the key practitioner interviews were used to underpin guided reflection by
the focus group participants. The two-hour focus groups were based on a structured
discussion process whereby participants interrogated the findings of the key practitioner
interviews and worked towards the development of possible strategies, solutions,
recommendations and policies in the three areas –engagement, resource identification and
contribution, and peer review and commentary. At the beginning of each session, a very brief
presentation was provided as to the nature and functions of the Carrick Exchange under
development.
The questions were posed by one member of the research team while a second member took
written notes of the session. The focus groups were also recorded with the consent of
participants and the recordings transcribed. These transcripts were analysed with the same
qualitative approach in Cycle 1 of using software to assist in deriving key themes and cross
checking between two members of the research team. The report of focus group themes and
illustrative quotes were subsequently shared with the ascilite reference groups (see Cycle 3)
to further inform the ongoing development of the Carrick Exchange. A full report of the focus
groups is included in ascilite Report 3 for the Carrick Exchange Project: Themes, Issues and
Concerns Emerging from the Focus Groups.
Cycle 3 Reference groups
To establish membership of three ascilite reference groups, in all 24 ascilite members were
selected from expressions of interest received in the period December 2006-March 2007.
Membership of reference groups was determined on the basis of stated interest and
expertise, and members were allocated to one of the following research areas: (1) user
engagement, (2) resource identification and contribution and (3) peer review and
13
commentary. To ensure representation across the sector both geographically and
institutionally, the following members were selected:
-
12 from NSW,
4 from Victoria,
2 from Queensland
1 from Western Australia, and
2 from the TAFE sector, NSW.
In addition reference group members were also selected from New Zealand (1), Singapore (1)
and Hong Kong (1).
Reports from Cycle 1 and Cycle 2 containing the data, analysis and recommendations for
each cycle were uploaded to the EdNA 3 site for reference groups to peruse and provide
comment. Comments submitted were thus for open view to all subscribers of the reference
group.
ascilite reference group members were requested to comment within a 3-week period and 10
of the 24 members submitted constructive and well considered responses to the forum (4 of
10 engagement; 2 of 7 peer review, and 4 of 6 resource identification and contribution). Some
interaction occurred among the reference group members as evident by the average number
of postings per respondent within forums being between two and three. One member of the
reference groups reported they had read the reports discussed with colleagues in their
institution and subsequently provided a synthesis of responses for the forum. An additional
three people read through the comments but did not contribute themselves. All those who did
not respond were encouraged to bring their perspectives to the symposium to be held at the
ascilite conference (see Cycle 4).
All comments submitted to reference group forums were analysed using the approach to
deriving themes that was used throughout previous cycles. The key themes,
recommendations and guiding principles were included in the overall report for Cycle 3
(ascilite Report 4 for the Carrick Exchange Project: Final Report). The final set of design
principles for Cycle 3 are included in the Conclusion to this report.
Cycle 4 Symposium
An international symposium is to be hosted during the 24th Annual Conference of the
Australasian Society for Computers in Learning in Tertiary Education (ascilite). In the lead-up
to this symposium the ascilite Carrick Exchange final report will be circulated to reference
group members. In addition three papers have been prepared and will be provided to
symposium registrants for stimulus. The authors will report on current issues in the
repositories and their communities developed in UK, USA and Australia.
Using these stimulus papers for pre-reading by participants, the symposium will be structured
to allow for small group interaction and an active approach to deriving issues themes and
concerns that will guide future research into the Carrick Exchange. The symposium will also
be recorded with the consent of participants and one member of the research team will take
notes including any that are put up on a whiteboard. These recordings and notes will be
transcribed and the key themes will be derived. Illustrative quotes will also be used in the
report that will be delivered following the symposium.
Overall, the aim of all four cycles was to inform the development of possible strategies,
solutions, recommendations and policies in the three key areas: engagement; resource
contribution and identification; and peer review and commentary.
3
Education Network Australia, collaborative workspace for project, http://www.edna.edu.au/edna/go
14
Figure 2: Project Methodology Showing Four Cycles
15
Literature review
______________________________________________
Further review of the literature is to be found in ascilite Report 1 for the Carrick Exchange Project:
The Literature Review.
Engagement of the sector and communities of practice
To promote knowledge sharing in the domain of learning and teaching, and engage the Australian
higher education community within the Carrick Exchange, it is not sufficient to build a repository of
resources, nor merely to provide social networking software and spaces for collaboration and
community building. The literature increasingly documents a general lack of user engagement with
repositories and online services (e.g. Gunn, Woodgate & O’Grady, 2005; Hummel, Tattersall,
Brugos, Brouns, Kurvers and Koper 2005; Littlejohn, 2003; Phillips, Aspin, Hull, & Oxley, 2004). It
is acknowledged that effort should be directed towards the process of engaging the target
audience, familiarising groups with the affordances of the Carrick Exchange and its potential to
support user needs, and proactively facilitating the development of networks and communities of
practice engaged in sharing and developing high quality resources.
The potential benefits of repositories are generally well documented. The following list, generated
by combining Deakin University institutional repository information, (Monahan & Owies, 2005,
pp.1-2) and the Jorum national repository information from the UK (Jorum, 2007), indicates
significant advantages.
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Participation in the advancement and sustainability of e-learning at a national level;
Conservation of time and effort through reuse and sharing of knowledge and resources;
Facilitation of communities of practice and online professional networks at a national level;
Project dissemination as a requirement of funding obligations;
Archival functionality for publicly funded project output;
Seamlessly integrated knowledge management systems;
Easy cost effective searching and retrieval of resources;
Improved sharing and storage systems;
Improvements to the quality and currency of courseware and methodologies in units of study;
Automated review notifications for content expiry;
Workflow visibility and effectiveness - documented processes for contribution and reuse;
Authenticated and secure access;
Automated tracking and control over the reuse of materials;
Legal security for contributors through a robust tagging and licensing system;
Management and protection of copyright; and
Metadata and cataloguing conforming to accepted standards, enabling easy search and
retrieval.
However, realising these advantages is a complex task. Ignoring the sociocultural issues relating
to learning object repositories is to run the risk of creating an under utilised service. It cannot be
assumed that reuse will follow existence of the repository (Margaryan & Littlejohn, 2007; Philip,
2007). Margaryan and Littlejohn point to the effect various cultural dimensions (organisational,
professional, disciplinary and national) may have on the impact, uptake and usage of the system.
These factors are further influenced by community size, member proximity to the resource, the
roles of stakeholders and types of tasks for which the resources in the repository are intended and
used. In addition, barriers to uptake will cluster around socio-cultural, pedagogic, organisational
and informational management, and technological issues (Margaryan, Currier, Littlejohn, & Nicol,
2006). Margaryan et al. offer the following general solutions to each of these areas (pp.4-5):
16
Solutions to socio-cultural issues
•
Design of the LORs (learning object repositories) should be based on understanding of cultural
norms and expectations of the user communities.
Solutions to pedagogic issues
•
•
•
•
•
•
Emphasis on pedagogy pull vs. technology push.
Development of LORs by multidisciplinary teams (including teaching practitioners as well as
learning technologists and librarians).
Provision of examples of successful use of LORs related specifically to teaching and learning.
Development of LOR models involving co-construction of resources by the students.
Demonstration of impact for learning and added value for individual users.
User development and support in information literacy.
Solutions to organisational and information management issues
•
•
Incentives and rewards linked to community needs and goals.
LORs linked to organisational strategy and objectives.
Solutions to technological issues
•
•
•
Facilitating ease of use, engagement, efficiency and pedagogic effectiveness.
Conceptualisation of LORs as a context rather than isolated tools.
Effective policies and practices for metadata creation.
For the Carrick Exchange to become integrated into the everyday work practices of the Australian
higher education sector, there are considerable challenges to address. Staff are habitually time
poor (Koppi & Lavitt, 2003) and there is no well developed culture of sharing and reuse. Reusing
materials developed by others and dissemination of one’s own personal teaching materials are not
necessarily seen as incentives to effect the practice of reciprocal sharing (Wetterling & Collis,
2003; Koppi & Lavitt, 2003). Barriers to change are not easily surmounted as the VET sector has
found in the evaluation of the initial phases of the Australian Flexible Learning Framework (Phillips
et al., 2004). The Carrick Exchange will be initiating technical change, and educational and cultural
change. The literature suggests that change management issues should be addressed early and
that to effect major educational change commitment at the highest organisational level will be
required (e.g. Ely, 1999; Kenny, 2002, in Kenny 2003; McKenzie, Alexander, Harper, & Anderson,
2005). Support championed and led from the top, with “bottom up” support for innovators, and
provision of well supported resource development for those in the middle is the best model (Nicol
et al., 2004 in Weedon, Bricheno & Chidwick, 2004). Scott (1999, p.10) notes that one of the
reasons why many “intrinsically worthwhile” innovations founder in educational settings is that
administrative and support personnel are not sufficiently apprised of key information and
integrated well enough into change processes. The study by Ehrmann, Gilbert and McMartin
4
(2007) into the impact of icampus in five American universities noted that “the ‘market’ for higher
education responds weakly, if at all, to changes in educational effectiveness" and that “widespread
dissemination of such activities is very difficult” (p.2). Also:
External forces and internal dynamics both tend to create a short institutional attention span
for educational reform. . . It can easily take 8-12 years or more to create a visible, rewarded
change in the outcomes of a degree program or a university education. (Ehrmann et al., 2007,
p.3).
Ehrmann et al. (2007) emphasise the need for rewards to “feed” the demand for adoption of new
4
iCampus is an initiative between Microsoft Research and MIT. The consortium aimed to build and research
technologies that could make a significant difference in university teaching and learning. Because of the
failure of a number of large-scale projects which had not moved beyond the pilot phase, Ehrmann et al.
(2007) and iCampus decided to evaluate five successful projects.
17
ideas (p.iii); and importantly suggested that cross-institutional and intra-institutional links should be
“nurtured” amongst central services that offer educational and professional development support
(p. iii).
The use of social networking software to facilitate communication, networking, collaboration and
sharing amongst users will be an important aspect of the Carrick Exchange. In general, social
software supports collaborative online spaces such as blogs and wikis; sharing of and commentary
on photo, audio and video files; digital storytelling; 3D virtual worlds (such as Second Life,
http://secondlife.com/); and social spaces such as Facebook (http://www.facebook.com/).
Technology usage by individuals is undergoing a change from consumption of content to usercentric creation of content and collaboration via networks. However, there is often an unchallenged
assumption that the software on its own promotes communities. The critical elements for
successful use of social software are identified in the Australian Flexible Learning Framework
report for the VET sector (Evans, 2007, p.13) as,
•
•
•
“Authenticity” – there is a real and established need to use the software;
“Relevancy” - its use is “relevant to the need, and appropriate for the client”; and
“Support” - there is support for the software within an “enabling culture”.
Furthermore, Evans (2007) indicates that the best strategies to help staff learn how to use social
software tools are action learning, just-in-time mentoring, coaching and work-based learning. The
tools considered to be most useful in assisting staff to learn how to use social software are virtual
conferencing and online forums. The best professional development strategies were reportedly
blended ones using a variety of opportunities and strategies. The importance of modelling as an
enabler as well as a professional development strategy was emphasised. Digital storytelling
through the use of multimedia case studies was considered very effective, along with “learning by
stealth”, i.e. blending usage into everyday practices and inviting managers to ‘see, hear and feel
what happens’ (Evans, 2007, p.22).
The Carrick Exchange may well support fully formed communities of practice plus other looser and
more brittle networks. Wenger, McDermott and Snyder (2002), in defining communities of practice,
support the notion that a website on its own is not a community of practice. Communities of
practice are “groups of people who share a concern, a set of problems, or a passion about a topic,
and who deepen their knowledge and expertise . . . by interacting on an ongoing basis” (p.4).
Development of the necessary domain of knowledge, the community and shared practice over
time (through sharing of stories, experiences and methodologies) will present ongoing challenges
to the community. Wenger et al. suggest that domain, community and practice are not only
defining terms for a community of practice, but that they represent “different aspects of
participation that motivate people to join a community” (Wenger et al., 2002, p.44). Some
members will join because they are interested in the domain, the specific area of knowledge; some
will join because it is the community aspect they seek; others will be seeking to learn about
practice, the standards, the tools and the lessons learned by experts in the field. McDonald and
Star (2006) point to five key challenges in the formation of a community of practice: the need for
financial support; issues of academic time poverty; the need for well-placed institutional
champions; the difficulty of identifying and quantifying outcomes from communities of practice; and
the question of sustainability and ongoing support.
In addition, policies should be stated clearly “and not form unwanted obstacles” for communities to
develop and the threshold for entry into the system should not be unnecessarily complicated or
high (Hummel et al., 2005, p.66). There must be sufficient room for the community to self organise
its own structure and facilities. Hummel et al. also suggest that the system should begin with a
minimal set of activities and forums and, when more participation is required, this might be better
promoted through synchronous collaboration rather than through asynchronous means. “Clear
policies, usability and reward systems are of importance when facilitating a learning network”
according to Hummel et al. (Hummel et al., 2005, p.55). The importance of systems of reward and
recognition for effort in teaching, and activities such as contribution to the development, sharing
and reuse of resources is gradually being recognised within the higher education sector. Some
institutions are beginning to take action which recognises the important role of teaching: for
example, the University of Queensland, in an internal review (University of Queensland, 2007),
18
has recognised that different staff roles demand varying amounts of emphasis on teaching and
research, which should be acknowledged. The argument is that both are of value and worthy of
recognition and reward. In addition exemplary repositories and exchanges such as MERLOT and
CLOE, and the Carrick Institute itself, have systems of reward in place that provide models for
recognising and rewarding members through awards and conferences. The collaborative RUBRIC
project for regional universities also has in place recommended methods for tying contribution to
their repository with promotion and career advancement processes. The issue of rewards and
recognition is not dealt with in depth in this report as it is to be researched further in another study;
however, the ascilite research recognises its central importance in the development and
sustainability of the Carrick Exchange.
Resource contribution, identification and models of use
As well as the communities, networks and workspaces expected to establish around the Carrick
Exchange, there will be a repository of quality learning and teaching materials either deposited on
the site or linked from websites and databases elsewhere. In a review of repository development in
the UK (the CD-LOR Report), the authors warn all repository managers to clearly establish the
need for any collection (Margaryan, Milligan & Douglas, 2007; Margaryan & Littlejohn, 2007). This
view is shared by findings from the research of Gosper, Woo, Gibbs, Hand, Kerr and Rich (2005),
and Ringan, Corley and Campbell (2005). Communicating the purpose of the repository and its
community will be important to the success of the Carrick Exchange. The Australian Partnership
for Sustainable Repositories (APSR) investigation into the ten major issues facing repository
service providers, relating to research output in Australia (Henty, 2007), noted that defining the
collection was essential communication task in the process of repository development and user
engagement. In addition, the APSR research uncovered similar issues of lack of engagement to
those noted in the CD-LOR report. Senior academics responsible for repository services and data
management were interviewed for the APSR research. It was noted that open access repositories
have not been taken up with great enthusiasm: an exception was the Cornell University Physics
eprint archive (http://www.arxiv.org). An interesting finding was that in the research context,
mandating article deposit resulted in high levels of contribution, as illustrated by the Queensland
University of Technology experience. However the high cost and effort required to deposit articles
in the system was seen as a barrier by some senior managers.
This lack of engagement with repositories has been reported elsewhere (Margaryan et al., 2007;
Wenger et al., 2002). While studies such as Najjar, Ternier and Duval (2004) may document
usage patterns from the logs of repositories, they do not reveal the broader picture of engagement.
Some studies of digital repositories and engagement (Bradley & Boyle, 2004; Littlejohn, 2003;
Hand, Gosper, Woo, Gibbs, Kerr & Rich, 2004) list incentives for use put forward by target users,
but engagement still remains relatively low. Repository managers catering to more diverse users,
e.g. around national rather than discipline-based repositories, are likely to face greater problems in
this area (Margaryan & Littlejohn, 2007). Furthermore, the way repositories are used depends not
only on the “dimensions of repositories” (purpose, scope, target audience etc.), but also on “key
characteristics” of the communities (p.4336).
In their study of the needs of academics in the research domain, Foster and Gibbons (2005) noted
that academics characteristically require the following:
[To be able to] work with co-authors; keep track of different versions of the same document;
work from different computers and locations, both Mac and PC; make their own work
available to others; have easy access to other people's work; keep up in their fields; organize
their materials according to their own scheme; control ownership, security, and access;
ensure that documents are persistently viewable or usable; have someone else take
responsibility for servers and digital tools; be sure not to violate copyright issues; keep
everything related to computers easy and flawless; reduce chaos or at least not add to it; and
not be any busier. (Foster and Gibbons, 2005, p.4 of 12)
19
This suggests that collaborative workspaces offered through the Carrick Exchange, where
resources can be shared and contributed may well meet a need within the higher educational
community. The research of Breslin, Nicol, Grierson, Wodehouse, Juster and Ion (2007) within the
discipline of Engineering in the DIDET project at Strathclyde University also indicates the value of
such shared workspaces for teachers and students to share and manage resources. Further, the
need for a critical mass of resources within the repository is reported as critical for a viable
implementation (Foster & Gibbons, 2005; Breslin et al., 2007). Breslin et al. also suggest that it
may take two to three years for a cross-institutional repository project to become established and
be able to offer benefits to others beyond the core implementation group.
One of the design aims of the Carrick Exchange is to promote processes characterised by Stuckey
and Arkell (2006, p.7) as “connection” not just “collection”. Therefore, diverse communities and
database or repository models are being explored to determine the elements that contribute to
success. Examples as varied as amazon.com (eCommerce), domain.com.au (real estate),
Wikipedia.org (a collaborative encyclopaedia), Youtube.com (video sharing), Edna (collection and
collaborative network) and MERLOT (educational exchange) are being reviewed. Their strategies
for engagement and networking, using Web 2.0 technologies, may inform the development of
personal and group workspaces on the Carrick Exchange. On amazon.com, for example,
registered users are greeted by name and have delivered to them recommendations and updates
about available products. The user can collate chosen resources from the site, and develop their
own profile of recommendations, ratings and preferences. The system monitors users’
preferences, locates resources based on previous searches and feeds back information that
connects users with other like-minded members. A mix of these features may be of value to the
Carrick Exchange. MERLOT (http://www.merlot.org) uses similar techniques to support its
educational community.
Peer review and commentary
The quality and currency of resources added to a repository is critical to ensure user needs are
met. A national repository is well-placed to implement a formal peer review system. Peer review as
quality assurance could include an assessment of the currency, educational design and
construction of resources; compliance with copyright, intellectual property and digital rights
management policies; and technical accuracy and reliability. Formal peer review, a lengthier and
more demanding review process, might replicate the scholarly peer review process which leads to
publication in the higher education sector. Peer review may also be an informal process whereby
members of the community voluntarily respond to others’ contributed resources, or resources
stored elsewhere but linked to via the Carrick Exchange. These resources might be finalised
products which the authors publish and share widely or to a narrower select group of colleagues,
or resources “under development”. This informal sharing of ideas could be an important element of
the Carrick Exchange and the basis for various communities of practice.
One of the best models of formal peer review for educational resources is that of the MERLOT
system in the USA (McMartin, 2004; Nesbit, Belfer & Vargo, 2002). This system is based on the
academic peer review practices for scholarship and publication in higher education, an “expertiseorientated” approach (Worthen, Sanders & Fitzpatrick, 1997). Peer review is recorded for formal
recognition of contributors, and usage pattern updates are sent to contributors each year. Building
on the MERLOT model, Nesbit et al. (2002) have created a convergent participation model for
evaluation of learning objects where resources undergo a two cycle process: two individual
experts assess the resource, and then a combined group assessment is made, amalgamating the
feedback from both assessments. Students are included on the panel of reviewers. The Australian
ACELL project, Advancing Chemistry by Enhancing Learning in the Laboratory
(http://acell.chem.usyd.edu.au/), is an example of a discipline-based repository which also makes
use of learner feedback in the evaluation process. Other repositories which implement peer review
are Intute in the United Kingdom (http://www.intute.ac.uk/policy.html), and Educause in the USA
(http://www.educause.edu). The Jorum national repository in the UK (http://www.jorum.ac.uk/) is
currently investigating peer review processes.
20
Though peer review is seen as a value adding process, realisation of peer review processes is
problematic. The Taylor and Richardson report on Validating Scholarship in University Teaching
(2001) concluded that at the time of publication, there was a “window of opportunity” to establish a
peer review scheme to assess information and communication technologies in Australian
universities (p.87). The project set out to develop conceptual and procedural bases for a national
scheme. Reviewers would be editors of journal and conference proceedings and it would be the
responsibility of these editors to advertise the scheme. A supporting professional development
strategy was to be developed and resourced. The project generated a number of options for the
peer review process. Despite the positive outcomes of the project, the scheme has not been
adopted nationally.
Taylor and Richardson (2001, p.7) offer four reasons for peer review:
•
•
•
•
Need for the evaluation of quality in ICT-based resources;
Need for recognition for the developer of ICT-based resources;
Need for the collaboration and dissemination of resources and knowledge relating to the
design and construction of these resources; and
Need for this quality assurance, recognition and collaboration to be grounded in the concept
of scholarship.
Taylor and Richardson stress that it is the focus on scholarship that is central to the “value-adding”
work of academics (p.7). Peer review is seen as a means of validating and recognising that
scholarship of teaching is important, and this is essential if effort around teaching is to receive
parity with scholarly research endeavours in higher education.
Recent changes to academic publishing have seen some journals try a move towards a more
“open” approach to publishing which may be of interest to those conducting peer review of
educational resources. One of the arguments against traditional blind peer review is that the
reviews produced and the research accepted favour tradition and not innovation (Rogers, 2006).
The peer review system evolved when dissemination was difficult and expensive. There is an
argument that open publication may allow innovation to be disseminated more speedily. Anderson
(2007) argues that the Web may become the first place of publication in the future, and only the
very best and most enduring works will be published in paper. Cornell University’s open access
5
Library e-print site, arXiv (“archive”), has allowed pre-publication of papers in the sciences since
1991, and for these papers to be open for comment. The prestigious science journal Nature 6
began trialling a more open peer review process (Rogers, 2006) in June 2006. Authors can
choose a 'pre-print' option. This pilot is now out of the beta stage and is called Nature Precedings
(Nature, 2007). Once a research article has passed an initial quality check, the author posts their
paper on the journal’s website, and anyone, provided they give their name and email address, can
comment on the research; the traditional blind peer-review process continues in the background.
The traditional method of review is lengthy, and can take between four and twelve months, after
two or three reviewers have made their assessment, the editor has compiled and reviewed the
reports, amendments are requested from the author and finally the work is published. The open
approach where a pre-print is out in the public domain immediately the research article is
completed allows for much quicker access. This could become a model for repositories like the
Carrick Exchange which undertake peer review, where currency of resources is a considerable
issue. Nature warns readers to treat the findings in any of the research published in Nature
Precedings with caution as results may be preliminary or speculative, but all documents are citable
(have a DOI), and are archived under a creative commons license where derivatives are allowed
(Nature, 2007).
Whilst the open review process speeds publication, the process of soliciting informal peer review
comments was less than successful. Reaction to the trial from authors and scientists was mixed.
While there was considerable traffic on the site where the pre-prints were posted during the four
month trial, and the concept was well received, few readers commented on the papers. Survey
results and analysis of the comments (Nature, 2006) indicated that: obtaining comments was
5
6
http://arxiv.org/
http://www.nature.com/nature/index.html
21
difficult; attempts to solicit comments produced only limited success; competition in the field may
have been a reason for lack of input; potential commenters thought that open peer review was a
good concept but still did not provide feedback; and editors’ analysis of the comments indicated
that generally comments were low level and did not add to the review process. This reluctance to
provide substantive comment in general on other sites was noted by some participants in the
ascilite research. Nonetheless, the argument for early publication and dissemination provided in
the Nature example probably overshadows the counter argument that informal peer review is likely
to be minimal and of little real value for development. In furthering the debate on new directions in
academic peer review and publishing, Siemens’ (2007) integrated model of formal peer review
combined with informal peer review commentary is of interest 7 , and points to the emerging
changes brought about by changes to technology: “Quite simply – as academics, we do not have a
clear model of implementation for scholarship in light of current online trends”. The new
democratic environment online brought about by Web 2.0 technologies raises issues of “identity,
fairness [and] civility” according to Siemens, which need to be addressed with adaptable
approaches.
7
http://www.elearnspace.org/Articles/journal.htm
22
Engagement guiding principles and recommendations
______________________________________________
Introduction
As part of the development of the Carrick Exchange, ascilite was engaged to conduct research
into:
•
What are the successful methods for establishing and maintaining engagement by the
higher education community with the Carrick Exchange?
o
o
o
What are the key factors for establishing and maintaining engagement in academic and
non-academic online communities e.g. online support groups, interest and resource
sharing groups?
What recommended policies and procedures are currently in use?
What are the policy implications for the Carrick Exchange?
The following protocols and mechanisms to foster engagement and uptake of the Carrick
Exchange have been synthesised from knowledge and expertise existing in: the literature,
current practice, exemplary community development to foster engagement with online services,
and key practitioners and networks in both national and international settings.
While one of the aims of the Carrick Exchange is to promote knowledge sharing amongst
teachers and other professionals within the higher education sector, there is a general
understanding that this is not an easy task. There is an expectation that resources of many
types that support learning and teaching will be available for reuse and development via the
Carrick Exchange, and that networks and communities will evolve around these knowledge
sharing systems. However, research and practice suggest that establishing viable communities
around repositories and exchange systems requires facilitation and an understanding of the
needs of the users. The more diverse the target group, the more issues there are: sociocultural, pedagogical, organisational, technical, and information management issues. Through
the ascilite research into engagement of the sector the aims have been to:
•
•
•
•
•
discover what methods and strategies might best apply in the Australian national context,
and ultimately work towards improved learning and teaching praxis;
investigate models of practice and exemplars for user engagement with both academic and
non-academic repositories, including methods for development and maintenance of
community;
locate stakeholder groups and individuals with a need to collaborate and share resources
within a resource information and networking environment;
identify the most significant demand-side issues, especially in relation to understanding
user needs and contexts of use; and
recommend options for establishing and maintaining engagement with the Carrick
Exchange of a range of users within the higher education community.
With repositories and online services emerging world-wide, a general lack of user engagement
has been identified as an ongoing cause for investigation and attention. This is considered the
most significant risk to the success of any repository. In its entirety, this report represents a
change management strategy to ensure uptake and engagement of the Carrick Exchange.
23
Recommendations in this section are structured around the following key themes that emerged
from the ascilite research findings:
1.
2.
3.
4.
5.
The purpose of the Carrick Exchange
Multiple levels of engagement
Management of engagement
Ease of use
Promotion and publicity
Recommendations related to the purpose of the Carrick
Exchange
1.1.1. The purpose of the Carrick Exchange should be well articulated to the
sector.
The Carrick Exchange is “a new online service that will provide learning and teaching
resources, and functions to support communication and collaboration across the national and
international higher education sector”. 8 The adequate promotion and communication of this
purpose to the higher education sector will ultimately be a key factor in the success of the
Carrick Exchange. One interviewee echoed this sentiment in commenting:
The purpose of the Carrick Exchange and what it intends to achieve needs to be
made quite clear to all key stakeholders and the intended audience from the very
beginning (Interviewee).
Efforts should be directed towards familiarising groups with the features, benefits and
affordances of the Carrick Exchange and its potential to support user needs. A clear statement
of the purpose of the Carrick Exchange with explicit details of the advantages of the system and
how it is differentiated from existing institutional repositories, databases, work spaces and
communities should be communicated to potential users. Defining the collection, its standards
and quality assurance methods, and effectively communicating these is an essential strategy in
the process of developing the Carrick Exchange, and enhancing user engagement.
The Carrick Exchange has been developed for those who teach, manage and lead learning and
teaching in Australian higher education. As such the target audience for promotion and publicity
should not be limited to people applying for and winning awards and grants, but should be open
to both academic and non-academic staff in the sector. Its currency and leadership in the
education field should be promoted as reasons for using the Carrick Exchange.
The Carrick Exchange should be central to all Carrick Institute functions. As one interviewee
commented:
It should sit across everything that Carrick does. [For] anything to do with the
Carrick Institute, the Carrick Exchange is where you go. Even the forms for grant
applications should reside on the Exchange. There has to be a degree of
integration that makes it work, otherwise it will just be a great idea that’s not
sustained over time. (Interviewee)
It is recommended that the Carrick Exchange is central to communication and information
provision concerning all Carrick Institute functions and activities, including notifications of
events and provision of forms for registrations, grant applications and awards.
8
http://www.carrickinstitute.edu.au/carrick/go/home/rin/pid/381
24
1.1.2. The quality of the processes, resources and functionality of the Carrick
Exchange should meet the expectations of the sector.
According to the research participants, the expectations of the higher education sector for the
Carrick Exchange is that it will be the focal point for sharing ideas, resources and expertise
about learning and teaching in the Australian higher education context. Research participants
warned that there must be a clear demand or need for the Carrick Exchange, and that it must
meet those demands or needs.
Quality was seen as a key success factor for the Carrick Exchange. Therefore, a strategic plan
and operational policies, procedures and systems should ensure that the Carrick Exchange
delivers a superior standard of services and products. This includes:
•
•
•
accuracy and currency of resources, requiring ongoing maintenance of resources by
originating authors;
a reliable functioning system with a sophisticated search engine that is easy and intuitive to
use and incorporates functionalities such as a federated search mechanism to search,
harvest and use resources in other higher education institutional repositories; and
robust “familiar” processes; as one interviewee suggested, “It is important to use familiar
processes that the target audience are used to, for example most people are used to
Google. So to mirror the search protocols on the Google search mechanism is a good idea”
(Interviewee).
Engagement and uptake of the Carrick Exchange is dependent on the experiences of users,
especially the initial encounter. The experiences according to research participants must be
positive and fruitful; which will occur if the Carrick Exchange establishes and ensures quality
processes, resources and functionality.
Strategies to convey the purpose, quality and scope of The
Carrick Exchange
Strategies for the Carrick Institute and the Carrick Exchange
1. Develop a strategic plan and associated policies to engage the higher education sector.
This plan should focus on a staged process for the implementation of the Carrick Exchange
and strategies to support engagement by the sector and should address issues of
sustainability into the future.
2. Develop operational policies, procedures and systems to enable the accuracy and currency
of resources to be maintained.
3. Ensure all Carrick Institute communication and information dissemination is managed
through the Carrick Exchange.
Strategies for the higher education sector
1. Enable access to existing higher education and institutional repositories.
Technical strategies
1. Develop technical systems to provide:
•
Automated alerts established to contact originating contributors to update submitted
resources, thereby maintaining accuracy and currency of resources. This should be
copied to Carrick Exchange administrative staff for follow up; and
25
•
a search engine that is easy and intuitive to use (perhaps modelled on Google) that has
a federated search mechanism to search, harvest and access resources from other
higher education institutional repositories.
2. Ensure the system is robust and well integrated when released. There is a risk in releasing
the system to the whole sector before it is ready.
Exemplars
Intute example document about purpose of the repository and community: Intute (2006). The
New Best of the Web. Ariadne, 48 (July). Retrieved September 1, 2007, from
http://www.ariadne.ac.uk/issue48/williams/
Many repositories provide an ‘about us’ section that details information about the purpose,
affordances, features and benefits of the repository. This feature clearly communicates
accurate, relevant and current information about the repository to users. These include:
•
•
•
Connexions: Provides detailed information in the ‘about us’ section in relation to the
philosophy, project history, key project stakeholders, and technology.
http://cnx.rice.edu/aboutus/
EducaNext: Provide detailed information in the ‘about us’ section in relation to the
governing organisation the service including case studies to illustrate aspects of use, and
related and relevant publications about the service.
http://www.educanext.org/ubp;
MERLOT: provides an overall statement of the mission of MERLOT with links to additional
pages detailing information about the resources, processes and functions of the system
and services.
http://taste.merlot.org/
Some repositories provide a similar communication of information through a link to “frequently
asked questions” such as:
•
•
•
Carnegie Foundation for the Advancement of Teaching: Frequently asked questions are
related to - Grants & Scholarships, Publications and Previous Work, the Carnegie
Foundation and the collection classifications.
http://www.carnegiefoundation.org/about/index.asp?key=17
Educause: has an extensive list of frequently asked questions across a broad range of
relevant and related aspects.
http://www.educause.edu/FrequentlyAskedQuestions/660
Health Education Assets Library (HEAL): questions are in relation to users, user groups,
and management of user profiles, cataloging issues, searching, contributing and using
resources, and technical issues.
http://www.healcentral.org/utils/help/faq.jsp
A number of repositories invite feedback from users about their experiences in the repository or
invite users to report unsatisfactory aspects of the repository. “Contact us” links that generate
an email addressed to repository managers are also provided by some repositories. Feedback
functions enable manages to keep abreast of the expectations of users particularly in regard to
processes, resources or the repository systems. Repositories providing this include:
•
Intute: invites users to lodge questions, comments or requests using a feedback form.
Users are also invited to ‘suggest a site’ for inclusion in the Intute database of Internet
resources.
http://www.intute.ac.uk/
26
•
Jorum: has a submission form for feedback that collects the details of the person providing
the feedback.
http://www.jorum.ac.uk
Key References
Barton, M. & Waters, M. (2004). Creating an institutional repository, Learning About Digital
Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved June 19, 2007, from
docushare.lib.rochester.edu/docushare/dsweb/Get/Document-19824/IR%20bibliography.pdf
Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning.
London and Sterling: Kogan Page.
Margaryan, A., Milligan, C. & Douglas, P. (2007). Community dimensions of learning object
repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object
repositories. Retrieved July 23, 2007, from http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Structured_Guidelines_v1p0.pdf
Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible Learning
Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian Flexible
Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from
http://flexiblelearning.net.au/aboutus/keydocuments.htm
Recommendations related to multiple levels of engagement
1.2.1. A strategic plan for the engagement of the sector at multiple levels should
be developed and implemented
The Carrick Exchange needs to foster the engagement of individuals and institutions within the
higher education sector. As one interviewee stated “Buy in from all stakeholders is required
from the start if the initiative is going to be successful”. Senior staff (Senior Executives, ViceChancellors, Deputy Vice Chancellors, Pro-Vice Chancellors and Deans) will be key to getting
the “buy in” of universities. The Carrick Exchange should also target:
•
•
•
•
the engagement of special groups or individuals with an institutional responsibility for
collaborating with teaching staff on development of teaching and learning resources;
those disciplines with a history of sharing resources, particularly in the early phases of the
project, to promote effective use of the system and provide models of usage and
community building;
staff support to help with orientation, troubleshooting, educational support and change
management; and
institutional Carrick Exchange Champions to raise awareness through such things as
demonstrations, seminars, and workshops at their local level.
Research by Rogers (2003) suggests five factors as critical for the adoption of technological
innovations:
•
Relative advantage: potential users need to see an advantage for using the Carrick
Exchange;
27
•
•
•
•
Compatibility: the Carrick Exchange must fit in with potential users’ current practice and
values
Complexity: the Carrick Exchange must be easy to use;
Trialability: potential users must be able to test the Carrick Exchange before making a
decision to adopt. For example they might be able to view the exchange without having to
join; and
Observability: potential users need to see the observable results of what the Carrick
Exchange can do for them in their specific circumstance. This can be provided through
exemplars or case studies of successful reuse of Carrick Exchange contributions.
Exemplars could be made available through workshops or training sessions.
According to Dormant (1997), when people are adopting a new innovation they move through
stages of acceptance. Therefore strategies corresponding to a person’s stage of acceptance is
required. Likewise, for optimum engagement in the Carrick Exchange, strategies that address
the stages of acceptance across the sector should be planned and implemented. Table 2
provides an overview of the stages of acceptance and suggests possible strategies for the
Carrick Exchange in each stage. Appendix E further explores these stages of acceptance and
how each might be addressed through the example of a Carrick Exchange user group (Carrick
Grant holders).
The Carrick Exchange should endeavour to explicitly address the five factors for the adoption of
technological innovations (Rogers, 2003), and the stages of acceptance (Dormant, 1997). At
the same time the Carrick Exchange should also consider the various key stakeholder groups
across the sector when developing a strategic plan for the engagement of the sector.
28
Table 2: Strategies for enabling acceptance and uptake
Stage Of
Awareness
Awareness
•
•
•
Passive
regarding the
change
Little/no
information
about change
General
Strategy
Carrick Exchange Strategy
Advertise
Advertise
Be an
advertising
agent
Provide information on the website, email
& phone details for 1st contact
Appeal to his or
her needs and
wants
Provide reasons for need for the Carrick
Exchange
Be credible and
positive
Have evidence of a critical mass of users,
including highly regarded experts &
leaders in the disciplines
Little/no
opinion about
change
Support Roles &
Personnel
CE Managers &
Champions,
CE promotional staff
Provide reasons for use and added value
provided by the CE in relation to other
repositories, networks & communities
Have evidence of national & institutional
support for the CE.
Local Champions
Relevant
associations (e.g.
disciplinary groups
with sub-groups
focussed on
teaching and
learning; crossdisciplinary
associations such as
ascilite, ODLAA,
HERDSA).
Have evidence of a quality database of
professional development & teaching
resources derived from Carrick Institute
projects
Curiosity
•
•
•
More active
regarding
change
Express
personal job
concerns
Asks
questions
about own
work and
change
Envisioning
•
•
•
Active
regarding
change
Expresses
work-related
job concerns
Asks
questions
about how
change works
Inform
Inform
Identify specific
concerns
Identify special needs of discipline groups
& use ‘push’ technology to reach target
groups (e.g. email, RSS feeds).
Provide clear
information
about concerns
Provide information at short f2f sessions
or as part of other related activities.
CE discipline-based
Champions &
collections managers
Emphasise
pluses,
acknowledge
minuses
Demonstrate
Demonstrate
Give success
images
Provide digital stories & case studies on
the CE
CE & Local
Champions,
Provide
demonstrations
Provide 1-2 hour face to face sessions
Staff & Educational
Developers
Connect with
peer users
Offer showcasing – stories from the
members of CE sharing & utilising
facilities
Encourage registration so as to view
activities of available communities &
networks.
29
Stage Of
Awareness
Tryout
•
•
•
Active
regarding
change
Has opinions
about change
Interested in
learning howto
General
Strategy
Carrick Exchange Strategy
Train
Train
Provide
effective training
Support & provide local face to face
training initiatives
Offer training in peer review for the CE
Provide job
aids, checklists
Promise
technical followup
Provide fact sheets, FAQs, guidelines,
tutorial animations, and digital stories on
the CE;
Provide forums on CE for getting started,
for Q&A, pedagogical and technical
Support Roles &
Personnel
CE Trainers
Help desk
supporters
Librarians
Educational
Developers
Technical Writers
Provide a database of members available
to support others in specialist areas
Use
•
•
•
Active
regarding
change
Uses change
on the job
Asks detailed
questions
about use
Support
Support
Provide
necessary
technical help
Provide help desk and centralised support
for resource submission issues
Provide
pedagogical
help
Provide support for embedding reuse and
sharing into the curriculum
Provide support clearing copyright,
metadata entry and checking, rights
management and version control
Conduct conferences and workshops
which focus on resource identification &
submission, and peer review of submitted
material.
Provide
recognition
Provide
reinforcement
CE and institutional
information
specialists
Educational and staff
developers, Carrick
and local
Champions, CE
community members
Librarians, copyright
managers
Carrick Institute
managers,
Institutional
managers
Provide awards for contribution and
excellence which support career
advancement; link with promotional
policies and procedures.
Provide time, funding and recognition for
preparing resources and activities which
support the CE; for submission and
maintenance of resources on the CE; for
ongoing networking and mentoring those
using the CE
Source: Adapted from Dormant 1997;
Key: CE = Carrick Exchange
1.2.2. Membership and the conditions of membership for the Carrick Exchange
should be specified.
The Carrick Exchange is being developed for those who teach, manage and lead learning and
teaching in Australian higher education. This criterion for members needs to be clearly
communicated to the sector so that those who qualify for membership are made aware of their
entitlement. It is also imperative that the corresponding conditions of Carrick Exchange
membership are communicated. Carrick Exchange membership conditions that might need to
be developed and communicated include:
30
•
•
•
•
•
•
•
Copyright and intellectual property;
Rights associated with the various levels of user accounts (e.g. browsing, searching
contribution and commentary rights, allowing only registered users certain privileges);
Reuse of resources (e.g. licensing agreements, terms of use for materials accessed from
the Carrick Exchange);
Cost of membership (to the individual and institution);
Membership of international stakeholders and professional associations;
Code of behaviour guidelines for individuals, groups, communities of practice or institutions
(e.g. detailing expectations such as non offensive contributions or commentary, adherence
to legal practices, human rights and professional accountability); and
Privacy requirements.
Research participants saw a cost for membership of the Carrick Exchange as a disincentive to
engagement. As such it is recommended that Carrick Exchange membership be free to all staff
within the Australian higher education sector and that other methods are explored for
sustainability through extended government funding or institutional commitment.
1.2.3. The Carrick Exchange should meet the diversity of members’ needs and
contexts of use.
An important process for the Carrick Exchange to undertake, in developing a strategic plan for
the engagement of the sector is:
1. Identify the needs of the sector; and
2. Ensure that these identified needs are met.
The shape of the Carrick Exchange and the networks and communities that develop will
emerge with the diversity of users’ needs. Ongoing attention needs to be given to maintaining
and growing the communities that develop around the Carrick Exchange, and supporting
existing and future working relationships and cultures. To some degree communities should be
able to self organise their own structures and facilities within the Carrick Exchange. The use of
social networking software to facilitate communication, networking and collaboration amongst
users is an important aspect of the Carrick Exchange and there is a need to promote and
support the effective use of these tools.
While the Carrick Exchange should have a discipline focus, cross-discipline and trans-discipline
activities and initiatives are of interest to the sector. The Carrick Exchange should be seen as a
mechanism for ongoing support of the diverse needs of members including collaboration. One
interviewee noted that:
The Carrick Exchange has to assist in the real work that is undertaken by staff in
universities, so it has to add to actual real stuff that people are doing on a day-today basis, and not be seen as something that is just an extra. (Interviewee)
The introduction of solutions, which the Carrick Exchange can provide, to meet the needs of
members should be managed in a strategic manner. Hummel et al. (2005) suggest that this
system of introduction should begin with a minimal set of resources, activities and forums that
meet the needs of the user or community. Gradually as participation increases and users’
needs develop, appropriate mechanisms and offerings are introduced. The promotion of these
introduced solutions may be supported through synchronous collaboration and communication.
31
Strategies to engage multiple levels of users
Strategies for the Carrick Institute and the Carrick Exchange
1. Devise strategies of engagement relevant to:
•
•
the varying levels of engagement (See Table 2); and
different key stakeholders including international members, active communities and
professional associations, and disciplines already engaging within repositories (e.g.
engineering and science)
2. Provide access to examples of how communities, groups and individuals are effectively
using the Carrick Exchange particularly from teaching and learning support areas such as
libraries and student learning centres. Provide user scenarios and case studies indicating
how the Carrick Exchange can work from several points of view.
3. Target groups or individuals (e.g. librarians, staff and educational developers) particularly in
the early phases of uptake, and assist these groups to effectively use the system including
Web 2.0 technology features. Exemplars could be drawn from these targeted cases.
4. Identify key leaders in teaching who do not readily engage with technology and investigate
what possible incentives would lead them to engage with the Carrick Exchange. Also
investigate the incentives for engagement in the Carrick Exchange through networks of new
and sessional teachers, who might have a real need for high quality teaching resources.
5. Provide tips and links on how to, e.g. use particular resources in teaching, implement new
pedagogical ideas, use new technologies.
6. Define the terms and conditions of membership of the Carrick Exchange
Strategies for the higher education sector
1. Involve Carrick Fellows and grant holders with the Carrick Exchange.
Technical strategies
1. Provide multiple levels of access and permissions to the Carrick Exchange e.g. view,
browse, review, collaborate.
2. Develop a tour of the Carrick Exchange (e.g. tutorial, overview) with examples of specific
actions, searches and details of how the functions of the Carrick Exchange can be used by
communities, groups and individuals.
Exemplars
Examples of multiple levels of engagement are found within repository “Conditions of
membership” documents, such as those found on:
•
•
•
EducaNext Code of Behaviour and terms of use:
http://www.educanext.org/ubp/PUSH/usermgmt@regrTermsOfUse
Educause Membership Terms and Conditions http://www.educause.edu/Membership/5
Jorum “User Terms of Use”: http://www.jorum.ac.uk/user/termsofuse/index.html
Communities develop and grow in online environments such as Flikr (http://www.flickr.com/)
Delicious (http://del.icio.us/?url), Youtube (http://www.youtube.com/), and Facebook
(http://www.facebook.com/home.php?). Strategies used by these online communities to engage
members include:
•
Providing mechanisms for individuals to share resources such as photos, news and
descriptions of websites, with identified people, groups and networks;
32
•
•
Providing mechanisms for individuals to manipulate/modify/develop/enhance and organise
things (e.g. photos, videos, favourites); and
Providing notifications, suggestions and alerts of items that may be of interest (e.g. videos
that match or align to previous video search queries; the recent activities of others within
the network)
Access permissions categories used by the Rights and Rewards in Blended Institutional
Repositories project (http://rightsandrewards.lboro.ac.uk/index.php?section=1): include
•
•
•
•
Private (just for own use);
logged on user access;
restricted open access; and
public access.
Key References
Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan, and
P. MacGillis (Eds.), The guidebook for performance improvement: Working with individuals and
organizations. San Francisco: Pfeiffer.
Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating
participation: From EML web site to the learning network for learning design. Interactive
Learning Environments, 13(1-2), 55–69.
Margaryan, A., Milligan, C. & Douglas, P. (2007). CDLOR Deliverable 9: Structured Guidelines
for Setting up LORs. Retrieved September 2, 2007, from http://academy.gcal.ac.uk/cdlor/documents/CD-LOR_Structured_Guidelines_v1p0_001.pdf
Rogers, E. (2003). Diffusion of innovations. New York: Free Press
Recommendations relating to the management of engagement
with the Carrick Exchange
1.3.1. The Carrick Exchange policies and practices should be aligned
strategically with institutional policies and practices.
The Carrick Exchange is being developed for the Australian higher education context. The
governing policies and operational practices of this broad target group are not currently aligned.
However, as a national online service the Carrick Exchange policies and practices will need to
align to those of institutions thereby alleviating obstacles to engagement. It is recommended
that the Carrick Exchange policies and practices:
•
•
•
are developed in consultation with institutions to ensure alignment of policies and practices
particularly in regard to practices of sharing, ownership of resources, intellectual property,
moral rights, and copyright;
are aligned to national, higher education and institutional initiatives; and
offer suggestions for consistency and alignment in the ways that universities engage with
the Carrick Exchange.
33
Engagement with the Carrick Exchange should be recognised by Australian institutions as an
important part of core teaching and learning activities.
1.3.2. The Carrick Exchange should engage with the National Authentication
Framework to ensure integration.
The Carrick Exchange, as a national initiative, should be integrated and authenticated with
existing national infrastructure and technical systems, for example:
•
•
•
The Australian National Data Service (ANDS), part of National Collaborative Research
Infrastructure Strategy (NCRIS) project,
the Australian Access Federation Project, and
the Australian Research Repositories Online to the World (ARROW) project.
Research participants supported the notion of a trusted, easy to use authentication system, one
interviewee explaining:
[There is a need for] spaces outside their own institutions that are easily
accessible and not restricted by institutional firewalls, administrative practices
and other technical constraints. People use what is easy, especially as there are
so many people out there who do not use technology to a great degree.
(Interviewee)
Linking the Carrick Exchange to existing university databases is also important. Seamless
connection and the ability to search the Carrick Exchange when also conducting institutional
database searches was identified in the ascilite research findings as an important design
feature. This integration of authentication and databases will simplify the number of search
systems that users have to understand and operate.
1.3.3. A strategy should be developed to support, encourage and manage
champions to promote the Carrick Exchange.
Promoting a “human face” to the Carrick Exchange will be a key success factor. Developing
networks and communities will require people as well as technology. Advocates and champions
from within the Carrick Institute and the higher education community should be identified.
These champions are critical as leaders, driving and promoting the Carrick Exchange and
enlisting help from others within institutions and the sector to encourage and facilitate
engagement.
Carrick Fellows and grant holders may be potential champions. However, champions should
not be just the current Carrick grant holders, or people well-known in the higher education
sector. Engagement with the Carrick Exchange has to go deeper and broader than this so
choosing people with the right attributes to be advocates and supporters, and resourcing them
adequately is important. There should be a good balance across disciplines, gender and role,
including faculty based academics, support staff and librarians. Educational developers, staff
developers and librarians may be the most common and frequent users of the Carrick
Exchange. They are potentially important institutional champions along with senior leaders in
teaching and learning.
34
1.3.4. Partnerships with Australian universities should be established to ensure
longevity and sustainability of the Carrick Exchange.
Managing the issues of sustainability and ongoing support for the Carrick Exchange is critical.
Carrick Exchange resources and functions that support communication and collaboration
across the higher education sector must be well maintained into the future. The investment of
financial and human resources to establish and operate the Carrick Exchange must not be lost
as a consequence of limited long-term planning. As one repository representative noted:
One thing that has been learnt from the experience of CLOE is that there is a need for
long-term stable funding, not reliant for example on grants and project grants but
possibly some sort of Government funding. (Interviewee)
Strategies for the Carrick Exchange to maintain momentum, currency, quality and relevance
after the initial phases of implementation should be negotiated and explicitly established across
the sector. A Memorandum of Understanding should be developed to establish the ongoing
commitment and a partnership with Australian universities to ensure the longevity and
sustainability of the Carrick Exchange.
1.3.5. Collaboration should be encouraged and supported as a means of
fostering engagement and contribution to the Carrick Exchange.
While institutions are keen to collaborate, there are issues of competition in resource
development that can hinder collaborative ventures. The Carrick Institute is already playing a
key role in addressing this dilemma through the Carrick Institute Grants Scheme 9 . A clearly
articulated goal of the programs in this scheme is “to facilitate innovation and collaboration”.
The Grants Scheme is just one way that the Carrick Institute is addressing and establishing a
culture of sharing and collaboration across the sector.
The Carrick Exchange is also supporting this culture in the provision of a service that supports
communication and collaboration. Use of the Carrick Exchange for collaboration and
communication in Carrick projects should be promoted and recognised. There was even the
suggestion by many of the ascilite research participants that the Carrick Institute should make it
conditional that Carrick Institute grant holders contribute to the Carrick Exchange as part of
their obligations in return for financial support.
1.3.6. The Carrick Exchange should support the needs of networks and
communities of practice.
The target audience for the Carrick Exchange will comprise existing networks and communities
of practice within the higher education sector. These networks and communities have
communication and collaboration needs that the Carrick Exchange may be able to support. The
Carrick Exchange should:
1. Identify the specific needs of user groups - networks and communities; and
2. In partnership with the network/community integrate the resources and functions of the
Carrick Exchange that address the identified needs of the group.
As well as addressing the specific needs of existing networks and communities, the Carrick
Exchange should provide:
9
http://www.carrickinstitute.edu.au/carrick/go/home/grants
35
•
•
•
•
Details of networks and communities of practice, and the facility to contact members of
these groups;
A network of contacts within the higher education community that can contribute to and
inform areas of interest to individuals;
Links to resource creators and their related networks; and
Alerts about new community items and shared spaces.
1.3.7. An evaluation plan should be developed to measure levels of engagement
including participation, collaboration and contribution.
Regular evaluation of the Carrick Exchange, measurement of uptake and engagement to inform
ongoing development and maintenance of the repository and its community should be
undertaken. There is a need to have processes to report how those extracting content from the
repository are using that material. This information would be very useful in demonstrating the
importance and value of the repository and community to key stakeholders and potential
funders. Mechanisms to track and evaluate how useful the forums and other communication
exchange areas are to members would also be advantageous. The project’s evolution should
be tracked from the earliest stages. Other repository managers have reported that this is a key
element in developing a successful service. If managed through traditional research practices,
this information will ensure that the ongoing lessons learned can be properly documented and
evaluated.
The Carrick Exchange has to be able to demonstrate its value if it is to be regarded a success.
The value of the Carrick Exchange has to be seen in relation to the needs of the primary target
audience – university staff. Therefore it is important that the evaluation plan gather information
specifically detailing the uptake and engagement of these users.
Strategies to manage engagement
Strategies for the Carrick Institute and the Carrick Exchange
1. Invite universities to identify an institutional representative to be part of an expert reference
group to inform ongoing aspects and activities of the Carrick Exchange, including the
development of Carrick Exchange policies and practices and to encourage and support
institutional “buy in”, collaboration and partnership with the Carrick Exchange.
2. Promote engagement by using Carrick Exchange staff, institutional champions or project
coordinators who “travel around” and work with academics and support staff, within
institutions.
3. Support secondments for individuals to take on specific Carrick Exchange promotional and
operational roles, e.g. peer reviewing, holding events, publicising resources, putting groups
and individuals in touch with one another, providing regular updates and newsletters.
Thereby presenting a “human face” to the Carrick Exchange for enhanced engagement.
4. Ensure that rigorous data collection of the project’s evolution is facilitated from the start.
Managed through traditional research practices, this information should be properly
documented and a critical research lens applied to the processes, resources, development
schema and workflows to be evaluated.
Strategies for the higher education sector
1. Support the development, promotion and management of the Carrick Exchange in
partnership with the Carrick Institute
2. Enable institutional champions to engage in one on one conversation and dialogue with
intending users to enhance engagement, through a “just-in-time” approach.
36
3. Enable uptake of secondments from the institutional point of view, for individuals to take on
specific Carrick Exchange promotional and operational roles.
4. Support Champions to develop an institutional network to promote the value and worth of
the Carrick Exchange.
Technical strategies
1. Develop mechanisms for reporting how those extracting content and resources of any kind
from the Carrick Exchange are using that material. The information collected should be able
to demonstrate the importance and value of the Carrick Exchange and community to key
stakeholders and potential funding bodies.
2. Ensure users’ contact details are registered when downloading a resource from the Carrick
Exchange.
3. Develop mechanisms to enable an annual audit of resources that provides concrete
measures and details of the various resources.
Exemplars
LORN, Learning Object Repository Network Coordinators: These coordinators in the VET
sector, as part of the Australian Flexible Learning Framework, act as champions in their state
jurisdictions. These champions encourage and support members of the VET community to
access high-quality teaching and learning resources that support flexible delivery from the
learning object repositories.
http://lorn.flexiblelearning.net.au/Home.aspx
Partnerships with current institutional repository owners: Institutional learning object repository
owners such as Deakin, RMIT, Griffith, Monash etc., should be approached to establish links
with the Carrick Exchange, along with any research repositories associated with the ARROW
project.
http://arrow.edu.au/repositories
Key References
Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities.
D-Lib Magazine, 13(5/6). Retrieved September 27, 2007, from
http://www.dlib.org/dlib/may07/henty/05henty.html
Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating
participation: From EML web site to the learning network for learning design. Interactive
Learning Environments, 13(1-2), 55–69.
Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible Learning
Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian Flexible
Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from
http://flexiblelearning.net.au/aboutus/keydocuments.htm
Wenger, E., McDermott, R. & Snyder, W. (2002). Cultivating communities of practice: A guide
to managing knowledge. Boston MA: Harvard Business School.
37
Recommendations related to ease of use
1.4.1. The Carrick Exchange should devise strategies of engagement for the
varying levels of technical capability of users.
Promoting the benefits and affordances of the Carrick Exchange must be supported by the
positive experiences of users. If the engagement of users is not positive they will not return.
This was a view firmly held by one interviewee who stated:
One of the dangers for engagement is that if the promotion of the repository
commences before the system is working efficiently and effectively, users are not
just put off for that time, they're put off always. This applies to both accessing and
contributing to the repository. (Interviewee)
The fundamental requirement for ensuring positive engagement in the Carrick Exchange is
ease of use of the system, as one interviewee, in echoing the sentiments of numerous research
participants explained: “[The Carrick Exchange] needs to be as quick and easy and simple as
possible to use” (Interviewee). There will be varying levels of technological skill, ability and
experience across the target audience. Therefore, the experience and expertise of all users
should be addressed in the functionality of the system and the additional support provided not
only through documentation but through personal assistance.
1.4.2. The Carrick Exchange should be easily accessible and intuitive to use.
The Carrick Exchange should be easily accessible and not restricted by institutional firewalls,
administrative practices and other technical constraints. Research participants were adamant
that unless these barriers to engagement were removed people would not use the Carrick
Exchange. As one interviewee noted:
One of the other hindrances to using a repository, and ultimately for engagement is that
if the process of finding something, grabbing it and reusing it is too difficult, if it's not
intuitive, users will be put off. (Interviewee)
People use what is easy. Many of the target audience may be those who do not use technology
often, consequently the mechanism for conducting a search and using the functions of the
Carrick Exchange must be intuitive.
The Carrick Exchange needs to have an intuitive common sense approach to the
search engines and architecture of the site. (Interviewee).
In the early stages, all searches should return something useful, and collections that are low in
resources should not be advertised or made available until there is a critical mass of materials
or commentary. Ease of use should also extend to seamless authentication and straightforward
metatagging procedures.
Strategies to ensure ease of use
Strategies for the Carrick Institute and the Carrick Exchange
1. Provide training and support with respect to: technical capability; the use and operation of
the Exchange; and development of teaching and learning expertise and practices to
integrate the interactive and innovative resources available through the Carrick Exchange.
38
2. Work in partnership with universities to provide collaborative spaces that are easily
accessible and not restricted by institutional firewalls or administrative practices.
Strategies for the higher education sector
1. Institutional champions should provide and complement training and support offered by the
Carrick Exchange with respect to: technical capability; the use and operation of the Carrick
Exchange; and development of teaching and learning expertise and practices.
2. Assist the Carrick Exchange to provide users easy access to the system that is not
restricted by institutional firewalls or unnecessary administrative practices.
3. Provide details of networks and communities of practice to be advertised in the Carrick
Exchange.
4. Provide details of a network of contacts within the higher education community that can
contribute to and inform areas of interest.
Technical strategies
1. Ensure that the Carrick Exchange is easily accessible and not restricted by technical
constraints.
2. Provide the facility to enable users to contact members of communities through functions
within the Carrick Exchange.
3. Maintain an up-to-date tour of the Carrick Exchange (e.g. tutorial, overview) with examples
of specific actions, searches and how people are using the Carrick Exchange.
4. Include an online help function.
Exemplars
Facebook: An online social networking environment that promotes engagement and interaction
amongst members through resources and functions including:
•
•
•
•
a space in which items such as photos can be stored and shared with friends;
networks for users to join, and communicate and share with;
web-based games and social interaction activities; and
a support system for external parties to create plug-ins to enhance the system.
Facebook originally developed in the college sector in North America. It signs up more than a
million new members every month (AFP, 2007). Facebook reports that it currently has 200,000
people sign up to it every day and has amassed 42 million users in the three years since it was
created in 2004. Facebook is an intuitive, easy to use environment that caters to the varying
technological abilities of members. It offers users control over privacy settings for their profiles,
photos etc.
http://www.facebook.com
Intute ease of use: The Intute repository is easy to navigate and locate content.
http://www.intute.ac.uk/
Librarything.com: Cataloguing of books online. Works like an international book club. Uses
social networking software to create groups, blogs, personal profiles, tags and links to others
with similar interests
http://www.librarything.com/
Wikipedia: Free online collaborative encyclopaedia. Members create the content which is all
fully editable. Their code of conduct assumes that users will behave well. When they do not the
community tends to work to improve matters. Generally their code assumes respect for other
members even when you may not agree with them; civility and openness amongst members;
avoidance of personal attacks and use of sweeping generalisations. Some of Wikipedia’s
39
services include help and reference desks, volunteers to answer questions, a community portal
and a “village pump” for discussions about the site itself, site news and a “local embassy” for
communication in languages other than English.
http://en.wikipedia.org/
Key References
AFP. (2007). Facebook to post members profiles on online search engines. Retrieved
September 27, 2007, from
http://afp.google.com/article/ALeqM5iWsMIytjvcGtnN9Q6ob6X3YHC3BQ
Hand, T., Gosper, M., Woo, K., Gibbs, D., Kerr, S. & Rich, D. (2004). Learning objects: User
perspectives on the conditions surrounding their use. Proceedings of the Ed-Media 2004 World
Conference on Educational Multimedia, Hypermedia and Telecommunications, 21-26 June,
Lugano, Switzerland, 66–72.
Margaryan, A., Currier, S., Littlejohn, A. & Nicol, D. (2006). Community dimensions of learning
object repositories, CD-LOR deliverable 1: Learning communities and repositories. Retrieved
August 30, 2007, from http://academy.gcal.ac.uk/cdlor/CDLORdeliverable1_learningcommunitiesreport.doc.
Margaryan, A. & Littlejohn, A. (2007). Cultural dimensions of learning object repositories.
Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and
Telecommunications, Vancouver, Canada, 25 -29 June, 4335-4343.
Mckenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, adoption and
adaptation of project innovations in higher education. A report for the Carrick Institute for
Learning and Teaching in Higher Education. Sydney: University of Technology.
Recommendations related to promoting engagement with the
Carrick Exchange
1.5.1. There should be a strategic plan and adequate support for the promotion
and publicity of the Carrick Exchange to ensure successful engagement
by the sector.
Promotion of the Carrick Exchange is critical to successful engagement by the sector. The
Carrick Exchange needs to be proactively promoted and publicised within universities to ensure
members of the sector engage with it, know where it is and how to use it. Effective engagement
strategies and adequate resourcing are required in support of this.
Promotion and publicity of the Carrick Exchange should be strategic and include:
•
•
Communication of information about the benefits and affordances of the Carrick Exchange;
and
Promotion of the rewards that ensue from engagement with the Carrick Exchange.
Many of the research participants recommended that promotion of the Carrick Exchange was
imperative for successful engagement with comments from interviewees such as:
40
It is important to promote the repository especially in face-to-face forums. Taking the
time to go out and tell people about the repository and explain the details about the
repository is very important. (Interviewee)
A strategic plan for short- and long-term promotion and publicity of the Carrick Exchange
should be developed. Once developed it is critical that this plan is adequately resourced (both
financially and with personnel) to ensure that the strategic goals and objectives of the plan are
met.
1.5.2. There should be regular opportunities to disseminate and encourage
engagement of the higher education sector with the Carrick Exchange.
The ascilite research findings identified “push” technology (e.g. alerts to notify when new items
have been submitted in a particular area of interest or by a particular person and regular
updates, such as electronic newsletters which inform members of recent developments and
contributions) as an important design feature for the Carrick Exchange. One interviewee
commented:
A lot of good things get created that people don’t ever know about and unless
individuals go and look at the Exchange they will not know what is available to
them through it. So the Exchange needs to have some sort of push mechanism
that brings to the attention of users new and worthwhile resources. (Interviewee)
As a strategy for promoting engagement “push” technology can facilitate and sustain
discussions or remind and inform members and potential users about the Carrick Exchange, its
resources and functions. Members should be able to adjust the level of engagement and
amount of publicity they receive.
1.5.3. Web 2.0 technologies should be promoted and supported as a
mechanism for enhancing engagement with the Carrick Exchange.
The Carrick Exchange should provide access to communication technology (e.g. web
conferencing for virtual meetings, technological support tools with automated feedback on
learning), and Web 2.0 tools for collaboration and social networking. Through harnessing the
affordances of the technology and engaging elements that may not be available in other
repositories and collaborative spaces, the Carrick Exchange will become an exemplary model
for the higher education sector; as such the provision and promotion of these technologies as a
supported feature of the Carrick Exchange is important.
According to Evans (2007) the critical elements for successful use of social software are:
1. “Authenticity” – a real and established need to use the software;
2. “Relevancy” – its use is relevant to the need, and appropriateness for the client; and
3. “Support” – for the software within an “enabling culture”.
These elements for successful use of social software should inform and be addressed in the
overall Carrick Exchange strategic plan for promoting and enhancing engagement.
41
Strategies to promote engagement
Strategies for the Carrick Institute and the Carrick Exchange
1. Facilitate an annual Carrick Exchange conference to promote and publicise the resources
and functions available to potential users to support communication and collaboration
across the sector.
2. Promote the Carrick Exchange at all face-to-face forums.
3. Support champions who are driving and promoting use of the Carrick Exchange in
institutions.
4. Promote the benefits of contributing to the Carrick Exchange such as contributing to: the
advancement and sustainability of teaching and learning; the sharing of resources and
ideas with colleagues within and across disciplines; project dissemination at national and
international levels; promotion of achievements and scholarship in resource development;
the sharing of ideas about teaching and learning; making more accessible and archiving
publicly funded project output; and networks and communities of practice, both disciplinary
and cross-disciplinary based.
Strategies for the higher education sector
1. Promote the Carrick Exchange at all face-to-face forums.
2. Build use of the Carrick Exchange and its resources into existing institutional programs for
staff (e.g. Foundation teaching and higher education programs, and Graduate Certificates).
3. Enable champions who are driving and promoting use of the Carrick Exchange in
institutions.
4. Provide professional development funding to promote the Carrick Exchange in local and
institutional forums.
5. Showcase achievements, expertise and effective use of resources as an important means
of promotion and publicity.
6. Disseminate information about the Carrick Exchange via regular online newsletters with
links to stories and updates about projects.
7. Follow the strategies suggested by Margaryan, Currier, Littlejohn and Nicol, (2006, pp.4-5)
(see Literature Review section of this report) to address socio-cultural issues, pedagogic,
organisational and information management, and technological issues.
Technical strategies
1. Ensure the Carrick Exchange is continuously refreshed by establishing mechanisms and
processes to keep resources, contact details, network and communities’ information and
system functions current.
2. Identify on the Carrick Exchange ‘”Resources of the Week/Month” with a rating, based on
resources that are most often accessed or the newest peer reviewed resources.
3. Provide user scenarios detailing how the Carrick Exchange has been used in different
contexts. Scenarios could include details about design, contexts of use, implementation
exactly what and how people interact with the system and each other.
Exemplars
Amazon.com: This e-commerce site has a system that greets registered users by name and
delivers recommendations and updates about available products. The user can collate chosen
resources from the site, and develop their own profile of recommendations, ratings and
preferences. The system monitors users’ preferences, locates resources based on previous
searches and feeds back information that connects users with other like-minded members. A
mix of these features may be of value to the Carrick Exchange.
http://amazon.com
42
MERLOT: Uses similar techniques to support its educational community, with a strong focus on
disciplinary communities. MERLOT also holds regular conferences.
http://www.merlot.org
CLOE stories: Examples of learning object development and case studies.
http://cloe.on.ca/stories.html
Key References
Evans, V. (2007). Networks, connections and community: Learning with social software.
Australian flexible Learning Framework, Australian National Training Authority and
Commonwealth of Australia. Retrieved September 27, 2007, from
http://www.flexiblelearning.net.au/flx/go/pid/377
Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. & Koper, R. (2005). Facilitating
participation: From EML web site to the learning network for learning design. Interactive
Learning Environments, 13(1-2), 55–69.
Recommendations for engagement policies
Policies will need to be developed in the following areas. Barton and Waters (2004) in the
LEADIRS Workbook also have some useful tips about policies.
1. Policy for national integration with Australian universities seeking executive support for all
activities and functions of the Carrick Exchange, linking contribution with recognition and
career advancement.
2. Membership policy:
a. Who can be a member? e.g. the primary members are those who teach, manage and
lead learning and teaching in Australian higher education, and includes individuals,
institutions, consultants, professional associations and organisations. Other target
members may be international guests, and possibly students.
b. Preferably management of membership via trusted authentication, not self-asserting
systems.
3. Terms and conditions of use policy covering:
a.
b.
c.
d.
e.
f.
Vision statement giving guidance and direction overall;
Acceptable use;
Appeals policy;
Use of tools within the site;
Submission policy;
The standard of the content (digital assets and commentary) that may be contributed or
expected to be found on the Exchange;
g. Copyright, intellectual property, and moral rights;
h. Privacy and permissions;
i. Accessibility.
4. A code of conduct similar to the Wikipedia code should be adopted
(http://en.wikipedia.org/wiki/Wikipedia:Policies_and_guidelines), and an appeals policy for
disputes is required.
43
a. Respect other members even when you may not agree with them.
b. Be civil and act in good faith. Be open and welcoming.
c. Avoid making personal attacks or sweeping generalisations.
5. A reporting policy, stating how the Carrick Exchange:
a.
b.
c.
d.
e.
Tracks usage;
Reports on uptake;
Provides members with feedback on usage;
Protects privacy;
Evaluates development of the system over time.
6. Policy for engaging different stakeholders and discipline bases:
a. A funding and management model to manage initial engagement, allowing for future
scalability and sustainability.
b. Procedures to target special groups or individuals to establish an initial critical mass of
users of the system.
c. Management of Carrick Champions and champions within the sector.
d. Multiple strategies to manage staff engagement at all levels of universities.
e. Procedures to manage and generate activities, forums, workshops, conferences with
discipline and cross-discipline focus.
7. Support policies to encourage uptake, integration into everyday praxis within the higher
education sector, and management of educational and cultural change targeting:
a. Educational and staff support
b. Rights management, searching and metadata support, i.e. librarians.
8. Active dissemination policies that acknowledge:
a. Excellent contributions;
b. The latest contributions to the Carrick Exchange;
c. Award of the month and an annual award that is presented with Carrick citations or
integrated with the criteria.
44
Resource identification and contribution guiding
principles and recommendations
______________________________________________
Introduction
As part of the development of the Carrick Exchange ascilite was engaged to conduct research
into:
•
What resource contribution and identification methods will engage and encourage users to
contribute and collaborate within the Carrick Exchange?
o
o
o
o
What are the key factors to support resource contribution, identification and reuse?
What types of resources will be contributed?
What recommended policies and procedures are currently in use?
What are the policy implications for the Carrick Exchange?
The following protocols and mechanisms for resource identification and contribution for the
Carrick Exchange have been synthesised from knowledge and expertise existing in: the
literature, current practice, exemplary community development to foster engagement with
online services, and key practitioners and networks in both national and international settings.
As well as the communities, networks and workspaces expected to establish around the Carrick
Exchange, a repository of quality learning and teaching materials will be developed. Carrick
Exchange resources will either be deposited on the site or linked from websites and databases
elsewhere. The term “resources” is taken to include a broad range of formal and informal items
in digital format, e.g. products, processes, learning objects and designs, activities, teaching
strategies and tips, reports, assessment criteria, reviews, discussions on issues relating to
leading and managing teaching. In addition, people are very important to the Carrick Exchange
and considered “resources” themselves.
In general, defining and maintaining the collection as a quality resource that meets the needs of
the sector, overcoming barriers to contribution, and finding the right incentives that will reward
contribution of digital resources and commentary will be major challenges for the Carrick
Exchange.
Recommendations in this section are structured around the following key themes that emerged
from the ascilite research findings:
1.
2.
3.
4.
Resource identification
The culture of sharing in higher education
Rewards, incentives and recognition for resource contribution
Intellectual property (IP) and rights management
5. Management of resource contribution
45
Principles related to resource identification
2.1.1. The Carrick Exchange should strategically target resource contributors.
By identifying key contributors to the Carrick Exchange the likelihood and chance of resources
being submitted to the collection is enhanced. These key contributors can themselves be
regarded as ‘resources’. Specific groups or individuals who can make valued contributions
should be targeted including e.g. discipline groups; Carrick Institute grant holders; and;
educational and staff developers. A key finding of the interviews with practitioners in the field
was that the Carrick Exchange could have an important role providing shared workspaces for
collaborative and cross-institutional groups. There is also an emerging and important role for
librarians regarding resource discovery and identification. Librarians are well placed to identify
and target special contributors and disseminate knowledge about the system. Support and
professional development staff within universities are also likely to identify where crossdisciplinary connections can be made that link with Carrick Exchange initiatives. Further,
Carrick Exchange Champions could network across institutions, seek out key academics and
support staff within institutions and professional associations to act as mentors and initiate
contribution to the Carrick Exchange. Carrick Fellowships may be one way in which particular
resource development could be commissioned and promoted.
The notion of identifying resources worthy of publishing to the Carrick Exchange and sharing
across the sector should be carefully and strategically introduced, as this is not yet common
practice. It might be advantageous to start small and target cross-institutional projects with a
need for collaborative workspaces. Also, sharing may be more likely to occur in programs
where there is team teaching and large student cohorts. Finally, one of the key objectives of the
Carrick Institute is to ‘raise the profile and encourage recognition of the fundamental importance
10
of teaching in higher education institutions and in the general community’. This objective
could be achieved through the engagement and commitment of the higher education sector, to
the Carrick Exchange. A fundamental factor for this is to ensure that those who contribute
towards resource identification are adequately recognised and rewarded.
2.1.2. Existing resources that could be contributed or linked to the Carrick
Exchange should be identified.
One of the keys to the success of the Carrick Exchange is the range, extent and quality of the
contributed resources. If the resources meet the needs of users and are easy to access, then
the Carrick Exchange will become known as a central ‘hub’ for teaching and learning resources,
and the associated supportive communities and technologies. Locating resources that give
insight into exemplary pedagogical practice is important:
It’s not the stuff per se that matters … what’s really important is the creation of the
design, the approach, the tool - with good examples of application where other people
can come in and share their view and their examples. (Focus group participant)
Initially the Carrick Exchange should link to other existing repositories rather than store large
numbers of resources on the site. The identification of resources for the Carrick Exchange
should align with the needs of members and the Carrick Exchange should continue to evaluate
the relevance and currency of the collection. A discipline focus is important from the outset:
For the Carrick Exchange to have as wide an impact and accessibility as possible it
needs to have disciplinary presences so that users can see that the resources within
the Carrick Exchange have relevance to them and their needs. (Interviewee)
10
http://www.carrickinstitute.edu.au/carrick/go/home/about
46
Resource content types that the Carrick Exchange could aim to identify as a priority are:
•
•
•
•
•
Professional development and teaching support documents that are the outcomes of
Carrick Institute projects and can be made publicly available;
Reports and updates on Carrick Institute funded projects in their development stages - prior
to completion, as well as final project reports;
Resources characterised as ‘works in progress’, i.e. developing resources, which may be
linked to and open for comment from select members to assist and inform development;
Secondary supporting documentation – this documentation supporting specific resources
could outline design, development, use, and contexts of use and re-use and function as a
guiding framework to accompany some resources contributed to the Carrick Exchange; and
Generic teaching resources e.g. tips and strategies, learning designs, assessment ideas,
problem-based and case-based learning and group management resources, laboratory
activities, and research-style projects.
Strategies for advancing resource identification
Strategies for the Carrick Institute and the Carrick Exchange
1. Facilitate conference days or workshops with the aim of identifying resources and potential
contributors (using the Camp CLOE model for inspiration). Different user groups should be
targeted for different days and events.
2. Identify a range of high quality resources from existing institutional and higher education
repositories and collections to provide a critical mass of resources in the initial phases of
the Carrick Exchange.
3. Provide seed funding for the development of resources for the Carrick Exchange, targeting
contributors to produce resources likely to be reused.
4. Target Carrick Grant holders to produce resources for the sector as part of the ‘exchange
process’, in return for the investment granted to them by the Carrick Institute.
5. Target collaborative teams and programs, e.g. discipline based initiatives, academics
teaching large classes, and cross-institutional projects for contributions to the Carrick
Exchange.
6. Adopt the MERLOT Teaching Commons model whereby champions and task forces work
with discipline-based teams to create a critical mass of resources for publication as a
collection. Partner status with MERLOT allows for reciprocal services between the
institution and MERLOT regarding implementation.
Strategies for the higher education sector
1. Identify existing high quality resources in repositories and collections for contribution to the
Carrick Exchange.
2. Target institutional collaborative teams and programs, e.g. discipline based initiatives,
academics teaching large classes, and cross-institutional projects for contributions to the
Carrick Exchange.
Technical strategies
1. Provide shared workspaces that support collaborative and cross-institutional teams.
2. Provide mechanisms for individuals and groups to nominate resources they have identified
for possible contribution to the Carrick Exchange.
47
Exemplars
California State University (CSU) Teaching Commons (USA), which links to MERLOT.
http://www.cdl.edu/cdl_projects/teachingcommons_home
CLOE (Canada): Co-operative Learning Object Exchange: Camp CLOE is an intensive
collaborative training and resource development event for partner institutions (for faculty
developers, instructional designers, faculty, learning centre personnel, summer student workers
etc.) to assist participants in the design of learning objects. Collaboration on research and
learning object creation is the focus.
http://tlc.uwaterloo.ca/projects/cloe/CaseStory/
Intute (UK): See their collection and development framework (quality consistency and
interoperability). Intute employs a network of subject specialists to find, evaluate and catalogue
the best resources of the Web.
http://www.intute.ac.uk/policy.html
See also the guidelines for linking the repository and searches of the content to personal web
pages.
http://www.intute.ac.uk/integration/
MERLOT Teaching Commons (USA): Exemplary discipline based method for identifying and
sharing content and ideas.
http://www.merlot.org/merlot/materials.htm
See also MERLOT collections:
http://taste.merlot.org/merlotcollection.html
Key References
Australian Partnerships in Sustainable Repositories APSR. Populating a repository – selected
references (bibliography). http://apsr.anu.edu.au/apsrfw/rr/index.php?pid=94
Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of
learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The
COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119).
Macquarie University E-Learning Centre of Excellence: Sydney, NSW.
Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities.
D-Lib Magazine, 13(5/6). Retrieved September 27, 2007, from
http://www.dlib.org/dlib/may07/henty/05henty.html
Margaryan, A., Milligan, C. & Douglas, P. (2007). Community dimensions of learning object
repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object
repositories. Retrieved September 10, 2007, from http://academy.gcal.ac.uk/cdlor/documents/CD-LOR_Structured_Guidelines_v1p0.pdf
48
Principles related to addressing the culture around sharing in
higher education
2.2.1. A strategic plan for establishing and maintaining institutional
partnerships for sharing between institutions and the Carrick Exchange
should be developed.
The Carrick Exchange should implement institutional partnerships with Australian universities
binding both in a relationship of shared responsibility. This should assist in the recognition and
management of issues around sharing, engagement of the sector, rights management and
resource maintenance. Universities should be encouraged to understand their mutual role
supporting the Carrick Exchange and being supported by it. There should be a commitment
from universities that a proportion of teaching and learning resources developed using
university funds will be made available through the Carrick Exchange.
As one interviewee said about the Carrick Institute:
The scholarship of teaching brings academics together. The Carrick Institute has
brought the concept of learning and teaching more status since its inception in 2004, so
the Carrick Institute should use its credibility, status and power in the sector to initiate
change in the culture. (Interviewee)
2.2.2. The Carrick Exchange should systemically address the culture
surrounding sharing in higher education.
A key finding of the interviews was that to engage the sector, effort and resources should be
put towards managing cultural and pedagogical change around the issues of sharing. Those
already familiar with the management of repositories said that addressing the culture of
universities was more time consuming than training people in the use of the system. Sharing
learning and teaching resources is not common practice within higher education and
competition amongst academics and institutions is a perceived barrier to widespread sharing.
However, there is a culture of sharing for the greater good that some interviewees
acknowledged should be promoted to individuals and their institutions. There is also evidence
of sharing in some of the science-based disciplines (engineering, chemistry, astronomy) and
quite often in management. Those more cautious in the sector will align themselves with this
key practitioner:
People in an academic environment are so used to thinking around the idea of IP and
are so nervous about how polished their material is and whether it should go out if it’s
not polished. So there needs to be a staged process in the movement to a sharing
culture. (Interviewee)
However, others may see the following correlations with scholarly publication:
If like journal articles, teaching resources are developed and given away through
‘repository sharing’, and assessed via peer review, this will reflect positively on the
individual for tenure and promotion, the quality of the university and the quality of
teaching at that university. (Interviewee)
The growth of institutional repositories (e.g. Deakin, RMIT, Monash and Griffith), and the
development of scholarly research repositories to meet Research Quality Framework (RQF)
requirements will raise awareness about sharing within the sector. Key to changing the culture
will be strategies that focus on promoting the benefits of sharing and clarifying the rewards. The
Carrick Institute carries prestige and has been urged by interviewees to use its place in leading
teaching and learning to effect cultural change. Change management strategies must be
49
adopted progressively and consistently. “We need to align technology with existing practice in
order to facilitate this change and not hope that the technology will change existing practise”
(Bates, Loddington, Manuel & Oppenheim, 2006, p.25).
While there is a foundation of practice in sharing resources across institutions via discipline
based repositories, some institutions will see sharing courseware which could be reused by
competitors as problematic. These same institutions may be more likely, however, to share
generic resources that do not raise the same concerns. There is also a skill set around sharing
that must be acquired before resources can be effectively reused (Littlejohn, 2003). This
includes skills in locating and contextualising relevant resources, and designing resources for
reuse. Skill development and pedagogical support should be addressed at the institutional
level.
2.2.3. Networks and partnerships should be nurtured as a means of fostering
contribution to the Carrick Exchange.
Effort should be directed towards proactively facilitating the development and continuity of
networks and communities of practice engaged in sharing and developing high quality
resources around the Carrick Exchange. Networks developed elsewhere may not at first see
advantages in relocating to the Carrick Exchange. Professional development and support is
likely to be required to assist the higher education community in making best use of the Carrick
Exchange for individual and specific group contexts and purposes.
Strategies to address culture change
Strategies for the Carrick Institute and the Carrick Exchange
1. Develop communicate the benefits of the Carrick Exchange to institutions and the sector,
and to support new pedagogical processes and relevant strategies. Follow the LORN
proactive methods of engagement (see Exemplars).
2. Clearly communicate information about availability, stages of development and future plans
of the Carrick Exchange and its sharing and reuse polices with key stakeholders and
interested parties .
3. This approach including resource development support, is regarded as a successful
formula for implementation (Nicol et al., 2004 in Weedon, Bricheno & Chidwick, 2004).
4. Begin with the notion of resource management and/or commentary on resources, then
move to the newer notion of sharing (RMIT and Loughborough Universities’ models).
5. Provide and support training in the Carrick Exchange with respect to:
•
•
•
Technical capability;
The use and operation of the repository; and
Development of teaching and learning expertise and practices to integrate the
interactive and innovative resources available through the repository.
6. Target strategies to address cultural and pedagogical change for sharing, across the
diverse range of members, by considering the various roles, groups and levels of
awareness:
•
•
Roles – e.g. academics, educational and staff developers, librarians, senior and middle
managers, sessional staff, research officers, support staff, staff new to teaching;
Groups – communities, communities of practice, networks, disciplines, organisations,
and professional associations; and
50
•
Levels of awareness – aware, curious, envisioning, trying out, using (Dormant, 1997)
(see Table 2 in Engagement guiding principles and recommendations section of this
report).
Strategies for the higher education sector
1. Implement consistent national policies across institutions to support equitable contribution
of resources to the Carrick Exchange.
2. Support cultural change from the top providing leadership and credibility, as well as from
the bottom up, supporting those willing to take risks, and providing resource development
support.
3. Support activities which promote mentoring and skills development in reuse of digital
resources.
4. Support new and beginning teachers, associate lecturers and demonstrators, who may be
more inclined to reuse resources that they have not created.
Exemplars
Learning Object Repository Network project (LORN): Flexible Learning Coordinators within the
VET sector act as champions for their state jurisdiction, encouraging and supporting
contribution and use of the repositories. Project representatives work actively to integrate and
network the repositories across four states of Australia: DET NSW, the NSW Centre for
Learning Innovation (CLI), TAFE NSW, TAFE South Australia, TAFE Victoria, TAFE Tasmania
and the Flexible Learning Toolboxes Project. LORN’s mission is to link repositories with
teachers. There is also a technical link across the repositories via the LORN search facility.
MIT Open Courseware (OCW): Open courseware initiative began in 2001. Anyone can
download materials (students or teachers). 1700 courses shared.
http://ocw.mit.edu/OcwWeb/web/home/home/index.htm
RUBRIC: A project for smaller and regional universities. The Toolkit provides a structured
framework for partner institutions to evaluate, trial and implement an Institutional Repository
solution.
http://www.rubric.edu.au/
The University of Ballarat (TAFE): Example of an institution which places all resources
developed within the institution on the intranet for all teaching staff to use.
VET Learning Object Repository Network (LORN): Proactive work by coordinators to promote a
culture of sharing by seeking out teachers in each state and helping them engage with the VET
repositories.
http://pre2005.flexiblelearning.net.au/projects/resources/2005/vlorn/
Key References
Cohere Group. (2002) The learning object economy: Implications for developing faculty
expertise. Canadian Journal of Learning Technology. 28(3), 121-134.
Ehrmann, S., Gilbert, S. & McMartin, F. (2007). Factors affecting the adoption of facultydeveloped academic software: A study of five icampus projects. Retrieved September 19, 2007,
from http://www.tltgroup.org/iCampus/iCampus_Assessment_Full.pdf
51
Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of
learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The
COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119).
Macquarie University E-Learning Centre of Excellence: Sydney, NSW.
Kenny, J. (2003). A research-based model for managing strategic educational change and
innovation projects. Annual Conference Proceedings of HERDSA (the Higher Education
Research and Development Society of Australasia), University of Canterbury, Christchurch, NZ,
6–9 July 2003, Retrieved September 27, 2007, from
http://surveys.canterbury.ac.nz/herdsa03/pdfsref/Y1102.pdf
Margaryan, A., Milligan, C., Douglas, P., Littlejohn, A. & Nicol, D. (2007). Community
dimensions of learning object repositories, CD-LOR deliverable 10: Recommendations to JISC
for future research and development. Retrieved September 27, 2007, from
http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Final_Recommendations_v1p0.pdf
Stuckey, B. & Arkell, R. (2006). Development of an e-learning knowledge sharing model: 2005
Knowledge Sharing Services Project. Australian Flexible Learning Framework project report.
Funded by Australian National Training Authority (ANTA) and DEST.
Principles related to rewards, incentives and recognition for
contribution
2.3.1. The Carrick Exchange should establish a system of rewards, incentives
and recognition that is valued, appropriate and manageable, and aligned
to institutional policies and practices.
If user needs are addressed, there is an inherent incentive to contribute to the repository.
Additional incentives need to be at both the institutional and national levels to be worthwhile.
Rewards and recognition from the Carrick Institute should be linked to institutional promotional
systems and acknowledged. The Carrick Institute Awards provide a good model which is
influencing internal institutional promotional systems. Provision of expertise on the Carrick
Exchange by individuals should be recognised, and the resulting collaborations and
connections rewarded in some way. Evidence of use and reuse will be important and
mechanisms which support this should be automated where possible. Tracking the number of
accesses and downloads has relevance but is limited in that the information does not describe
how the resources are being used once they have left the repository, or e.g. how useful
engagement in shared commentary was, in informing and changing teaching praxis.
The following are possible rewards and incentives for various forms of contribution to the
Carrick Exchange. The findings from the research indicate that these were probably of most
importance:
•
•
•
•
•
•
Recognition from the home institution for contributions to the Carrick Exchange by providing
time and/or funds, in the same way that time and/or funds are allocated for scholarly
research.
Reports about resources submitted to the Carrick Exchange that could be used for
promotion and tenure applications.
Acknowledgement of the resource creator each time a resource is reused.
Grants to develop and share resources to the Carrick Exchange.
Awards to formally recognise submissions made to the Carrick Exchange.
Acknowledgement for contributions in various forums such as teaching and learning
conferences and forums, and in online newsletters.
52
•
Funds for time release or project development are probably considered preferable to many
staff as a reward, rather than direct salary supplementation. Financial remuneration of this
kind is considered a low priority. This is probably because many staff are engaged in
related activity as part of their brief regarding teaching, but want some recognition and
reward for that effort, not just more work.
Other suggested incentives and rewards:
•
•
•
A trusted system of rights management and IP protection for individuals and their
institutions for shared resources, covering deposit and distribution.
A system that is equally open for consumers and contributors. Staff are unlikely to
participate in a system that requires contribution as a prerequisite before resources, forums
and tools can be accessed.
The Carrick Exchange could act as a broker between individual academics and commercial
customers, but generally the ascilite research indicates that selling resources via the
Carrick Exchange is a low priority.
2.3.2. The Carrick Institute should ensure that the established system of
rewards, incentives and recognition is adequately resourced and
supported into the future.
To be effective, the reward and recognition system should be maintained through stable
funding systems. The funding model for rewards should be ongoing and not project based.
Incentives, rewards and recognition for the effort put towards contributing in any way to the
Carrick Exchange need to be strongly communicated and promoted to the sector. Some of the
exemplars below provide models of awards and award systems that could be adopted.
2.3.3. A strategy for impact and process evaluation should be developed to
assess the relevance and value to the sector of the rewards, incentives
and the system of recognition.
As one of the objectives of the Carrick Institute is to ‘promote and support strategic change in
higher education institutions for the enhancement of learning and teaching’ 11 it will be important
to measure, across the sector, the impact of rewards and incentives developed to recognise
contributions to the Carrick Exchange. The value and impact of these rewards and incentives
for the user, institutions and the sector will need to be assessed. Rewards for scholarly
research output are well established, but the same is not true for teaching and learning output.
A considered change management strategy is required to address this anomaly between
scholarly research and teaching and learning products.
Strategies for valuing, rewarding and recognising contribution
Strategies for the Carrick Institute and the Carrick Exchange
1. Initiate policies from the Carrick Institute that contribute to changes in university policy so
as to link contribution to the Carrick Exchange to promotion and career advancement.
2. Recognise contribution to the Carrick Exchange as a mechanism to support applications for
Carrick and institutional awards.
11
See ‘About the Carrick Institute’: http://www.carrickinstitute.edu.au/carrick/go/home
53
3. Establish awards from the Carrick Institute for excellence in resource development,
contribution and sharing.
4. Create awards for resource and commentary contribution such as ‘Resource of the Month’.
Criteria for this could be based on ratings (comments and star ratings), and the number of
times a resource is accessed. A more prestigious award could be best (formally) peer
reviewed item.
5. Raise awareness amongst university staff involved in promotions and appointments,
through a communication campaign about the Carrick Exchange and what contributing to it
means.
6. Preserve original attribution for the creator of a resource through licensing agreements.
7. Conduct research and evaluation to track reuse. This could be implemented through
targeted Carrick Institute grants, for example. Statistics should be compiled about how the
resources are being used, preferably backed up by periodic qualitative evaluation. This
record allows for follow up so users can be contacted on how they have used a
downloaded resource etc. and provides a measurable record for use in promotion, awards
etc.
Strategies for the higher education sector
1. Certificates noting contribution to the Carrick Exchange and the extent of re-use of
resources should receive recognition and contribute towards teaching portfolios for
promotion and tenure.
Technical strategies
1. Design and implement automated systems within the Carrick Exchange that provide
acknowledgements of the number of resource contributions (including commentary) per
member, along with access details of submitted resources. The system should
automatically track resource contribution, downloads and commentary.
2. Where a contributed resource has been reused and repurposed, the original contributor
could receive acknowledgement for their provision of the initial resource. Perhaps they
could be sent a link to the repurposed resource.
3. Provide access for individuals to be able to retrospectively review commentary they have
initiated and responded to over the long term.
4. Periodically conduct polls asking members about resources they have used or reused.
Exemplars
ascilite awards: The awards recognise the exemplary use of technologies in teaching and
learning in tertiary education. Awards are open to current financial members of this professional
organisation.
http://www.ascilite.org.au/index.php?option=com_content&task=view&id=20&Itemid=36
BetterPhoto monthly competition: BetterPhoto can be regarded as a hobby related learning
community which includes both amateurs and professionals. The community aims to support
sharing of photos, skill sharing and development. Networking opportunities and courses are
available. Small prizes are offered as incentives but recognition and personal sales from
professional sites are the key incentives to contribute.
http://www.betterphoto.com/contest.asp
Carrick Institute Awards: Awards for teaching and learning in higher education. The model
could be extended to the Carrick Exchange.
http://www.carrickinstitute.edu.au/carrick/go/home/awards
Carrick Institute Teaching Quality Indicators Project: Research into rewarding and recognising
quality teaching. Outcomes from the project have relevance to the Carrick Exchange.
http://www.carrickinstitute.edu.au/carrick/go/home/pid/370
54
CLOE: Awards for contribution and reuse.
http://cloe.on.ca/awards.html
MERLOT: ‘Jump start CV’. MERLOT members can export a portfolio of their profile to use for
CV and promotions.
http://www.merlot.org/merlot/members.htm MERLOT also offers the following awards:
•
•
•
•
Exemplary Online Materials Award (peer reviewed);
Volunteer of the Year Award (contribution to the peer review Editorial Boards);
Distinguished Service Award (individual who exemplifies the vision that brings MERLOT
members together creating the MERLOT Community and its services); and
Application of MERLOT awards.
See http://taste.merlot.org/awardsoverview.html
University of Queensland: Investigation into bringing parity between scholarly research and
effort towards teaching.
http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesRe
portApr07.doc
Key references
Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project,
Academic survey: Final Report. Loughborough University & JISC. Retrieved September 27,
2007, from
http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/11374231
50_SurveyReport.pdf
Mckenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, adoption and
adaptation of project innovations in higher education. A report for the Carrick Institute for
Learning and Teaching in Higher Education. Sydney: University of Technology.
University of Queensland. (2007). Report of the Working Party on the Diversity of Academic
Roles. Internal review. Retrieved September 27, 2007, from
http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesRe
portApr07.doc
Principles related to Intellectual Property and Digital Rights
Management
2.4.1. The Carrick Exchange should establish IP and Rights Management
policies and practices that align to institutional policies and practices.
The whole sector must become much more aware of the policies and procedures that are in
place regarding resource ownership and sharing.
There is . . . a sense of proprietary ownership of intellectual property . . . Generally
academics are keen to look at the resources of others but more reluctant when it
comes to sharing their own work (Interviewee).
55
Academics are frequently unclear about their rights over resources created for teaching
purposes. In many institutions (e.g. RMIT), the institution holds copyright on material produced
by staff, but staff retain the moral rights. The Carrick Institute has a key role in aligning these
policies and practices across the sector and in providing leadership for this alignment through
the Carrick Exchange. The Carrick Exchange could promote the approach taken by some
institutions (e.g. Loughborough University) that the university owns the copyright but allows its
employees the right to share resources created during their employment.
2.4.2. IP and Rights Management should address the concerns and needs of
contributors
Some academics fear that credit for creating a resource will be “inaccurately or unevenly” given
away. Limited understanding of copyright issues can lead to concerns over misuse of reused
materials. The practice of sharing will be affected users levels of trust in the system and the
management of intellectual property rights. The concerns of university administrators should
also be acknowledged as one interviewee commented:
There will be resources and intellectual property that will require specific sign off from
the University before broad sharing can be enabled because of the compromise to the
‘competitive advantage’ that sharing may evoke. (Interviewee)
However, one practitioner interviewed from the VET sector acknowledged that teachers are
growing more circumspect about issues of IP and sharing, and teachers in that sector are
learning that resources become obsolete very quickly:
For teachers to cope with this they are realising that they have got to collaborate and
share resources [and therefore find solutions to issues such as IP]. (Interviewee)
2.4.3. The Carrick Exchange should develop systems and structures to ensure
that IP and Rights Management policies and practices are implemented
and appropriate.
The Carrick Exchange should draw from lessons learned in the international higher education
sector where issues of IP and rights management have been investigated and strategies
adopted for use in other repository systems, such MERLOT, CLOE, the Rights and Rewards
project (Bates et al, 2006), and draw on the findings of the OAK Law report. The greater use of
open source software in the future may also bring changes to approaches to licensing systems
that should be monitored.
Strategies to address intellectual property and rights
management
Strategies for the Carrick Institute and the Carrick Exchange
1. Clearly communicate to institutions the policies and protocols for the Carrick Exchange,
being explicit about the need for clarity on institutional positions on IP, copyright, moral
rights and academic exchange and sharing.
2. Devise a communication plan that will reach all staff within the sector, new and continuing,
about the IP, copyright and moral rights that pertain to individuals and institutions regarding
ownership and sharing of learning and teaching resources.
56
3. Develop and disseminate policies for access and rights, management of resources, and
intellectual property (IP) for the Carrick Exchange.
4. Use the forums of the Carrick Exchange to discuss rights management and ensure the
systems put in place are those that suit the sector best and reflect changing technologies
and contexts of use. (Following LORN project strategy 12.)
5. Implement the Creative Commons licensing system as a simple and fair method.
6. Have a clear statement of copyright as it affects reuse (distribution) within the licensing
agreement. Loughborough University research (Bates et al, 2006) suggests that this clarity
is essential otherwise users are unclear how materials can be legally and fairly modified
within the guidelines.
Strategies for the higher education sector
1. A possible framework for sharing is to adopt a model where ownership of resources created
by academics and support staff in the course of their employment is claimed by the home
institution; however, the institution grants its employees the right to share that material.
2. Have designated staff such as librarians to support and possibly manage the copyright and
intellectual property requirements for resources submitted to the Carrick Exchange (e.g.
Jorum staff help with metadata input).
Technical strategies
1. Provide templates for copyright and IP that can be used when submitting resources to
make the process uncomplicated.
2. Provide the means to clearly brand resources shared through the Carrick Exchange with
the name of the originating institution as well as the contributor/author. Many journals
automatically insert a title page (e.g. Routledge) but this will vary according to the type of
resource.
Exemplars
CLOE and Creative Commons: CLOE recommends the most recent Canadian version of the
Creative Commons Attribution-NonCommercial-ShareAlike (BY-NC-SA) licence, as the default
copyright licence for the CLOE repository.
http://cloe.on.ca/creative_commons.html
Loughborough University Repository: Manages licensing and rights for sharing by claiming
copyright on resources created by its employees, but giving them the right to share these.
http://rightsandrewards.lboro.ac.uk/index.php?section=1
MITOpencourseWare (OCW): Initiative uses Creative Commons licensing.
http://ocw.mit.edu/OcwWeb/web/home/home/index.htm
OAK Law Project: Aims to develop legal protocols for managing copyright issues within an open
access framework. Its focus is research repositories, but its policies have implications for the
Carrick Exchange project (as noted in the digital rights Discussion Paper, commissioned by
Education.Au by Mason, Macnamara and Galatis, 2007).
http://www.oaklaw.qut.edu.au/
RMIT and Loughborough University: Both universities are developing workflow models
regarding contribution and other aspects of their repositories which could be of interest to the
Carrick Exchange.
12
Interviewee communication.
57
RUBRIC project metadata: Support for repository managers,
http://www.rubric.edu.au/RUBRIC_Toolkit/docs/Metadata.htm
Second Life (SL) digital rights policy example alternative: A multi-user 3D online world created
by its residents. Members move around the virtual world using an avatar and interact with other
members much as people do in real life. Communities self establish around social interests,
education and business etc. Universities and other educational institutions have begun, e.g.
holding classes in SL and staffing virtual libraries with real world volunteers. Many blogs and
places for commentary exist not only inside the virtual world but outside as well. The owners of
SL act as service providers and enforce few controls over residents. It is generally selfregulating within the set terms of service. However, due to pressure from community members
and commentators, for example, gambling was banned. Residents retain the right to full
intellectual property protection for any digital content they create in Second Life, including
avatar characters, scripts, objects, designs etc. Residents however must grant the owners of
SL, Linden Lab a license to, amongst other conditions, use, reproduce and distribute their
content in any media for marketing and/or promotional purposes.
http://secondlife.com/
Key References
Barton, M. & Waters, M. (2004). Creating an institutional repository, LEarning About Digital
Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 27, 2007,
from http://www.dspace.org//implement/leadirs.pdf
Pappalardo, K., Fitzgerald, A., Fitzgerald, B., Kiel-Chisholm, S., O'Brien, D. & Auston, A.
(2007). A guide to developing open access through your digital repository. DEST funded Open
Access to Knowledge Law Project OAK Law Project. Retrieved September 27, 2007,
http://eprints.qut.edu.au/archive/00009671/01/9671.pdf
Mason, J. Macnamara, D. & Galatis, P. (2007). An investigation into digital rights management
issues in the Australian higher education sector: A discussion paper. Commissioned by
Education.Au. Internal report.
Principles related to management of resource contribution
2.5.1. The Carrick Exchange should develop clear policies, guidelines and
procedures for resource contribution.
A national approach regarding the submission policy for the Carrick Exchange should be
implemented. There should be a stratified system of classification, with peer reviewed
contributions at the top level of approved or endorsed content, and allowing for other categories
of submission that are not formally peer reviewed or may be under review. Clarity about the
status of resources will help selection of items and commentary by users. As peer review may
be a disincentive for some to contribute to the system, a tiered system of quality assurance and
review should be implemented (see Peer Review and Commentary section of this report).
The system should allow for the relative merits of a resource to be assessed by the user before
an item is downloaded. Clear labeling will assist in this.
Contribution policies should include guidelines and criteria:
58
•
•
•
Encouraging productive feedback to authors, and commentary around problematic issues
of teaching and learning;
Ensuring the quality of submitted resources meets the expectations of the sector. This
includes systems to ensure currency of resources through auditing and archiving
procedures.
Providing useful secondary metadata, such as details of the context of use, changes
required when adopted in other disciplines.
2.5.2. The Strategic Plan should include guidance on resource management,
archiving, sustainability, and support and encouragement for resource
contribution.
The Strategic Plan should define what will be included in the Carrick Exchange collection,
including the more formally contributed digital materials and the informally contributed
commentary. Some resources will need to be preserved well into the future; others will be more
ephemeral. Currency may diminish with some resources, and these should be removed.
However, there may still be a need to archive a selection or resources for perpetuity.
Information specialists should be consulted on how this should best be managed. The Carrick
Exchange should determine whether all commentary will be stored indefinitely or for a defined
period. Auditing, archiving and version control will need to be managed through quality
assurance mechanisms (see Peer Review and Commentary section of this report).
Contribution tools and templates should be simple, accessible and intuitive. So as not to
become a barrier to contribution, user defined metadata must be easy to assign and supported
by as much system assigned metadata as possible. Some focus groups suggested that
resources that are examples of ‘poor’ teaching practice or poor technology use are of value in
certain contexts, just as high quality resources are valuable; but each resource needs
appropriate tagging to help its discovery for multiple purposes and users. User defined
metadata and tagging is often poorly and inconsistently applied. Many Web 2.0 technologies,
however, are predicated on this folksonomy approach. This affects search and retrieval of
resources and should be factored into user engagement and contribution strategies for
particular content types.
Each discipline will have different views about what is appropriate, what taxonomy best suits
retrieval of resources, the benefits of contributing teaching materials versus the merits of
conversations about learning and teaching etc. The relative value of various content types, the
granularity of resources added, their ability to be customised, and ease of use will all be
assessed differently by different individuals and disciplines.
2.5.3. An evaluation plan should be developed to assess and report on resource
contribution to the Carrick Exchange.
Evaluations should be used as a basis for identifying funding needs for the maintenance and
further development of the Carrick Exchange and as justification for ongoing investment in the
Carrick Exchange. Evaluation of the evolving networks and communities, the value of
commentary and contribution should be investigated.
Impact evaluation for example, could assess the use of the Carrick Exchange as:
•
•
•
•
an appropriate workspace for team collaboration and facilitator of group processes;
an impetus for increased effectiveness in the use of information and communication
resources;
a promoter of communities of practice;
a space for facilitating the sharing of teaching and learning materials; and
59
•
an enabler of greater reflection on pedagogy within higher education.
Strategies for managing resource contribution
Strategies for the Carrick Institute and the Carrick Exchange
1. Provide long-term, stable funding and not rely solely on short-term grant funding. However,
additional funding options should be explored, such as advertising, sponsorship and
income generation through innovative project work and staff development initiatives. The
Carrick Exchange could also act as a broker in the future between individual academics
and commercial customers.
2. Through the Strategic Plan, outline the funding models for supporting resource
development for the Carrick Exchange within institutions, e.g. grants for Carrick Exchange
resource development, commissioned resources required by the sector. Also especially
provide assistance for resource contribution to institutions whose staff contribute resources
to the Carrick Exchange.
3. Within a strategic plan that outlines a phased roll out approach beyond the initial
implementation, devise activities across institutions that bring different stakeholder groups
including academics and support staff together, sharing resources and expertise. As the
novices, i.e. users new to the Carrick Exchange, develop expertise and skills, they in turn
become mentors to others within their own institutions, and in cross-institutional projects.
4. Promote the contribution benefits of the Carrick Exchange that are pertinent to the potential
contributor, e.g. The Carrick Exchange could be promoted to some academics as a space
where they can store and develop ‘works in progress’ used in collaborative projects, and
invite others (probably a small selected network of colleagues) to give feedback to assist in
further development. Tools such as wikis may aid in this type of collaborative contribution.
5. Model the contribution policy documents for the Carrick Exchange, on those found on the
Edna and MERLOT sites, with input from the Jorum guidelines. (Note: Jorum operates on
an institutional model of contribution, whereas CLOE and MERLOT offer open individual
membership.)
6. Develop an evaluation plan to track evolution of the system and its efficacy over time.
Technical strategies
1. Adopt a combined approach to metadata contribution, to meet the needs of different users
of the system. Metadata options should include standardised subject headings that users
can select from and assign to the shared resource, plus a keyword search function and a
browse option. Resource contributors could assign metadata about, e.g. what they think the
item is and what it could be used for; other members could add metadata about how useful
they found the resource or contribution; and peer reviewers (for some contributions) could
add tags around formal review categories.
2. Allow contributors to have the option to elect whether their resource should be peer
reviewed, as formal peer review may be seen as a barrier to contribution for some
academics.
3. Provide a dual track publishing system where all resources can be published immediately
but informally through the forums, at the same time as a resource may be undergoing
quality assurance and/or peer review. This allows for immediate publication (‘use as is’ but
‘user beware’), but also allowing time for quality and peer review processes. (see Peer
Review and Commentary section of this report).
4. Contribution tools and templates should be simple, accessible and intuitive.
5. Establish mechanisms and systems to enable version control and unique identifiers to
manage and promote reuse and repurposing of content and the ability to improve materials
that have been contributed to the Carrick Exchange.
6. Resource management at the personal level will be enhanced by tools on the Carrick
Exchange which mirror systems already in use on sites such as Amazon.com or
Librarything.com, where users can create and manage ‘baskets’ or personal lists that store
links to resources of interest tagged by the user, and to create lists of other members with
similar interests.
60
7. Consult information specialists about an appropriate preservation and archiving policy for
short-term and long-term purposes.
Exemplars
CLOE archiving policy: Contributors agree to share a resource for at least two years. The
resource is then reviewed for continuing inclusion in the collection after a five year period. If the
resourcee has not been accessed for two years it is archived.
http://cloe.on.ca/
VET Learning Object Repository Network project (LORN): Working on version control and
reuse.
http://lorn.flexiblelearning.net.au/Home.aspx
Contribution policies for digital resources – see the following models and the policy
recommendations at the end of this section:
Edna: http://www.edna.edu.au/edna/go/about/policies
MERLOT: http://taste.MERLOT.org/policies.html
Jorum: http://www.jorum.ac.uk/contributors/chelp/guidelines.html
The Jorum Workflow Report provides some clarity on processes, although it is based on
resource contribution that is initially institutionally based rather than individually based, and
there is no formal peer review process (only informal rating and commentary). Also Jorum
resources are published before cataloguing. The process is: Contribute, Publish, Catalogue,
Review. See Jorum Publications/Jorum R&D reports currently available:
http://www.jorum.ac.uk/publications/
Intute guidelines and the Desire Information Gateways Handbook (section 2): Relevant for
ideas about formulating policy.
Intute Collection Development Framework and Policy
http://www.intute.ac.uk/policy.html
Desire Information Gateways Handbook
http//www.desire.org/handbook/2-1.html
MITOpenCouseWare (OCW): Business model - open sharing of resources funded by
Massachusetts Institute of Technology and donations from around the world.
https://giving.mit.edu/givenow/ocw/MakeGift.dyn
Contribution policies for commentary resources – see the following models and the policy
recommendations at the end of this section:
MERLOT acceptable use policy:
http://taste.MERLOT.org/acceptableuserpolicy.html
Edna content standards:
http://www.edna.edu.au/edna/go/about/policies/pid/119
Yahoo data storage policy:
http://info.yahoo.com/privacy/us/yahoo/datastorage/details.html
61
Key References
Barton, M & Waters, M. (2004). Creating an institutional repository, LEarning About Digital
Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 29, 2007,
from http://www.dspace.org//implement/leadirs.pdf
Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities.
D-Lib Magazine, 13(5/6). Retrieved May 16, 2007, from
http://www.dlib.org/dlib/may07/henty/05henty.html
Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning.
London and Sterling: Kogan Page.
McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and
implementation. Journal of Digital Information, 5(3), Retrieved September 21, 2007, from
http://jodi.tamu.edu/?vol=5&iss=3
Recommendations for resource identification and contribution
policies
Policies will need to be developed in the following areas. Note: Barton and Waters (2004),
provide sound advice in their LEADIRS Workbook, Creating an Institutional Repository.
Resource identification policies
1. Collections policy
a. State how the Carrick Exchange differentiates itself from other collections, databases,
portals and communities.
b. Identify who manages and promotes the collections.
c. Specify the selection criteria and methods used to ensure quality, e.g. quality of
content, usability (ease of use, consistency, interoperability), educational effectiveness.
2. Classification of resources policy
a. Specify what resource types are collected by the Carrick Exchange e.g. Carrick grant
project resources; learning, curriculum and teacher resources; technological resources;
peer reviews and social resources; policy resources; and archived resources.
b. Identify the process used for different classifications: resource under development or
completed, resource for/not for quality assurance, for/not for formal peer review, for/not
for informal peer review.
c. Carrick award resources; highly rated resources; resources by individuals; resources
by institution.
d. Methods for version control.
e. Metadata management.
f. Licensing system. (Suggest beginning with Creative Commons.)
3. Registration policy
a. Guidelines or steps for registration and contribution (objects and commentary),
including templates to assist resource contribution and getting started.
62
4. Policies for archiving resources:
a. Policy on data storage;
b. Checks every 3- 5 years if authors wish their resource retained on the Carrick (issues
of currency and relevance);
c. Carrick Exchange selective archiving for the long-term – some resources should be
archived for historical purposes or for reference even when apparently past their
perceived currency.
d. Annual audit of resources based on downloads and access records.
5. Policies to manage resource identification and contribution through:
a.
b.
discipline-based initiatives or Editorial Boards; and
annual conferences or workshops where resources are generated or reviewed.
Resource contribution policies
1. Resource contribution guidelines and policies:
a. Determine who can contribute, what resources, when, for how long (2-3 years?), quality
assurance and peer review procedures.
b. Workflow charts to show how resources are ‘triaged’, according to classification:
Category 1: For full, formal peer review and quality assurance before submission to the
public repository, following a system of “triage”. Concurrently made available (marked
as “under peer review”) for additional commentary by members of the Carrick
Exchange (dual track publication).
Category 2: Not for peer review, quality assurance only – but concurrently made
available for additional commentary by members of Carrick Exchange. Marked as
‘under quality assurance testing’.
Category 3: Non-reviewed resources – items submitted “as is”, i.e. with no quality
assurance, informal commentary or formal peer review attached, contributed at owners’
own risk - subject to commentary by members of Carrick Exchange and marked as
such.
2. Policy regarding Carrick Institute grant holders: Mandatory contribution of project
development reports, project outcomes and deliverables to the Carrick Exchange as a
condition of receipt of a grant.
3. Metadata policy: This should ensure that metadata is easy to apply, complete and
maintained, and aids retrieval. Should address:
a. Classification of each resource and
b. Secondary metadata about use of the resource, context of use where possible.
4. Rights management/licensing policy and guidelines should specify:
a. For individuals – how their IP and moral rights are protected, and the implications for
sharing.
b. For institutions – the implications of sharing and contributing resources to the Carrick
Exchange by staff who created the resources in the course of their employment. The
policy and guidelines should address issues of competition and ownership.
c. Policy type
63
o
o
Creative Commons or standard copyright;
Indicate if any likelihood of e-commerce or brokering in the future.
d. Three specific Guidelines (derived from CLOE):
o
o
o
o
Statement of Copyright (including contact details of copyright holder)
Terms of Use (note that ‘fair use’ is insufficient for terminology)
Modification Rights (grant, or contact details to request permission)
Should cover deposit rights and distribution rights.
5. Management of shared workspaces, rights and responsibilities policies and guidelines.
6. Rewards and incentives:
a. Carrick Exchange Memorandum of Understanding with all Australian universities,
recognising the value of Carrick Exchange contributions and the role they play in the
core business of teaching and learning within higher education, and their importance
for promotion and career advancement. This should facilitate alignment of institutional
policy with Carrick Institute aims and vision to transform teaching and learning.
b. Within the Strategic Plan outline the methods for rewarding individuals and institutions
who contribute and use the Carrick Exchange such as:
o
o
o
Awards for voluntary work, resource contribution excellence, contribution to the
peer review process, contribution to the community etc. (see Strategies 3.3.1, 3.3.2
and 3.3.3);
Downloadable certificates and/or e-portfolio providing evidence of contributions and
reuse that can be used for promotion and tenure;
Acknowledgement for authors as the original creator of a resource.
64
Peer review and commentary guiding principles and
recommendations
______________________________________________
Introduction
As part of the development of the Carrick Exchange ascilite was engaged to conduct research
into:
•
What Peer Review and Commentary protocols and mechanisms can be derived from the
higher education community?
o
What new and current methods exist for peer review and commentary of resources
that can be adapted for use for the reviewing of teaching and learning resources?
−
−
−
−
o
o
What gets peer reviewed?
What methodologies apply to different types of resources?
What process for peer review is used?
Who are the reviewers?
What recommended policies and procedures are currently in use?
What are the policy implications for the Carrick Exchange?
The following protocols and mechanisms for peer review and commentary for resource
contribution to the Carrick Exchange have been synthesised from knowledge and expertise
existing in: the literature, current practice, exemplary community development to foster
engagement with online services, and key practitioners and networks in both national and
international settings.
Peer review for this research refers to the evaluation of teaching resources and commentary
contributed to the Carrick Exchange. Review is conducted by equals, those with qualifications
and standing in the higher education sector who are capable of assessing the worth and
value of ICT-based teaching resources (artifacts and discussions). Peer review may be formal
and conducted by teams of experts, it may be limited and more of a quality assurance
process, or it may be informal and include discussion, feedback and comments made on
resources and ideas contributed to the Carrick Exchange. Peer review as quality assurance
may include assessment of the currency, educational design and construction of resources;
compliance with copyright, intellectual property and digital rights management policies; and
technical accuracy and reliability. Formal peer review, a lengthier and more demanding
review process, might replicate the scholarly peer review process which leads to publication
in the higher education sector.
Peer review may also be an informal process whereby members of the community voluntarily
respond to others’ contributed resources, or resources stored elsewhere but linked to via the
Carrick Exchange (known as “commentary” in this research). These resources could be
finalised products which the authors publish and share, or resources under development. This
informal sharing of ideas could be an important element of the Carrick Exchange and the
basis for various communities of practice. Informal commentary can be seen as another
useful mechanism for peer review and academic recognition. It can be recognised as a
support mechanism for peers around best practice in learning and teaching and essential to
the development of communities of practice. It can also be a mechanism by which individuals
show leadership in the field.
Peer review can be conducted against specified criteria, and/or as a means of giving and
receiving structured feedback to improve teaching resources. Resources identified as formally
65
peer reviewed are likely to be perceived as high quality examples of best practice. The
implementation of rigorous peer review processes is key to maintaining high standards. As
one focus group participant stated:
Rigorous peer review is important for the outcomes and status of the review
process. Having peer review processes in place supports the perception of quality
resources and communication of best practice. (Focus group participant)
The findings of the ascilite research indicate that a formal peer review system for the Carrick
Exchange is required. It is recommended that the system chosen mirror the current system of
scholarly research review. Peer review of learning and teaching resources does not currently
have the same status as peer review of scholarly research. This is a key issue that should be
addressed to recompense staff for the effort and resources put towards this activity and raise
the status of scholarly review of educational resources.
For formal peer review, the traditional system of “blind” peer review where reviewers are not
identified versus a system where reviewers are known was discussed in the interviews and
focus groups. Some participants favoured the latter system because of the accountability
demanded; others favoured the traditionally anonymous system because it is intended to
assist with objective and democratic evaluation. Others suggested that where an author was
able to engage in a dialogue with the reviewers this would be beneficial for developing a full
understanding of the design dimensions and intended and actual learning outcomes. Some in
the reference groups pointed to the developing open peer review systems trialled by a
number of scholarly journals such as Nature (see Literature Review section of this report).
Open publication of a paper before it has been through the formal peer review process allows
for speedier dissemination of innovation. In the case of Nature magazine, once a research
article has passed an initial quality check, the author may post their paper on the journal’s
website, and anyone, provided they supply their name and email address, can comment on
the research; the traditional blind peer-review process continues in the background. This
option has positive implications for submission of resources to the Carrick Exchange where it
will often be important to expedite publication to maintain currency. However, although
publication is hastened, the process of soliciting informal peer review comments, while
received positively by journal readers, has had limited success and comments have not been
substantive. This may be different in smaller communities that gather around the Carrick
Exchange but in the more public areas it may follow the results of the Nature research (see
Literature Review section of this report).
Regarding informal peer review in this research, participants indicated that it should be an
open process where opinions were identified with an author, not anonymous contributions.
The credibility of the person contributing the comments was an important factor for members
considering the worth and value of others’ commentary. Further, it was though that “named
contributions” would promote scholarly, thoughtful review of resources and lessen the
likelihood of work being insensitively treated or “rubbished”. As one reference group member
said:
Web 2.0 has sparked a growing number of "free" and "open" movements that
challenge current publishing and peer review models — including the Free and
13
14
Open Source Software movements, the Open Access Movement , Open
15
16
Source Journalism , and Creative Commons ought to be considered in the
13
Software which the user can use for any purpose, study the source code of, adapt to their needs, and
redistribute - modified or unmodified.
14
Immediate and unrestricted online access to digital scholarly material, primarily peer-reviewed
research articles in journals.
15
commonly used to describe forms of innovative publishing of online journalism, rather than the
sourcing of news stories by a professional journalist
16
The Creative Commons (CC) is a non-profit organization devoted to expanding the range of creative
work available for others legally to build upon and share. The organization has released several
copyright licenses known as Creative Commons licenses. These licenses, depending on the one
chosen, restrict only certain rights (or none) of the work
66
new era of publishing as each contributes different perspectives and concerns on
the emerging practices surrounding digital repositories and scholarly publishing.
(Reference group member)
Peer review and commentary were seen to have the following goals:
•
•
•
•
•
Support the delivery of high quality, reviewed resources;
Enhance resources submitted to the Carrick Exchange which are “under development”;
Foster professional development around the development and review of teaching and
learning resources;
Encourage collaboration and the development of networks and communities of practice;
and
Reward and recognise resource creators and reviewers.
Recommendations in this section are structured around the following key themes that
emerged from the ascilite research findings:
1.
2.
3.
4.
Peer Review Goals
Peer Review Processes and Procedures
Informal Commentary
Rewards and Incentives for Formal and Informal Peer Review
Protocols and mechanisms related to the goals of peer review
3.1.1. Formal peer review policies and procedures of the Carrick Exchange
should align with institutional policies and practices, and promote the
scholarship of teaching and learning.
The products of formal peer review (reports, ratings, documentation providing
acknowledgement of peer review etc.) must have currency and credibility. To ensure this
there should be alignment nationally with institutional recognition and reward processes and
systems, providing incentives to both reviewers and contributors. The peer review scheme
adopted should promote the scholarship and advancement of teaching for those who have
resources reviewed and those who undertake those reviews.
Recommended policies and procedures for peer review are provided below (see Strategies
for Carrick Institute and Carrick Exchange).
3.1.2. The goals of formal peer review should be established and inform
related policies and procedures.
The purpose and expected outcomes of the formal peer review process should be established
and outlined in the strategic plan. It is suggested that these goals might be:
•
•
•
to assure the quality of resources;
formalise the process of peer review;
provide a review process that is recognised as transparent, fair, valid and reliable; and
67
•
foster recognition of contributors whose resources are peer reviewed, and reviewers who
conduct the evaluations.
The identified goals should then inform the development of related policies and procedures.
3.1.3. Peer review should enable judgements about the quality and usefulness
of a resource.
As a national repository the Carrick Exchange is in a position to implement a formal review
process – a facility that national repositories and their communities may find too resource
intensive to support. One interviewee suggested that the success of the Carrick Exchange
was dependent on the standard and effectiveness of the peer review process.
Peer review of educational resources is likely to be a more complex undertaking than review
of, scholarly research output which is usually presented as a single medium, text document.
The difficulty of assigning evaluation criteria, the context of use, and the variables that impact
on expected learning outcomes, add to this complexity.
The issue of assessing the quality of resources in Carrick is different to assessing
the quality of journal articles because of the practical bent of the things that are in
Carrick. (Interviewee)
In preparing policies and procedures for the formal peer review process criteria should be
established to address evaluation standards. MERLOT provides a useful set (see:
http://taste.merlot.org/evaluationcriteria.html). CLOE also uses a similar standard.
MERLOT uses three categories for its evaluation standards:
1. Quality of content
a) Does the software present valid (correct) concepts, models and skills?
b) Does the software present educationally significant concepts, models, and skills for
the discipline?
2. Potential effectiveness as a teaching-learning tool (the most difficult to evaluate,
especially where modules are taken out of context – assessing the purpose of the
resource is very important – the review should be contextualised).
3. Ease of Use – particularly for first time users (teachers or students).
Other criteria and standards against which the teaching resources can be evaluated can be
found on the AUTC Learning Design Project website, the CLOE and MERLOT websites, and
in documents by HEAL, the Health Education Assets Library (2006), the University of
Queensland et al (2004), and Souza and Persily (2005).
Other questions around the peer review process put forward by participants in the research
include:
•
•
Should feedback from learners be included in the review of resources where learning is
the focus?
Should the review process be fully online and integrated with the Carrick Exchange?
Finding a balance between quality review (particularly in informal contexts) and democratic
commentary open to all members of the Carrick Exchange will be a challenge. As one
reference group member suggested:
68
There is a need to develop models and processes for peer review alongside
issues of copyright, development of acceptable use policy and innovative use of
technologies that model open source, participatory and interactive knowledge
creation. (Reference group participant)
3.1.4. Resources contributed to the Carrick Exchange should be classified
according to the level of peer review received.
Formal peer review is not suited to all resources, therefore the Carrick Exchange should allow
for a number of levels of peer review. To speed up sharing and publication procedures,
resources placed in a peer review workflow should be made available to the community prior
to and during the peer review process, and labeled as “under review”. This allows for a dual
track system of publication and submission of resources (see Figure 3). Resources could be
concurrently submitted for informal review to either all members of the Carrick Exchange or a
restricted group of selected members, and at the same time submitted as a resource to
receive formal peer review. This would be a choice each contributor could make at the time of
submission of any resource, along with a nomination about the level of peer review that the
resource should receive.
To ensure that users of the system know the status of any resource retrieved through a
search of the Carrick Exchange, the following classifications could be adopted:
Category 1: For full, formal peer review and quality assurance before submission to
the public repository, following a system of “triage”. Concurrently made available
(marked as ‘under peer review’) for additional commentary by members of the Carrick
Exchange (dual track publication).
Category 2: Limited review resources, quality assurance only – but concurrently
made available for additional commentary by members of Carrick Exchange. Marked
as ‘under quality assurance testing’ (dual track publication).
Category 3: Non-reviewed resources – items submitted “as is”, i.e. with no quality
assurance, informal commentary or formal peer review attached, contributed at
owners’ own risk - subject to commentary by members of Carrick Exchange and
marked as such.
Other sub-classifications could include:
•
•
•
•
“Expert” reviewed resources;
User (or general member) reviewed resources (receiving informal commentary as
review);
Carrick funded projects – which receive a certain amount of review through the project
development process; and
Status of completion – “completed” or “under development”.
The limited review or quality assurance of resources could include an assessment of:
•
•
•
technical accuracy, reliability and interoperability;
compliance with copyright, intellectual property and digital rights management policies;
and
the relevance, currency, and construction of resources.
Formal review processes should follow the MERLOT criteria (see 3.1.3 above, and Figure 3).
69
3.1.5. The formal peer review process should be appropriately resourced and
adequately rewarded
The peer review process could be a resource intensive process, so only those resources
which are of sufficient quality to warrant this assessment should be submitted for full peer
review. The issue of adequate resourcing of the peer review process was a concern raised by
research participants in both Cycle 1 interviews and Cycle 2 focus groups. Practitioners in
both cohorts suggested that the resourcing of the peer review process could ultimately have a
significant impact on its success. This sentiment was echoed by the project team in the
collaborative Peer Review of Learning Materials Report (University of Queensland et al,
2004). The report also suggested that peer review processes be integrated into other existing
institutional processes.
Feedback from the sector and the experience of institutional repositories suggests that the
Carrick Exchange may need to employ individuals to manage the peer review processes. The
Carrick Exchange should not rely solely on volunteers. Further, it was proposed that not only
teaching academics be involved in the processes, but that educational designers and IT
personnel may need to be employed to assist with quality assurance and provide feedback on
the design and potential effectiveness of the resource.
The Carrick Exchange is advised to engage a team of experts with a mix of expertise (e.g.
disciplinary, educational and technical specialists) to undertake the formal peer review
process. Once a resource has been submitted to the Carrick Exchange staff employed by the
Carrick Exchange could complete the initial quality assurance checks, then pass the resource
to a review panel. This panel would be convened and chaired by a person nominated by the
Carrick Institute. Participants in one of the focus groups suggested this position could be
occupied by a Carrick Institute Fellow.
As well as managing the peer review process, the Carrick Institute should make provision for
a linked system of reward and recognition (see also Resource Identification and Contribution
section of this report and principles related to rewards, incentives and recognition for
contribution) for authors and reviewers. Focus group participants strongly urged that
membership of a peer reference group or formal review panel be viewed as prestigious and
that peer review responsibilities valued as part of an academic’s professional development.
The British Academy, (Shepherd, 2007) urge those responsible for the management of
universities and research institutes to encourage and reward peer review activity suggesting
that such a practice might stop high calibre academics, already overburdened with work, from
declining peer review duties, and may even attract reviewers.
Strategies to ensure the goals of the peer review process
Strategies for Carrick Institute and Carrick Exchange
1. Develop policies and procedures for peer review that:
•
•
•
•
•
•
•
•
stipulate parity of formal peer review process with scholarly peer review of research;
ensure alignment with institutional promotion policies;
identify formal peer review roles and responsibilities and differentiate voluntary and paid
roles;
provide quality assurance;
determine the processes and procedures, criteria and standards of formal peer review;
foster review from multiple perspectives (e.g. pedagogical, technical and disciplinary);
identify strategies for recognising and rewarding authors and peer reviewers; and
include grievance and appeals protocols, guidelines and procedures.
70
2. Lobby university senior executives to acknowledge the formal peer review of resources
submitted to the Carrick Exchange in institutional recognition and reward processes and
schemes.
3. Establish the goals of the formal peer review process.
4. Prescribe and resource peer review processes and procedures for the levels of peer
review (see Figures 3 and 4). Recognise and address in the development of these peer
review processes and procedures:
•
•
•
•
issues relating to effort and resources, time and staffing;
that peer review is perceived as both an incentive and disincentive for resource
contribution, so members should be able to opt for or against any kind of peer review or
quality assurance;
workflow alternatives; and
supporting materials (e.g. guidelines, protocols).
5. Provide templates, for both the reviewer and the contributor, outlining the criteria for
assessment of resources for both quality assurance and formal peer review.
6. Devise a strategy for rewarding and recognising peer reviewers (e.g. provide reward “in
kind” through reimbursement for attendance costs for reviewers at an annual Carrick
Exchange Conference).
7. Identify a pool of reviewers and potential review panel chairs (e.g. editors of journal and
conference proceedings, Carrick Institute Fellows and citation recipients, invite
contributors to identify potential reviewers, target specific communities of practice and
members of these communities to be reviewers).
Strategies for the higher education sector
1. Promote the view that resource sharing is not dissimilar to publication of articles and to
this end peer review (including quality assurance) of learning and teaching resources
provides a means for achieving this change.
Technical strategies
2. Ensure the level of review that contributed resources have undergone is clearly
communicated and easily identifiable, this could possibly be an automated feature of the
system.
3. Automate as many quality assurance processes as possible throughout the Carrick
Exchange: e.g. alerts to notify of broken links, automated metadata tagging etc.
Exemplars
Appendix G: An overview of formal peer review processes adopted by various repositories.
ACELL (Chemistry database): An example of a discipline based repository implementing peer
review. Resources on the site are not published until they have been through a thorough
evaluation by staff and students. See the workflow model as well.
http://acell.chem.usyd.edu.au/homepage.cfm
AUTC Learning Designs website: Products of the AUTC project on ICT-based learning
designs.
http://www.learningdesigns.uow.edu.au/index.html
CLOE: Proforma for peer review evaluation of learning objects.
http://cloe.on.ca/peerreview.html
71
MERLOT peer reviewers: MERLOT utilises individuals engaged in other institutional
initiatives as peer reviewers.
http://www.merlot.org/merlot/index.htm
The methods adopted by MERLOT to choose or find peer reviewers are:
a) Staff volunteer, receive training in the peer review system and then move through a series
of steps before becoming a full reviewer, an associate member of the editorial board or an
editor. With experience, as qualifications and the quality of work is established, individuals
progress up the ladder.
b) Institutions nominate peer reviewers who go through a training process and prove
themselves to their colleagues. As these individuals are noticed and the quality of their
work acknowledged they are asked to take on more responsibility.
Key References
Health Education Assets Library (HEAL) (2006). Editorial Policy. Retrieved September 22,
2007, from http://www.healcentral.org/services/policies/HEAL_Editorial_Policy_v2.0.pdf
Shepherd J. (2007). Help Wanted. The Guardian, 4 September. Retrieved September 22,
207, from http://education.guardian.co.uk/higher/research/story/0,,2161680,00.html
Souza, K.H. & Persily, G. (2005). National Peer Review of Digital Learning Materials in the
Health Sciences. Retrieved September 22, 2007 from
http://cit.ucsf.edu/events/event_mar05.php
University of Queensland, Griffith University & Queensland University of Technology (2004).
Peer review of learning materials report: A collaborative project involving the University of
Queensland, Griffith University and Queensland University of Technology. 30 August 2004.
Protocols and mechanisms related to the process and
procedures for peer review
3.2.1. The Carrick Exchange should establish processes, criteria and
standards that enable consistent, equitable and fair peer review.
Peer review should be viewed as “institutionally neutral” to promote its value within the higher
education community.
As noted in the Literature Review, the MERLOT system of formal peer review is considered
one of the best models. This system is based on the academic peer review practices for
scholarship and publication in higher education, an “expertise-orientated” approach (Worthen
et al., 1997). Building on the MERLOT model, Nesbit et al. (2002) created a convergent
participation model for evaluation of learning objects where resources undergo a two cycle
process: two individual experts assess the resource then a combined group assessment is
made, amalgamating the feedback from both assessments. A model for formal peer review
for the Carrick Exchange, based on the MERLOT and CLOE systems, and Nesbit et al
(2002), has been provided in Figure 3 and a model for limited review has been provided in
Figure 4.
72
Bruce (1997) and Smith (2005) identify a number of important features for a formal peer
review process. It is recommended that these be reflected in the design of the Carrick
Exchange formal peer review process:
1. The resource contributor should agree to, or nominate the resource for formal peer
review.
2. Peer review should occur according to an agreed timeframe, with specific goals and
outcomes and clear identification of exactly what is to be reviewed. Peer review protocols,
mechanisms and processes should address: What is being reviewed; who the reviewers
are; and the criteria for assessment.
3. The existence of a phase in the peer review process dedicated to the review of the
process itself may be warranted. It may also be worthwhile for the resource contributor to
have tools to conduct their own self-evaluation before the formal peer review process.
4. Individual outcomes of the review should be known only to the peer reviewers and should
remain confidential. However, a summarised version of the review could be made public,
following the MERLOT model.
5. The peer review process should lead to decisions about action to be taken to ensure
improvements.
What to review: resources that could be reviewed by peers, include, but are not limited to:
•
•
•
•
•
•
•
•
•
•
•
•
•
Course outlines,
Course materials,
Current curriculum,
Curriculum development documents,
Course syllabi,
Statements of teaching philosophy,
Assessment programs,
Marking schemes,
Learning objects,
Simulations,
Case studies,
Role plays, and
Learning designs.
Who should review: As discussed earlier a group of experts should undertake the review –
discipline, educational and technical/media experts, plus an Editorial Chair. (see
recommendation 3.1.5.) Peer review should not be conducted by those in a line-management
relationship. An effective means of establishing how useful a resource is may be gained from
the feedback of practitioners who have used the teaching resource. The collaborative Peer
Review of Learning Materials Report (University of Queensland et al, 2004), suggest that staff
for whom a significant part of their core business involves developing learning resources may
be appropriate peer reviewers.
Criteria for review: See recommendation 3.1.3 above.
Proformas and protocols: A proforma, developed and informed by the research, is
suggested in Appendix H: Carrick Exchange Peer Review Criteria and Standards. Suggested
protocols for peer review are provided in Appendix I.
Models for the peer review process: See Figures 3 and 4.
Roles: The HEAL (2006) Editorial Policy provides some useful detail on roles in the peer
review process (see Exemplars below) as does Barton and Warton (2004), in the LEADIRS
Workbook. See also Appendix F – Roles identified for the Carrick Exchange.
Administrator or Quality Assurance Editor: For both models it is suggested that the
quality assurance technical check be conducted by a special Carrick Exchange
73
administrator or editor. Checks on the resource could include accessibility,
functionality – checking that hyperlinks work, plug-ins are available, platform and
browser compatibility are identified, the resource meets criteria for inclusion in the
Carrick Exchange collection, adheres to copyright restrictions and that adequate
metadata has been provided. While housed in the non-peer reviewed collection
contributors could stipulate whether the resource is simultaneously available for
informal commentary, and who may contribute the informal commentary (e.g.
everyone, a particular community of practice, trusted colleagues, invited critical
friends).
Reviewers: Reviewers conduct and record their reviews independently, are then
convened by the Panel Chair (probably online – synchronously or a synchronously) to
discuss their ratings (numerical and written comments). Convening of the panel to
discuss the assessments helps to mitigate against bias by the Chair and gives a more
holistic evaluation, taking into account technical, educational and discipline specific
issues.
Panel Chair: May be appointed by the Carrick Institute. Oversees the process and
facilitates the discussion amongst the reviewers, and the final decision which is
communicated to the author as a numerical rating (perhaps for quality of content,
ease of use and potential educational effectiveness) and comments.
Author: May be contacted during the review process by the Panel Chair for further
clarification.
74
Figure 3: Formal peer review process model for
resource contributions to the Carrick Exchange
(Based on MERLOT and CLOE models, and Nesbit, Belfer & Vargo, 2002)
CE = Carrick Exchange
75
Figure 4: Quality assurance only (limited peer review) model
for resource contributions to the Carrick Exchange
Key: CE = Carrick Exchange
3.2.2. The peer review process should meet the expectations of the sector and
the needs of members.
Quality, accuracy, and currency of resources were elements identified by focus group
participants as very important in relation to peer review. The Carrick Exchange should be:
A high quality space, housing high quality resources, with peer review the key to this
perception. (Focus group participant)
Other expectations of the sector identified from the research include:
•
•
timely publication – the time between resource contribution and publication should be
as short as possible); Allowing resources to be published informally and be open to
comment at the same time as the resource undergoes peer review is a partial solution
to this problem (this follows the procedure trialled by the scientific journal Nature); and
quality metadata about the peer review outcome so that members can assess the utility
of resources with ease, particularly in large collections.
Resources that have been through the full peer review process will be associated with
scholarly information, (e.g. about the context of use for which the resource was designed,
theoretical underpinnings) and assessments by a number of experts. This additional
information will provide members of the Carrick Exchange with added value over and above
items not subjected to this process. Therefore it is important that the Carrick Exchange
regularly checks with members whether the process is meeting their needs. This could be
76
conducted, for example, through online forums on the Carrick Exchange and during
conferences.
Strategies related to the process and procedures for peer
review
Strategies for the Carrick Institute and the Carrick Exchange
1. Establish a formal peer review process that meets the expectations of the sector (Figure 3
is provided as a possible process).
2. Identify the roles and responsibilities required to implement the designed formal peer
review process. (see recommendation 3.2.1 above and Appendix F.)
3. Develop a strategic plan to resource roles and responsibilities, e.g. generate a databank
of invited or potential personnel, including panel chairs (such as Carrick Institute Fellows
and citation winners), and peer reviewers (with a mix of expertise), quality assurance
administrative or editorial staff, reviewers (some of whom may suggested by contributors).
4. Ensure those undertaking the formal peer reviews are adequately qualified and trained;
consider mandatory training or professional development for peer reviewers.
5. Develop standards and criteria for assessing educational resources, submitted to the
Carrick Exchange for formal peer review.
6. Check with members that the peer review process is meeting needs. This could be
facilitated via online forums and during conferences.
7. Ensure all resources are clearly identified as to their status regarding peer review, quality
assurance etc.
8. Enable contributors to identify their preference for the resources they submit, e.g.:
•
•
•
•
For/not for formal peer review;
For/not for limited review, i.e. quality assurance;
For/not for commentary and feedback; and
For public release/for limited (small group) release.
Strategies for the higher education sector
1. Position membership of a Carrick Exchange peer reference panel as a role with prestige,
valued as part of an academic’s or professional staff member’s professional development.
Technical strategies
1. Automate the initial quality assurance process as much as possible (e.g. alerts to notify of
broken links, automated metadata tagging etc.).
Exemplars
AUTC The Learning Designs project:
http://www.learningdesigns.uow.edu.au/index.html
The project used a formal peer review process comprising three main phases:
1. Prior to resource submission, contributors completed a “learning design submission form”
providing details of their learning design and themselves.
2. Two evaluators individually completed an assessment of the learning design (instructions
and a proforma for recording this evaluation were provided to evaluators).
77
3. The two evaluators confer with each other on their individual assessments of the learning
design and reach a consensus about the evaluation generating and submitting one
evaluation report.
CLOE, Cooperative Learning Object Exchange: Peer review processes.
http://cloe.on.ca/
HEAL editorial policy v2 (2006): Details about the roles in the peer review process.
http://www.healcentral.org/services/servicesPeerReview.jsp
MERLOT peer review model: This model is widely considered to be a good example of a peer
review process .
http://taste.merlot.org/aboutpeerreivews.html
Nature magazine “Precedings”: Pre-print publication of scientific research open for general
peer review by anyone who supplied their name and email address. This allows for dual track
peer review.
http://blogs.nature.com/peer-to-peer/2007/06/nature_precedings.html
Key References
Bruce, C. (1997). Peer Review: A Handbook, Queensland University of Technology.
Retrieved September 23, 2007, from http://sky.fit.qut.edu.au/~bruce/peerreviewhanbook/.
Health Education Assets Library (HEAL) (2006). Peer Review of Digital Scholarship.
Retrieved September 22, 2007, from
http://www.healcentral.org/services/servicesPeerReview.jsp.
McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and
implementation. Journal of Digital Information, 5(3), http://jodi.tamu.edu/?vol=5&iss=3
Nesbit, J. Belfour, K. & Vargo, J. (2002). A convergent participant model for evaluation of
learning objects. Canadian Journal of Learning and Technology, 28(3).
http://www.cjlt.ca/content/vol28.3/nesbit_etal.html
Smith, C. (2005). Guidebook for individuals embarking on peer review of teaching, The
Teaching and Educational Development Institute, University of Queensland. Retrieved
September 23, 2007, from
http://www.tedi.uq.edu.au/downloads/evaluations/Guidebook_PeerReview.pdf.
Taylor, P. & Richardson, A. (2001). Validating scholarship in university teaching. Evaluations
and Investigations Programme. Canberra, ACT: DETYA, Commonwealth of Australia.
The University of Queensland, Griffith University & Queensland University of Technology
(2004). Peer review of learning materials report: A collaborative project involving the
University of Queensland, Griffith University and Queensland University of Technology. 30
August 2004.
Worthen, B., Sanders, J. & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches
and practical guidelines (2nd Ed.). New York: Longman.
78
Protocols and mechanisms related to informal commentary
3.3.1. Guidelines and protocols for informal commentary should be
established to ensure that the needs and expectations of contributors
and users are met.
Peer review may be an informal process whereby members of the community voluntarily
respond to others’ contributed resources, or resources stored elsewhere but linked to via the
Carrick Exchange. Informal commentary might include discussion, feedback or comments
made on resources and ideas contributed to the Carrick Exchange. Commentary is seen to
be beneficial because it can provide informed users’ ratings, perspectives and assessment.
Research participants indicated that it would be preferable if all informal comments were
attributable to identified members. It was believed that this would help foster trust and
credibility within the community. Guidelines for informal peer review or commentary will
enhance the likelihood of submitted feedback being a valuable form of information especially
with regard to further and subsequent resource development.
Guidelines and procedures should ensure that commentary and feedback is managed via
established protocols and processes, reducing the incidence of derogatory or negative
comments. Guidelines should provide a model for commentary and develop members’ skills
in providing constructive feedback. In addition, members’ feedback should be aligned with
Carrick Exchange processes and policies to provide a desired standard of commentary.
Members of the Carrick Exchange will have diverse needs and varying disciplinary
backgrounds. Individuals may use submitted resources in ways the creator might not
necessarily have imagined, and it will be valuable to record these diverse contextual uses.
The issue is how to encourage those who have tried out a resource downloaded from the
Carrick Exchange to return and report on their reuse and/or modification, and record their
peer review. Communities such as the LAMS Community have found that only a very low
proportion of items in the repository get rated.
Research participants saw spaces within the Carrick Exchange where contributors can house
resources and invite trusted colleagues to review and make comments about their resource
as worthwhile. As one focus group participant explained:
You are more likely to make comments about a resource, and your use of a
resource, where you feel secure in knowing who the audience is, as opposed to
making a comment in a forum of strangers, who may pounce and ridicule your
comments. (Focus group participant)
As with formal peer review, there should be mechanisms to enable the contributor to indicate
if they want informal commentary or feedback in any form, when submitting a resource to the
Carrick Exchange.
A ratings system was seen to be both desirable and problematic. While setting criteria for
numerical ratings (e.g. one to five stars) was considered an asset, it was often seen as
difficult for users to know how consistently any criteria might have been be applied in an
informal ratings context. A resource may be useful to one member but not to another,
depending on e.g. context, target audience and experience. Questions about a ratings system
raised by research participants that the Carrick Exchange should address include the
following:
•
•
•
•
Would the system be an anonymous rating system, and therefore lack credibility?
How would you compare one rating of 5 stars with another rating of 5 stars?
What is being measured?
What are the rating criteria?
79
•
Who will return to the website and rate others’ contributions once they have
downloaded a resource? What incentive is there to go back?
What if a resource gets a poor rating? Is this detrimental for the contributor as well as
the Carrick Exchange? Or does the idea of “accountability when public”, assume a
certain standard and avoid this issue?
•
Solutions should also be found for how the experience of the person providing the
commentary or review might be judged. They may need to complete a profile of their
experience and rate the relevance of their expertise in reviewing a resource.
Strategies for informal commentary
Strategies for Carrick Institute and the Carrick Exchange
1. Develop guidelines and procedure/s for submission of informal commentary. These
guidelines and procedures should ensure that:
•
•
•
•
•
commentary and feedback is moderated (e.g. by Carrick Exchange personnel) so that
negative comments are managed appropriately; (users could check a box to say “This
comment is inappropriate”);
Carrick Exchange appoint staff and volunteers to model constructive feedback
approaches;
informal comments should be attributable to identified members;
the implemented (star) rating system has guiding criteria to assist those providing
feedback; it may be advantageous to allow this rating system to evolve over a period
of time (e.g. 6 months) as a consequence of the submissions and suggestions of
users; and
contributors are able to nominate, when submitting a resource, if they do not want
feedback in any form.
2. Develop an appeals process to manage members’ identification of an inappropriate
resource or unacceptable comments. This could be an icon that can be clicked beside
every comment or resource (e.g. following the YouTube model).
Strategies for the higher education sector
1. Provide a registry of staff willing to provide feedback and act as community reference
group members, e.g. name and times when available.
Technical strategies
1. Provide mechanisms to enable users to rate and provide discussion, feedback and
comments about resources submitted to the Carrick Exchange.
2. Ensure that a requirement of submitting informal commentary is identification of the
person submitting the feedback (i.e. not anonymous).
3. Provide functionality to allow informal commentary on developing work, with mechanisms
to enable contributors to restrict or prohibit access to work under development.
Exemplars
Amazon.com: An eCommerce company which sells goods over the internet advertises details
of available products listed in various merchandise categories. Each product has a star rating
80
and a link to existing customer reviews and the facility for users to submit their own review.
The publicised star rating is an average of all the submitted ratings from reviewers. Submitted
comments can be rated and commentary on the comments can also be submitted generating
in some instances an asynchronous discussion in regard to the initial submitted comment.
Reviewers are ranked according to the number of comments they have contributed across the
site. For example at the time of writing this report the top reviewer had submitted 14,794
reviews.
http://www.amazon.com
The Carnegie Foundation for the Advancement of Teaching, Teaching and Learning
Commons: An community space provided to enrich and encourage exchange of knowledge
about teaching and learning. This repository invites users to read, understand, and comment
on others’ work. The exchange provides resources, suggestions and a means to comment on
work in the Commons (e.g. suggestion of ways to annotate).
http://commons.carnegiefoundation.org/
EducaNext: A service supporting the creation and sharing of knowledge for higher education.
It is open to any member of the academic or research community. Informal peer review is
encouraged from those using the learning resources (faculty members and students) inviting
them to describe their perceptions and experiences of a learning resource after they have
used it. Descriptions are based on a tripartite inquiry on the technical, pedagogical, and
organisational aspects of the resource.
http://www.educanext.org/ubp
Jorum: A JISC 17 -funded collaborative venture in UK higher and further education sector. It is
a national repository developed to collect and share learning and teaching materials, allowing
their reuse and repurposing. Jorum has a star rating and comments service where users are
encouraged to add ratings and evaluative comments to resources. In addition, Jorum will be
establishing a web-based forum facility for comments and discussions held at the Jorum
website. Contributors and members of the Jorum team are notified when comments are
added. http://www.jorum.ac.uk
LAMS Community: Members can contribute and download learning designs and contribute to
discussion forums. Learning designs can be rated with a five star rating and qualitative
comments can be added.
http://lamscommunity.org
MERLOT: Invites users to rate linked resources from one to five stars and to indicate if the
resource has been used in a classroom. A comment requires remarks as well as the star
rating. Materials that have member comments posted can be found through an advanced
search. MERLOT also offers “From the Author Snapshots” which are short explanations from
authors of learning materials about the development and use of their learning materials.
http://www.merlot.org
MySpace: A social networking website offering an interactive, user-submitted network of
friends, personal profiles, blogs, groups, photos, music and videos. MySpace enables user
identified comments to be added to submissions such as videos and music. A percentage
rating based on the positive (“booyah”) or negative (“no way”) votes a submission receives is
also provided. The profiles of users providing the informal commentary can be accessed by
clicking on their user name listed next to the comment. The number of times a resource has
been accessed and the number of comments that have been submitted in relation to a
resource are also listed with the resource details.
http://www.myspace.com/
Slashdot: A science, science fiction, and technology-related news website that features usersubmitted and editor-evaluated current affairs news stories with a "nerdy" slant. Stories on the
17
The Joint Information Systems Committee (JISC) supports education and research by promoting
innovation in new technologies and by the central support of ICT services (http://www.jisc.ac.uk/).
81
site have linked discussions with comments generally submitted by site visitors which
Slashdot's editors accept or reject for general posting. Threading and moderation of submitted
comments are features of the slashdot discussions and contributors will also categorise the
nature of their comments/discussion (e.g. insightful, funny, informative, interesting, offtopic,
flamebait, Troll)
http://slashdot.org/
Warwick University Experts directory: A directory of “experts” - individuals who have offered to
help others in their department, faculty or the whole university by providing advice in their
area of expertise. Warwick University advertise that an expert may be able to:
•
•
give technical support.
help with choosing tools and techniques, and understanding how to best use them.
http://www2.warwick.ac.uk/services/elearning/experts/
YouTube: A video sharing website where users can upload, view and share video clips.
Unregistered users can watch most videos on the site, while registered users are permitted to
upload an unlimited number of videos. Details on the number of times a video has been
viewed, rated as a user’s favourite video, rated by users or had a comment posted in relation
to it are provided. Comments that have been provided can also be rated using a “thumbs up”
or “thumbs down”, denoting good or poor comments, and there is also the option to reply to a
posted comment. Responses to a video can be in the form of text comments or a video
response. Each posted video has a star rating that is an average of the collective submission
of ratings. Details on when the video was submitted, meta tags, and the contributor are also
provided.
http://www.youtube.com
Key References
Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education.
JISC Technology and Standards Watch. Retrieved May 1, 207 from
www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf
Protocols and mechanisms related to rewards and incentives
for formal peer review and informal commentary
3.4.1. The Carrick Exchange must implement rewards and incentives for
participation in either formal peer review or informal commentary
Rewards and incentives for participation in either formal peer review or informal commentary
underpin the ultimate success of the entire review process. For contributors and users to
engage in the review process there needs to be a perception of added value. This was firmly
supported by research participants with comments such as:
Given the level of high quality review that is being sought it may be necessary
to pay the reviewer. (Focus group participant)
Some sort of recognition needs to be given to peer reviewers. (Focus group
participant)
82
Recognition of effort in providing reviews. (Focus group participant)
What is the incentive for those doing the reviewing? (Interviewee)
Why should I review something? I’m already reviewing journal papers, grant
applications, conference papers – this is one more. (Focus group participant)
Research participants have identified the following as rewards and incentives for participation
in either formal or informal peer review:
•
•
•
•
Members are more likely to contribute commentary and engage in discussions if they trust
the community. This community may also be an incentive, as one focus group participant
explained: “There is an aspect of community that acts as an incentive for engaging in
commentary”.
Positive, productive feedback is more likely to encourage contributors to submit resources
for peer review and commentary.
Acknowledgment and recognition by the Carrick Exchange and institution for involvement
in formal peer review (either as a contributor of a resource for formal peer review or as a
peer reviewer).
Links to institutional promotion, as focus group participant explained:
As far as recognition of people’s contribution to teaching and learning goes,
promotion is the epicenter, and probably the next closest is actually applying for a job
… If Carrick can produce changes in that area . . . so that certain forms of feedback . .
.can be included in a teaching portfolio or equivalent, that would be a huge incentive
to contribute. (Interviewee)
Strategies for rewards and incentives for formal peer review
and informal commentary
Strategies for Carrick Institute and the Carrick Exchange
1. Establish protocols with Australian universities for encouraging the use of formally peer
reviewed resources as evidence for promotion and career advancement.
2. Establish criteria and processes for rewards and incentives for contributing resources that
are peer reviewed, and for being a peer reviewer (e.g. the ‘No 1 reviewer’ - see the
amazon.com example; reimbursement for attendance costs for reviewers at an annual
Carrick Exchange Conference for peer review).
3. Recognise the effort in providing commentary within an annual report.
4. Ensure that the products of peer review (reports etc.) have currency and credibility in
relation to established recognition and reward processes and contexts (especially
promotion).
Strategies for the higher education sector
1. Recognise formally peer reviewed resources as evidence for promotion and career
advancement.
Technical strategies
1. System facility that provides a “Monthly spotlight on No 1 reviewer”.
2. Automated system provided tally of contributions per person.
83
Exemplars
Amazon.com rewards: Informal reviewers rewarded by giving them a rank that is assigned
according to the number of informal reviews and comments they have submitted.
http://www.amazon.com
Key References
Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project,
Academic survey: Final report. Loughborough University & JISC. Retrieved July 16, 2007,
from
http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/1137423
150_SurveyReport.pdf
Recommendations for peer review and commentary policies
Policies will need to be developed in the following areas:
1.
Guidelines and policies to inform national policy on parity of formal peer review of
educational resources compared with scholarly peer review of research.
2.
Policy for reporting on peer review processes (communication with institutions) to link
with institutional promotion policies.
3.
Quality assurance practices and procedures:
a. Triage resources identified for this process.
b. Timeline indicating time between resource contribution and publication, depending
on classification.
c. Dual track processing if resource submitted for quality assurance and if submitted
concurrently as a resource for informal peer review.
4.
Policy and procedures for attribution and identification of reviewers:
a. Identify source of all informal commentary.
b. Identify reviewers in the formal peer review system.
c. Rated resources will be anonymous, but all commentary identified.
d. Identify what work is voluntary and what is paid work in the peer review system.
e. Composition of review teams (educational, discipline and technical/media experts)
5.
Policy and procedures for formal peer review.
6.
Criteria used for formal peer review, and for informal rating, e.g. quality of content, ease
of use and potential effectiveness as a teaching tool (Merlot criteria).
7.
Guidelines for submitting resources for peer review or informal commentary, including
protocols for submitting student feedback as part of the review process.
8.
Guidelines and policies for the editorial board.
9.
Appeals policy.
10.
Publication policy.
84
11.
Commentary guidelines to ensure thoughtful review, not damaging commentary (terms
of use).
12.
Policies for implementing training in peer review.
13.
Management policy for peer review conference day.
85
Conclusion
______________________________________________
The response from the sector through interviews, focus groups and reference groups, to the
development of the Carrick Exchange has been measured. Generally there is positive
anticipation, tempered by knowledge that the issues to be addressed during implementation
are not insignificant. A list of the benefits of the Carrick Exchange and challenges to adoption
that have emerged through the research are identified. The lists tend to show that the sociocultural challenges are greater than the pedagogic issues. In contrast, the pedagogic benefits
were seen to be considerable. If the Carrick Exchange can address the socio-cultural
challenges it is likely that it will have an initiative that meets the pedagogical needs of its
target audience.
Benefits
The potential benefits of the Carrick Exchange were seen to be:
Socio-cultural issues
•
•
•
Activities and personnel to support and encourage use;
Engagement strategies that target user groups with well-tailored strategies and
incentives; and
Acknowledgement as the author when other members reuse a resource contributed to the
Carrick Exchange.
Pedagogic issues
•
•
•
•
•
•
•
•
•
•
•
•
Provision of a database for the sharing and reuse of high quality professional
development and teaching resources (digital artefacts and communications), linked to
other similar databases and networks within and outside of Australia;
Access to Carrick Institute project reports, documents and output;
A well classified and maintained collection of teaching and learning resources which is
readily searchable and of sufficient substance to warrant further investigation;
A collection of resources with a discipline focus;
Links to discipline and cross-disciplinary communities and networks;
A formal peer review system based on scholarly research peer review processes;
A well maintained registry of national and international experts in teaching and learning;
Shared workspaces that support private and group projects;
Informal and timely feedback from peers on developing resources under development;
Personnel within institutions to support engagement and pedagogical processes
associated with use and reuse of resources sourced from or linked to the Carrick
Exchange (e.g. educational and staff developers);
Training in the use of the Carrick Exchange and the tools it provides; and
An annual Carrick Exchange conference for the purposes of sharing ideas and
knowledge, networking, distribution of awards, identifying resources to be contributed to
the Carrick Exchange, peer review of resources, training in peer review, and professional
development generally.
Organisational and information management issues
•
Champions placed within the Carrick Institute and universities to lead and promote the
Carrick Exchange;
86
•
•
•
•
•
Support personnel within institutions to help with information literacy, metadata creation,
and copyright and licensing issues (librarians);
The ability of the Carrick Institute to lobby senior management and achieve commitment
from key stakeholders;
Rewards and incentives for contribution, peer review and leadership relating to the
Carrick Exchange, that link to promotion and career advancement;
Policies and strategies that link home institutions with the mission of the Carrick Institute
and the Carrick Exchange; and
Active promotion and marketing of the system for its pedagogic benefits not only its
technical capabilities.
Technological issues
•
•
•
Incorporation of Web 2.0 technologies that foster connection and member creation of
resources;
Easy access to new software and tools for trial purposes; and
Seamless and intuitive integration with home institution and other databases.
Challenges
Using the organising framework of Margaryan, Currier, Littlejohn and Nicol (2006), the
clustering of perceived or actual challenges to use and engagement with the Carrick
Exchange were seen to be:
Socio-cultural issues
•
•
•
•
•
•
•
•
•
Initiating and maintaining interest in the innovation;
The real costs of contribution or membership;
Maintenance issues for the originating author after submission of the resource;
Time-poor academics whose workloads prohibit engagement in activities other than
immediate priorities;
Perceptions about increased workloads as a result of contribution to the Carrick
Exchange (contribution of artefacts or communications);
Compulsory contribution procedures (voluntary contribution preferred);
Cross-institutional rivalry and competition;
The impact of formal and informal peer review and commentary processes regarding
consumption of time and resources, cronyism and poorly moderated commentary; and
Issues of trust in: the system; the credibility of informal peer review processes; the
management of intellectual property rights by individuals and their institutions; and the
maintenance of rewards and incentives to support contribution to the system.
Pedagogic issues
•
•
•
Perceptions that the Carrick Exchange provides no additional benefits over and above the
systems and communities that are already in existence;
Currency and quality of the resources in the repository; and
Timely publication of peer reviewed resources (long timeframes compromising access
and currency of resources).
Organisational and information management issues
•
•
Inconsistent application of rights management, fears of losing ownership of resources,
intellectual and moral rights;
Lack of alignment between institutional policies on ownership of resources, intellectual
property rights and copyright, and individuals’ understanding of the policies;
87
•
•
•
•
The resource intensive nature of formal peer review;
Lack of parity between scholarship and effort in teaching, and scholarship in research;
Lack of benefits and rewards for contributing to the Carrick Exchange; and
Perceptions that the Carrick Exchange will be another interesting concept that is not
sustained into the future due to inadequate provision of management, policy and financial
resources.
Technological issues
•
Lack of ease of use of the system, including authentication and metadata procedures.
Initiating uptake in the early stages and maintaining engagement into the future, consistent
application and understanding of licensing and intellectual property policies, and cultural,
pedagogical and technical issues, all emerge as dimensions of development and
implementation that need clarity. The tension between the desire to share effective pedagogy
and resources, and the inherent competition that exists within and across the sector is of
considerable concern. There seems to be a general desire to promote best practice within the
sector, but academics and professional staff are not naive and realise they must also consider
how these practices can be factored into busy workloads and balanced against other
competing career options. Rewards and incentives will be crucial to address these issues and
encourage peer review, sharing and reuse.
Further, there are examples of sharing of learning objects and resources within and across
institutions from which lessons can be learned. At the institutional level, the new Research
Quality Framework (RQF) requirements that mandate the deposit of research quantum into
institutional repositories may effect some cultural change within the sector regarding
knowledge management and sharing. This could have positive ramifications for the Carrick
Exchange in terms of the practices of academics using repositories. However, the only
mandatory deposit that is likely to be sustained in the Carrick Exchange is the deposit of
Carrick Institute related documents and project outcomes as an obligatory responsibility of
acceptance of grant funding. This will, nonetheless, seed the Carrick Exchange collection and
help the establishment of a critical mass or resources.
The introduction of peer review processes will be an interesting new initiative to track. The
sector seems to regard formal peer review as an attractive option, and as a means of
rewarding and documenting good teaching and learning. However, the disincentives of the
formality itself, and the time taken to complete thorough peer review will be a barrier to some,
as will the risks of informal peer review (negative feedback and impact on competitive
advantage). Despite these problematic issues, there are a number of exciting new aspects to
the Carrick Exchange that could have significant benefits for the higher education sector,
such as: aggregation of a major collection of Australian professional development materials
relating to teaching and learning; collaborative workspaces for project development; formal
and informal peer review; networks and communities that support sharing and reuse of
resources; and a registry of Australian and international experts in diverse fields of teaching
and learning.
The Carrick Institute has considerable prestige within the sector, and is in a position to lead
effective change nationally and possibly internationally. Representatives of the sector in this
study have confirmed that by providing champions, acting as a sponsor, promoting
engagement, providing pedagogical support, addressing real needs and rewarding and
recognising effort for contribution and review of others’ work, the Carrick Exchange can find a
place in Australian higher education. As with any issue that requires change management,
effective communication will be the key.
88
Future research
The findings of this research have been commissioned to inform the development of the
Carrick Exchange. The next step after delivering this report to the Carrick Exchange
management team is to present the findings at a symposium of ascilite members at the 2007
annual conference. After receiving feedback from the members and developing the ideas
about engagement, sharing and reuse of resources, rewards and recognition and peer
review, the findings from the final cycle of research will also be delivered to the Carrick
Exchange management team, further informing the development process.
The findings so far indicate that further research needs to be undertaken particularly in the
area of rewards and recognition, and in refinement of the peer review model for evaluation of
educational resources. Evaluation of the initiative as a whole is another important area for
future research.
The Design-Based approach used in this research proved to be a robust methodology. Each
new data set was gathered and analysed, its significance and relevance was tested against
the literature and the views of a new group of informed practitioners. This iterative and
collaborative process strengthens critical evaluation and the final outcomes of the research. A
set of eight general design principles have been derived from the research.
1. Designing a community and a space for the sharing and development of teaching and
learning materials and processes for the higher education sector requires recognition of
the diversity of potential users and their needs. The development of community is
complex. Existing communities will not necessarily shift from their current homes to
relocate in the new space. There must be a significant advantage that will encourage
networks and communities to relocate, re-form or establish in the new space. This must
be well articulated to potential members.
2. Seed the repository with resources closely related to the mission of the sponsoring body
for the innovation. For example, a national collection of high quality professional
development and teaching resources (digital and human), linked to other similar
databases and networks of high quality resources, will be of considerable benefit to new
and experienced academics and educational support staff within the higher education
sector.
3. There is a tension between the formal and informal processes of a network and repository
such as the Carrick Exchange. The design and dynamics of the informal sharing and
review processes must be balanced against the formal requirements of standards, formal
peer review, rights management and preservation of materials.
4. Development of a community, built around a national repository with international
linkages, requires funding and management plans and strategies which address
development in a staged approach. Initial design and implementation resource
requirements and strategies will differ from those which sustain the initiative into the
future and embed it into the everyday work practices of the target audience.
5. To create a critical mass of resources (digital and human) from which the initiative can
build and grow, existing discipline communities, special interest groups and networks
should be leveraged. Members from these groups can provide leadership and models of
engagement in the identification and contribution of resources from which the collection in
all its forms can grow.
6. To be effective at the national level the community and its repository need to build on and
develop seamless integration with organisational and information management systems
that already exist within the sector. This includes aligning in particular with university
organisational strategies and policies, and systems of reward and recognition.
89
7. Change management strategies are difficult to implement, requiring time and systemic
support to be effective. Change cannot be effected through a ‘face-less” website. Key
change agents must be identified and supported using the affordances of the new
emerging community.
8. Peer review of learning and teaching resources is both an incentive and a disincentive for
contributors to a repository and community such as the Carrick Exchange. Rewards and
incentives to support contribution and peer review processes is required to surmount the
barriers.
90
References
______________________________________________
AFP. (2007). Facebook to post members profiles on online search engines. Retrieved
September 27, 2007, from
http://afp.google.com/article/ALeqM5iWsMIytjvcGtnN9Q6ob6X3YHC3BQ
Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education.
Joint Information Systems Committee (JISC) Technology and Standards Watch, February.
Retrieved August 15, 2007 from
http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf
Australian Partnerships in Sustainable Repositories APSR. Populating a repository – selected
references (bibliography). Retrieved September 27, 2007, from
http://apsr.anu.edu.au/apsrfw/rr/index.php?pid=94
Barton, M. & Waters, M. (2004). Creating an institutional repository, LEarning About Digital
Institutional Repositories, LEADIRS Workbook. Boston: MIT. Retrieved September 29, 2007,
from http://www.dspace.org//implement/leadirs.pdf
Bates, M., Loddington, S., Manuel, S. & Oppenheim, C. (2006). Rights and Rewards project,
Academic survey: Final report. Loughborough University & JISC. Retrieved July 16, 2007,
from
http://rightsandrewards.lboro.ac.uk/files/resourcesmodule/@random43cbae8b0d0ad/1137423
150_SurveyReport.pdf
Bradley, C. & Boyle, T. (2004). The design, development and use of multimedia learning
objects. Journal of Educational Multimedia and Hypermedia, 13(4), 371-389.
Breslin, C., Nicol, D., Grierson, H., Wodehouse, A., Juster, N. & Ion, W. (2007). Embedding
an integrated learning environment and digital repository in design engineering education:
Lessons learned for sustainability. British Journal of Educational Technology, 38(5), 805-816.
Bruce, C. (1997). Peer Review: A Handbook, Queensland University of Technology.
Retrieved September 23, 2007, from http://sky.fit.qut.edu.au/~bruce/peerreviewhanbook/
Dalziel, J. (2002). Reflections on the COLIS (Collaborative Online Learning and Information
Systems) Demonstrator project and the “Learning Object Lifecycle”. In A. Williamson, C.
Gunn, A. Young & T. Clear (Eds.), Winds of Change in the Sea of Learning: Proceedings of
the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary
Education. Auckland, New Zealand: UNITEC Institute of Technology.
Dalziel, J. (2006). Building communities of designers, In H. Beetham and Sharpe, R. (Eds.),
Design for Learning: Rethinking Pedagogy for a Digital Age. RoutledgeFalmer: London.
Design-Based Research Collective. (2003). Design-based research: An emerging paradigm
for educational inquiry. Educational Researcher, 32(1), 5-8.
Desire (2000). Desire Information Gateways Handbook. Retrieved March 31, 2007, from
http://www.desire.org/handbook/2-1.html
Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan,
and P. MacGillis (Eds.), The guidebook for performance improvement: Working with
individuals and organizations. San Francisco: Pfeiffer.
91
Ehrmann, S., Gilbert, S., & McMartin, F. (2007). Factors affecting the adoption of facultydeveloped academic software: A study of five icampus projects. Retrieved September 19,
2007, from http://www.tltgroup.org/iCampus/iCampus_Assessment_Full.pdf
Ely, D. (1999). Conditions that facilitate the implementation of educational technology
innovations. Educational Technology, 39(6), 23—27.
Evans, V. (2007). Networks, connections and community: Learning with social software.
Australian flexible Learning Framework, Australian National Training Authority and
Commonwealth of Australia. Retrieved July 10, 2007, from
http://www.flexiblelearning.net.au/flx/go/pid/377
Foster, N.F. & Gibbons, S. (2005). Understanding faculty to improve content recruitment for
institutional repositories. D-Lib Magazine. 11(1). Retrieved September 21, 2007, from
http://www.dlib.org//dlib/january05/foster/01foster.html
Gosper, M., Woo, K., Gibbs, D., Hand, T., Kerr, S. & Rich, D. (2005). The selection and use of
learning objects for teaching: User perspectives. In J. Dalziel, R. Philip & J. Clare (Eds.), The
COLIS Project: Collaborative Online Learning and Information Services, (pp. 99-119).
Macquarie University E-Learning Centre of Excellence: Sydney, NSW.
Gunn, C., Woodgate, S. & O’Grady, W. (2005). Repurposing learning objects: An alternative
to reusability? ALT-J, 13(3), 189-200.
Hand, T., Gosper, M., Woo, K., Gibbs, D., Kerr, S. & Rich, D. (2004). Learning objects: User
perspectives on the conditions surrounding their use. Proceedings of the Ed-Media 2004
World Conference on Educational Multimedia, Hypermedia and Telecommunications, 21-26
June, Lugano, Switzerland, 66–72.
Health Education Assets Library (HEAL) (2006). Editorial Policy. Retrieved September 22,
2007, from http://www.healcentral.org/services/policies/HEAL_Editorial_Policy_v2.0.pdf
Henty, M. (2007). Ten major issues in providing a repository service in Australian Universities.
D-Lib Magazine, 13(5/6). Retrieved May 16, 2007, from
http://www.dlib.org/dlib/may07/henty/05henty.html
Hummel, H., Tattersall, C., Brugos, D., Brouns, F., Kurvers, H. and Koper, R. (2005).
Facilitating participation: From EML web site to the learning network for learning design.
Interactive Learning Environments, 13(1-2), 55–69.
Jorum. (2007). Contributing to Jorum. Contributing to Jorum for Beginners. Retrieved
September 5, 2007, from http://www.jorum.ac.uk/contributors/chelp/guidelines.html
Kenny, J. (2003). A research-based model for managing strategic educational change and
innovation projects. Annual Conference Proceedings of HERDSA (the Higher Education
Research and Development Society of Australasia), University of Canterbury, Christchurch,
NZ, 6–9 July 2003, Retrieved August 11, 2007, from
http://surveys.canterbury.ac.nz/herdsa03/pdfsref/Y1102.pdf
Koppi, T. & Lavitt, N. (2003) Future directions. In E. Duval, W. Hodgins, D. Rehak and R.
Robson (Eds.) Learning Objects 2003 Symposium: Lessons learned, questions asked. EDMEDIA 2003 World Conference on Educational Multimedia, Hypermedia &
Telecommunications, Honolulu, Hawaii, USA, 24 June 2003, 39-43. Retrieved May 2, 2007,
from http://www.aace.org/conf/edmedia/LO2003Symposium.pdf
Learning Technology Standards Committee (2002). IEEE Standard for Learning Object
Metadata. IEEE Standard 1484.12.1, Institute of Electrical and Electronics Engineers, New
92
York. Retrieved September 18, 2007, from
http://ltsc.ieee.org/wg12/files/LOM_1484_12_1_v1_Final_Draft.pdf
Lefoe, G., O'Reilly, M., Parrish, D., Bennett, S., Keppell, M., & Gunn, C. (2007). The Carrick
Exchange: Not just another repository. Proceedings of the Ed-Media World Conference on
Educational Media, Hypermedia and Telecommunications, 25-29 June, Vancouver, Canada,
108-113.
Littlejohn, A. (Ed.) (2003). Reusing online resources: A sustainable approach to e-learning.
London and Sterling: Kogan Page.
Margaryan, A., Currier, S., Littlejohn, A. & Nicol, D. (2006). Community dimensions of learning
object repositories, Retrieved CD-LOR deliverable 1: Learning communities and repositories.
Retrieved August 30, 2007, from http://academy.gcal.ac.uk/cdlor/CDLORdeliverable1_learningcommunitiesreport.doc.
Margaryan, A. & Littlejohn, A. (2007). Cultural dimensions of learning object repositories.
Proceedings of the Ed-Media World Conference on Educational Media, Hypermedia and
Telecommunications, Vancouver, Canada, 25 -29 June, 4335-4343.
Margaryan, A., Milligan, C., & Douglas, P. (2007). Community dimensions of learning object
repositories, CD-LOR deliverable 9: Structured guidelines for setting up learning object
repositories. Retrieved July 23, 2007, from http://academy.gcal.ac.uk/cd-lor/documents/CDLOR_Structured_Guidelines_v1p0.pdf
Mason, J., Macnamara, D. & Galatis, P. (2007). An investigation into digital rights
management issues in the Australian higher education sector: A discussion paper.
Commissioned by Education.au. Internal report.
McDonald,J. & Star, C. (2006). Designing the future of learning through a community of
practice of teachers of first year courses at an Australian university. In R. Philip, A Voerman &
J. Dalziel (Eds), Proceedings of the First International LAMS Conference 2006: Designing the
Future of Learning (pp.65-76). 6-8 December 2006, Sydney: LAMS Foundation.
http://lamsfoundation.org/lams2006/papers.htm
McKenzie, J., Alexander, S., Harper, C. & Anderson, S. (2005). Dissemination, Adoption and
Adaptation of Project Innovations in Higher Education. A report for the Carrick Institute for
Learning and Teaching in Higher Education. Sydney: University of Technology.
McMartin, F. (2004). MERLOT: A model for user involvement in digital library design and
implementation. Journal of Digital Information, 5(3), Retrieved September 21, 2007, from
http://jodi.tamu.edu/?vol=5&iss=3
Monahan, C. & Owies, D. (2005). The DOMS at Deakin. Proceedings of the Educause
Conference of Australasia, Auckland, April 5-8, 2005. Retrieved September 10, 2007, from
www.educause2005.auckland.ac.nz/interactive/papers/A20.pdf
Najjar, J., Ternier, S. & Duval, E. (2004). User behaviour in learning objects repositories: An
empirical analysis. Proceedings of the Ed-Media 2004 World Conference on Educational
Multimedia, Hypermedia and Telecommunications, 21-26 June, Lugano, Switzerland, 43734378.
Nature (2006). Nature's peer review trial. December. doi:10.1038/nature05535. Retrieved
September 20, 2007, from http://www.nature.com/nature/peerreview/debate/nature05535.html
Nature (2007). Nature Precedings is live. Retrieved September 28, 2007, from
http://blogs.nature.com/peer-to-peer/2007/06/nature_precedings.html
93
Nesbit, J., Belfour, K. & Vargo, J. (2002). A convergent participant model for evaluation of
learning objects. Canadian Journal of Learning and Technology, 28(3). Retrieved August 3,
2007, from http://www.cjlt.ca/content/vol28.3/nesbit_etal.html
Pappalardo, K., Fitzgerald, A., Fitzgerald, B., Kiel-Chisholm, S., O'Brien, D. & Auston, A.
(2007). A guide to developing open access through your digital repository. DEST funded
Open Access to Knowledge Law Project OAK Law Project. Retrieved September 27, 2007,
http://eprints.qut.edu.au/archive/00009671/01/9671.pdf
Philip, R. (2007). Adaptable and reusable learning designs: Will they be shared? Proceedings
of the Ed-Media World Conference on Educational Media, Hypermedia and
Telecommunications, Vancouver, Canada, 25 - 29 June 2007, 2980-2985.
Phillips, I., Aspin, M., Hull, D. & Oxley, C. (2004). Evaluation of the Australian Flexible
Learning Framework 2000-2004, Part E – Evaluation Infrastructure. Report for the Australian
Flexible Learning Framework, Canberra: ANTA. Retrieved 20 May, 2007, from
http://flexiblelearning.net.au/aboutus/keydocuments.htm
Phillips, R., Rai, R., Sudweeks, F., Gururajan, R., Jones, M., Shiers, D. & O’Neil, R. (2005).
Use and usability of learning objects within the COLIS Demonstrator framework. In J. Dalziel,
R. Philip, & J. Clare (Eds.) The COLIS Project: Collaborative Online Learning and Information
Services. Macquarie University E-Learning Centre of Excellence (MELCOE), Macquarie
University: Sydney.
Phillips, R., Orrell, J. & Millea, J. (2007). Creativity in the Envisioning of a Collaborative
Learning Space: First stages in the development of the Carrick Exchange. Proceedings of
AusWeb 07, Coffs Harbour, NSW, 30 June - 4 July. Retrieved July 30, 2007, from
http://ausweb.scu.edu.au/aw07/papers/refereed/phillips/
Reeves, T. (2000). Enhancing the worth of instructional technology research through ‘design
experiments’ and other development research strategies. Paper presented at the Annual
Meeting of the American Educational Research Association, 27 April, New Orleans, LA.
Ringan, N., Corley, L. & Campbell, L. (2005). Evaluation of the COLIS Framework form a UK
perspective. In J. Dalziel, R. Philip & J. Clare (Eds.), The COLIS Project: Collaborative Online
Learning and Information Services, Macquarie University E-Learning Centre of Excellence:
Sydney, NSW. Ch. 11.
Rogers, A. (2006). Get Wiki with it. Wired. 14 September, pp.30-32. Retrieved September 18,
2007, from www.wired.com/wired/archive/14.09/start.html?pg=3 - 33k
Rogers, E. (2003). Diffusion of innovations. New York: Free Press.
Scott, G. (1999). Change matters: Making a difference in education and training. St Leonards
NSW: Allen and Unwin.
Shepherd J. (2007). Help Wanted. The Guardian, 4 September. Retrieved September 22,
207, from http://education.guardian.co.uk/higher/research/story/0,,2161680,00.html
Siemens, G. (2007). Scholarship in an age of participation. Elearnspace, March 27. Retrieved
September 29, 2007, from http://www.elearnspace.org/Articles/journal.htm
Smith, C. (2005). Guidebook for individuals embarking on peer review of teaching, The
Teaching and Educational Development Institute, University of Queensland. Retrieved
September 23, 2007, from
http://www.tedi.uq.edu.au/downloads/evaluations/Guidebook_PeerReview.pdf
94
Souza, K.H. & Persily, G. (2005). National Peer Review of Digital Learning Materials in the
Health Sciences. Retrieved September 22, 2007 from
http://cit.ucsf.edu/events/event_mar05.php
Stuckey, B. & Arkell, R. (2006). Development of an e-learning knowledge sharing model.
Knowledge Sharing Services Project. Australian Flexible Learning Framework. Retrieved July
23, 2007, from
http://www.flexiblelearning.net.au/…/kjohnson/public/Development_of_an_%20elearning_knowledge_sharing_model.pdf
Taylor, P. & Richardson, A. (2001). Validating scholarship in university teaching. Evaluations
and Investigations Programme. Canberra, ACT: DETYA, Commonwealth of Australia.
University of Queensland. (2007). Report of the Working Party on the Diversity of Academic
Roles. Internal review. Retrieved September 27, 2007, from
http://www.uq.edu.au/shared/resources/personnel/appraisalAcad/DiversityofAcademicRolesR
eportApr07.doc
University of Queensland, Griffith University & Queensland University of Technology (2004).
Peer review of learning materials report: A collaborative project involving the University of
Queensland, Griffith University and Queensland University of Technology. 30 August 2004.
van den Akker, J. (1999). Principles and methods of development research. In J. van den
Akker, N. Nieveen, R.M. Branch, K.L. Gustafson, & T. Plomp, (Eds.), Design methodology
and developmental research in education and training (pp. 1- 14). The Netherlands: Kluwer.
Weedon, E., Bricheno, P. & Chidwick, N. (2004). The Impact of Large-scale Network
Learning on Further and Higher Education Institutions. JISC report, December 2004.
Retrieved July 12, 2007, from http://www.sfeuprojects.org.uk/inlei/project_summary.html
Wenger, E., McDermott, R. & Snyder, W. (2002). Cultivating communities of practice. Boston
MA: Harvard Business School Press.
Wetterling, J., & Collis, B. (2003). Sharing and reuse of learning resources across a
transnational network. In A. Littlejohn (Ed.), Reusing online resources. London: Kogan Page.
Wiley, D.A. (2000). Connecting learning objects to instructional design theory: A definition, a
metaphor, and a taxonomy. In D.A. Wiley (Ed.), The Instructional Use of Learning Objects.
Retrieved March, 26, 2007, from http://reusability.org/read/chapters/wiley.doc.
Worthen, B., Sanders, J. & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches
and practical guidelines (2nd Ed.). New York: Longman.
95
Appendices
______________________________________________
Appendix A: Glossary of terms
Term
Definition for the purposes of this research
Commentary
Informally contributed feedback or narrative. This may be contributed, e.g. to an
online forum as part of a discussion on issues around teaching, or as part of an
assessment of a resource within the Carrick Exchange, contributing to informal peer
review processes.
Networks,
communities,
and
Communities of
Practice (CoP)
Network: A system of interconnected people (Macquarie Dictionary)
Contribution
Learning object
(LO)
Community: A group of people with a common background or shared interest.
Community of Practice: A more specific term than community. “A group of people who
share a concern, a set of problems, or a passion about a topic, and who deepen their
knowledge and expertise . . . by interacting on an ongoing basis” (Wenger et al. 2002,
p.4). CoPs are characterised by a domain of knowledge, a community and shared
practice.
Contribution to the Carrick Exchange includes:
a)
digital artefacts (resources such as learning objects, teaching support materials,
curriculum outlines, project reports, etc.); or
b)
a communication of some kind, such as a contribution to a discussion or a blog,
or as a comment on another resource in the Carrick Exchange.
The term is often used very broadly, but also has a very specific meaning. Some of
the various definitions are as follows. The broad definition given by Wiley is probably
the version most relevant to the Carrick Exchange context.
“Any digital resource that can be reused to support learning.” (Wiley 2000, p.7)
“An aggregation of one or more digital assets, incorporating metadata, which
represents an educationally meaningful, stand-alone unit” (Dalziel, 2002)
“Any entity, digital or non-digital, which can be used, re-used or referenced during
technology supported learning” (The Learning Technology Standards Committee,
IEEE definition, 2002)
Learning object
repository
(LOR)
A database of learning objects. Often used broadly to refer to any digital collection of
learning and teaching resources. The Carrick Exchange can be regarded as a
community of those who lead, manage and teach in higher education. The community
is supported by a LOR.
Peer review
The evaluation of teaching resources and commentary contributed to the Carrick
Exchange. Review is conducted by “peers” or equals. It may be:
a)
formal and conducted by teams of experts;
b)
for quality assurance purposes only: or
c)
informal and include discussion, feedback and comments made on resources
and ideas contributed to the Carrick Exchange.
96
Term
Definition for the purposes of this research
Resources
A broad term for any digital artefact relating to teaching and learning that can be
stored electronically, contributed and shared through the Carrick Exchange. For the
purposes of this research, the words “resources” and “materials” have been used
interchangeably.
a) It may, e.g. include formal and informal items, products, processes, learning
objects, learning designs, activities, assessments, reviews, criteria, reports, plans.
b) In this context resources also include communications – comments made about
items contributed to the Carrick Exchange, or discussion sessions. Once contributed
this “commentary” becomes “resources” for further comment or development by
others.
c) People can also be considered to be important resources for the Carrick Exchange,
as they share, network, provide expertise, peer review and comment on matters
relating to teaching and learning.
97
Appendix B: Repositories, associations and projects referred
to in the report
Repository, association or
project
ACELL
Advancing Chemistry by
Enhancing Learning in the
Laboratory
Description
ACELL is a peer reviewed database of resources for undergraduate
chemistry. ACELL aims to provide educationally sound chemistry
experiments, evaluated by students and academic staff; professional
development opportunities for chemistry academic staff; and facilitate the
development of a community of practice in chemistry education.
http://acell.chem.usyd.edu.au/homepage.cfm
AESharenet
AEShareNet's role is to help create a comprehensive, efficient system that
is needed to streamline reciprocal copyright licensing practices in
education.
http://www.aesharenet.com.au/
ANDS
ANDS has outreach services which bring in data that needs to be archived
for the nation; it has stewardship services, metadata, and federation
services.
the Australian National
Data Service
APSR
Australian Partnership for
Sustainable Repositories
ARROW
Australian Research
Repositories Online to the
World
APSR aims to establish a centre of excellence for the management of
scholarly assets in digital format. The partnership includes major research
universities, the National Library of Australia, and APAC (the Australian
Partnership for Advanced Computing).
http://www.apsr.edu.au/
The ARROW project identifies and tests software or solutions to support
best practice institutional digital repositories comprising e-prints, electronic
theses, e-research and electronic publishing. The project is funded by the
Australian Commonwealth Department of Education, Science and
Training, under the Research Information Infrastructure Framework for
Australian Higher Education.
http://www.arrow.edu.au/
The Australasian Society
for Computers in Learning
in Tertiary Education
A society for those involved in tertiary computer-based education and
training, including educational interactive multimedia. It provides a forum to
stimulate discussion in the educational use of technology as well as
promoting research and evaluation.
http://www.ascilite.org.au/
Australian Access
Federation Project
National infrastructure and authentication project.
http://www.aaf.edu.au/)
Australasian Association
for Engineering Education
AAEE is a professional association committed to fostering excellence and
innovation in engineering education, bringing together people across
Australia and New Zealand.
http://www.aaee.com.au/
ascilite
AAEE (A2E2)
Australian Flexible
Learning Framework
The Australian Flexible Learning Framework provides the vocational
education and training (VET) system with e-learning skills, professional
development opportunities, products, resources and support networks to
meet today's increasingly technology-driven learning environment. The
Framework is a national strategy collaboratively funded by the Australian
Government and all states and territories.
http://www.flexiblelearning.net.au/flx/go/home
98
Repository, association or
project
Description
Carrick Exchange
The Carrick Exchange is an online service created by the Carrick Institute
of Australia that will provide learning and teaching resources and support
communication and collaboration across the higher education sector in
Australia and internationally. It will contain collaborative workspaces, tools
and communication facilities.
http://www.carrickexchange.edu.au/index.html
Carrick Institute
The mission of the Carrick Institute for Learning and Teaching in Higher
Education is to promote and advance learning and teaching in Australian
higher education.
http://www.carrickinstitute.edu.au/
CD-LOR
The CD-LOR project in the United Kingdom aims to identify and analyse
the factors that influence practical uptake and implementation of learning
object repositories within a range of different learning communities. The
aim is to benefit UK higher and further education.
http://www.academy.gcal.ac.uk/cd-lor/
Community Dimensions of
Learning Object
Repositories
CLOE
the Co-operative Learning
Object Exchange
A collaboration between Ontario universities and colleges in Canada for
the development, sharing, and reuse of multimedia-rich learning
resources. This occurs through the CLOE Learning Object repository.
http://cloe.on.ca/index.html
Cornell University Physics
eprint archive
Cornell University Physics eprint archive.
http://www.arxiv.org
Council for the
Humanities, Arts and
Social Sciences (CHASS)
As a peak body for the humanities arts and social sciences, CHASS aims
to represent the interests of the sector; to promote the contribution of the
sector to government, industry and the public; to provide a forum for
discussion between the humanities, arts and social sciences sectors in
Australia; and to build the innovative capacity of Australia, through better
linkages between the sector and industry, science and technology.
http://www.chass.org.au/
DIDET
The DIDET Project is led by the University of Strathclyde, Stanford
University and Olin College. The aim is to enhance learning through
students’ participation in global team-based design engineering projects.
http://www.didet.ac.uk/
Edna
Free online network for educators housing online resources and offering a
collaborative network for the education and training community.
http://www.edna.edu.au/edna/go
Education Network
Australia
HERDSA,
Higher Education
Research and
Development Society of
Australasia
Scholarly society for improvement of teaching and learning in higher and
tertiary education.
http://www.herdsa.org.au/
Intute
The Intute service is created by a network of UK universities and partners.
Subject specialists select and evaluate websites in the database and write
high quality descriptions of the resources.
http://www.intute.ac.uk/
LORN project
Part of the Australian Flexible Learning Framework, LORN is an initiative
supporting the Australian VET community, providing access to high-quality
learning and teaching resources across a number of repositories.
http://lorn.flexiblelearning.net.au/lorn/go/home/pid/119
Learning Object
Repository Network
99
Repository, association or
project
MAMS
Meta Access Management
Project
MERLOT
Multimedia Educational
Resource for Learning and
Online Teaching
Description
The MAMS project is concerned with the infrastructure and integration of
multiple solutions to managing authentication, authorisation and identities
across systems, together with common services for digital rights, search
services and metadata management.
http://www.melcoe.mq.edu.au/projects/MAMS/
A user-centred, searchable collection of peer reviewed, higher education,
online learning materials created by registered members, and a set of
faculty development support services. where learning materials and
pedagogy are shared by the members.
http://www.merlot.org/merlot/index.htm
Minister of
Communications Digital
Strategy Advisory Group
(NZ)
The NZ Digital Strategy is a government action plan for ensuring New
Zealand is a world leader in using information and technology.
http://www.digitalstrategy.govt.nz/
MIT open courseware
model
Massachusetts Institute of Technology
http://ocw.mit.edu/index.html
NCRIS
Through NCRIS, the Australian Government provides funding to
researchers for major research facilities, supporting infrastructure and
networks necessary for world-class research.
http://www.ncris.dest.gov.au/
National Collaborative
Research Infrastructure
Strategy
OAK Law Project
Open Access to
Knowledge
ODLAA
Open and Distance
Learning Association of
Australia
RRBIR
Rights and Rewards in
Blended Institutional
Repositories
RUBRIC
Regional Universities
Building Research
Infrastructure
Collaboratively
SNUSE, Sydney Basin
Network of University
Science Educators
The project aims to make sharing knowledge across domains, and the
world, both legal and efficient. The project will develop legal protocols for
managing copyright issues in an open access environment. At a technical
level it will investigate provision and implementation of a rights expression
language. The project will integrate with existing open access repositories
at both legal and technical levels.
http://www.oaklaw.qut.edu.au/about
ODLAA is a professional association that aims to advance the practice and
study of distance education in Australia; foster communication between
distance educators; and maintain and extend links with other national and
international associations with related aims and objectives.
http://odlaa.une.edu.au/
University of Loughborough institutional repository, UK
http://rightsandrewards.lboro.ac.uk/index.php?section=1
This DEST funded project aims to: build capability across smaller research
universities in the IRUA group and country areas; enable the research
output of those institutions to be available trans-nationally; and contribute
to the research mission of higher education in Australia and internationally
through collaboration with New Zealand partners. Partners are: the
University of Southern Queensland (USQ), the University of New England,
the University of the Sunshine Coast, the University of Newcastle, and
Massey University in New Zealand.
http://www.rubric.edu.au/
Network of Science Educators
http://www.science.unsw.edu.au/guide/slatig/snuse.html
100
Appendix C: Key practitioners interviewed in Cycle 1
Category
Stakeholders
Anticipated
contribution to
project
Project focus
area
Comparisons
with international
initiatives
Models of use
and peer review
Comparisons
with national
initiatives
Resource
identification and
contribution
Comparisons
with national
initiatives
Models of use,
peer review
Institutional
repositories
perspective
Peer review,
models of use,
resource
contribution
MERLOT (USA)
Intute (UK)
CLOE (Canada)
International
connections (6)
MOD4L Project (UK)
NZ E-learning Collaborative Development Fund &
Open Educational Resources (NZ)
EduForge (NZ)
Digital Repository Team, Griffith University
National
initiatives (3)
RUBRIC project (regional initiative)
MAMS – Meta Access Management Systems
National Research and Policy Advice Project Australian Flexible Learning Framework
Technical and
Further
Education
(TAFE ) sector
(4)
Learning Object Repository Network Project
(LORN) - Australian Flexible Learning Framework
Teaching and Learning Exchange
TAFE (general)
Rights and Rewards in Blended Institutional
Repositories (UK)
Institutional
repositories (2)
Deakin University
RMIT University
Associations
(discipline and
professionally
based) (10)
Open and Distance Learning Association of
Australia (ODLAA)
Council for the Humanities, Arts and Social
Sciences (CHASS)
Association’s
needs for
communicating
and resource
sharing
Resource
identification and
contribution,
peer review,
models of use,
engagement
Sydney-Basin Network of University Science
Educators (SNUSE)
Australian Council of Deans of Science
Australian Council of University Art and Design
Schools
Engineering Australia
Australian Mathematical Society
101
Category
Stakeholders
Anticipated
contribution to
project
Project focus
area
Higher Education Research & Development
Association (HERDSA)
Australian Council of Deans of Science
Council of Australasian Directors of Academic
Development
User
perspectives on
the
development,
utilisation and
review of digital
repositories
Resource
identification and
contribution,
models of use,
engagement
Deputy/Pro
Vice-Chancellor
(1)
User
perspectives on
the
development,
utilisation and
review of digital
repositories
Models of use,
peer review
Carrick grant
holders and/or
Carrick Fellows
(1)
Project needs for
communication
and resource
sharing
Resource
identification and
contribution
Australian
Tertiary
Institutional
Users
(educational
developers) (2)
University of Queensland
University of Wollongong
102
Appendix D: Focus group participants – Cycle 2
Universities represented and participants’ details
Universities represented (22)
Melbourne (12)
Brisbane (4)
Sydney (6)
Deakin University, Vic
Southern Cross University, NSW
Macquarie University, NSW
La Trobe University, Vic
Central Queensland University,
QLD
The University of Sydney, NSW
Monash University, Vic
RMIT University, Vic
The University of Melbourne, Vic
Griffith University, QLD
University of New South Wales,
NSW
University of Southern
Queensland, QLD
University of Technology,
Sydney, NSW
Victoria University, Vic
Flinders University, SA
University of Western Sydney,
NSW
The University of Adelaide, SA
University of Wollongong, NSW
Curtin University of Technology,
WA
Edith Cowan University, WA
Murdoch University, WA
The University of WA, WA
Participants’ details
(Note: some participants represented more than one category)
Frequency
Staff Development
8
Members of cross-institutional teams
(e.g. Carrick Institute or other collaborative project)
7
Educational Development
6
Librarians
5
Repository representatives
3
‘Early adopter’ in the use of technology
2
Academics interested in learning and teaching but with limited
experience of technology
2
Females
13
Males
9
103
Appendix E: Engagement table specific example – Carrick Grant Holders
The following table has been adapted from Dormant’s 18 (1997, p144) stages of awareness framework. This example has been developed to illustrate how one
specific user group (Carrick Grant Holders) might be engaged with the Carrick Exchange, depending on their level of awareness.
Stage of awareness
General strategy
Carrick Exchange strategy
Awareness
Advertise
Advertise
• Passive regarding the
change
• Little/no information
about change
• Little/no opinion about
change
Be an advertising agent
Provide information prior to and throughout the grant application process.
Provide information on the website, email & phone details for 1st contact
Appeal to his or her needs
and wants
Support roles &
facilitating personnel
Carrick Institute Grant
Scheme coordinators,
Provide reasons for need for the CE and benefits such as:
CE Managers &
Champions,
a)
central place for communication, such as project meetings;
CE promotional staff
b)
project content management, within private space; make publicly available progress
of grants & outcomes;
Local Institution
Champions
c)
broaden network - point of contact for other projects.
Be credible and positive
Curiosity
Inform
Inform
• More active regarding
change
Identify specific concerns
Identifying what CE offers grant holders such as public dissemination space for projects,
private and public project workspaces, shared resources, links to literature and other
previous and current related projects (either Carrick or not), and networks of experts.
Provide clear information
about concerns
Provide information as website documentation and at short f2f sessions or as part of
other Carrick Institute related activities on receipt of grant (include in grant criteria).
Emphasise pluses,
acknowledge minuses
Have evidence of national & international benefits for using the CE.
Local Carrick Grant
coordinators
Have evidence of a quality database of professional development & teaching resources
derived from other Carrick Institute projects
Members of the Grant
team
• Express personal job
concerns
• Asks questions about
own work and change
CE discipline-based
Champions & collections
managers
Carrick Institute Grant
Scheme coordinators,
18
Dormant, D. (1997). Planning change: past, present, future. In R. Kaufman, S. Thiagarajan, and P. MacGillis (eds.), The guidebook for performance improvement: working with
individuals and organizations. San Francisco: Pfeiffer.
104
Stage of awareness
General strategy
Carrick Exchange strategy
Support roles &
facilitating personnel
Envisioning
Demonstrate
Demonstrate
• Active regarding
change
Give success images
Develop training materials accessible from the CE.
CE & Local institutional
Champions,
Provide demonstrations
Provide 1-2 hour f2f demonstrations (f2f and/or online).
Staff & Educational
Developers
• Expresses workrelated job concerns
Provide digital stories, showcasing & case studies on the CE demonstrating use of the CE
by other grant holder.
• Asks questions about
how change works
Present a template or exemplar about how groups could engage with the system for public
and private spaces.
Connect with peer users
Grant holders and
others with experience
in the use of the CE
Use ‘push’ technology, customisable to individuals’ needs, to connect groups and members
(e.g. email, RSS feeds, newsletters and alerts).
Identify related project material, other experts, links to the literature & related projects,
previous workshop material.
Encourage registration so as to view activities of available communities & networks.
Tryout
Train
Train
• Active regarding
change
Provide effective training
Support & provide local f2f training initiatives
CE Trainers
Offer training in peer review and evaluation.
Help desk supporters
• Has opinions about
change
• Interested in learning
how-to
Provide job aids, checklists
Promise technical follow-up
Phone support and ‘just-in-time’ mentoring for establishing project space and using the
tools.
Provide documentation, fact sheets, Frequently Asked Questions (FAQs), guidelines,
tutorial animations contact details of institutional Champions.
Librarians
Educational Developers
Technical Writers
Institutional Champions
Provide forums on CE for getting started, for Q&A, pedagogical and technical
Provide a database of members available to support others in specialist areas
105
Stage of awareness
General strategy
Carrick Exchange strategy
Use
Support
Support
• Active regarding
change
Provide necessary
technical help
Develop documentation and resources to support engagement.
• Uses change on the
job
• Asks detailed
questions about use
Provide pedagogical help
Provide help desk and centralised support for engagement.
Provide support for embedding reuse and sharing into the curriculum e.g. activities, forums,
conferences and guidance at the personal and group level
For ease of use provide support clearing copyright, metadata entry and checking, rights
management and version control
Modelling good practice in the use of communication and collaboration tools (through
hosting and facilitating exchange).
Engage people f2f by conducting conferences and workshops which focus on resource
identification & submission, and training to undertake peer review of submitted material.
Provide recognition Provide
reinforcement
Support roles &
facilitating personnel
CE and institutional
information specialists
Educational and staff
developers, Carrick and
local Champions, CE
community members
Librarians, copyright
managers
Carrick Institute
managers, Institutional
managers
Provide recognition and awards for contribution (e.g. commentary and participation) and
excellence which support career advancement; link with promotional policies and
procedures, e.g. certificates, Carrick awards.
Source: Adapted from Dormant 1997;
Key: CE = Carrick Exchange
May need different strategies for individuals and groups
106
Appendix F: Roles identified for the Carrick Exchange
A number of roles have been identified in the ascilite research. Some are identified below.
Role
Focus
Activity
Carrick Exchange
Manager
Overview
Promote and manage the Carrick Exchange so that it
continues to grow the collection, providing relevant,
accurate and sustainable resources (human and digital).
Carrick Exchange
Champion
Engagement
Promotion of the Carrick Exchange amongst the
academic community. Engages with local Carrick
Exchange Champions, research and education focus.
High level engagement at the national level with
Executive in each university. Contributes to the annual
conferences on Peer Review and Resource Identification
and Contribution.
Institutional
Champions
Engagement
Located at each Australian university. Promotion and
education about the Carrick Exchange at the institutional
level. Liaison with the Carrick Exchange Champions and
staff within own institution.
Technical Writer
Engagement
Generate promotional and informational material for the
website. Suggest various media for presentation. Work
with discipline specialist and Carrick Exchange Manager.
Contributions Editor
(digital resources)
Resource
contribution and
identification
Sort and select or reject digital resources submitted to the
Carrick Exchange for deposit. Send selected items for the
quality assurance process and/or peer review. Liaise with
authors and contributors. Promote contribution to the
Carrick Exchange and identification of resources.
Contributes to the annual conferences on Resource
Identification and Contribution.
Contributions Editor
(informal
commentary)
Resource
contribution and
identification
Monitor specified, but not all, discussions. Mediate and
monitor disputes. Generate activities on certain forums
that contribute to engagement. Support and encourage
group participation in the private workspaces. Promote
contribution to the Carrick Exchange and identification of
resources (human and digital). Contributes to the annual
conferences on Resource Identification and Contribution.
Collections Manager
Resource
contribution and
identification
Probably discipline based, but may be generalist in early
stages of the Carrick Exchange. Identification of
resources to maintain and sustain the collection. Promote
contribution to the Carrick Exchange. Contributes to the
annual conferences on Resource Identification and
Contribution.
Quality Assurance
Manager and/or
Administrative Officer
Peer Review and
commentary
Manages process of quality assurance (technical,
copyright & basic educational integrity checks). Liaison
with Peer Review Chair. Contributes to the annual
conferences on Resource Identification and Contribution.
Copyright officer
Resource
contribution and
identification
Manages copyright checks on deposited items, promotes
understanding about digital rights issues, contributes to
forums on digital rights, intellectual property, moral rights
etc. Contributes to the annual conferences on Resource
Identification and Contribution. Liaises with individuals
and institutional representatives on copyright and
licensing matters.
107
Role
Focus
Activity
Editorial Board
Members
Peer Review and
commentary
Members provide guidance and direction for the peer
review processes and about the collection and its
integrity. Contribute to the annual conference which
supports training in peer review and deliberation on the
issues of peer review of technology based educational
resources.
Peer Review Editorial
Board Chair
Peer Review and
commentary
Manage and lead the Editorial Board’s activities,
supervising and overseeing the peer review processes,
inviting reviewers to participating and selecting reviewers
as required. Contributes to the annual conferences on
Peer Review.
Reviewers for formal
peer review
Peer Review and
commentary
Specialists in discipline areas, education, or
media/information who comprise the review panel for the
purpose of evaluating resources accepted for the peer
review process. These positions may be a voluntary
unless the amount of work becomes sufficient to warrant
payment for regular work. Each review panel may consist
of a different set of members (other than the Editor),
according to the nature of the resource to be peer
reviewed.
Staff and Educational
Developers/Designers
and Librarians
Engagement,
Resource
contribution and
identification, and
University based. Specialists in curriculum, educational
technology, professional development and information
management located within institutions who have a key
role in promoting the Carrick Exchange to other staff
within their institutions. Conduct workshops and work
one-to-one with interested personnel. Liaison with
institutional Carrick Exchange Champions.
Peer Review and
commentary
Technical Helpdesk
Training Support
Officer
Provides just-in-time support for personnel using the
Carrick Exchange, using email, chat, and phone. May
also conduct web conferencing training sessions.
108
Appendix G: Examples of formal peer review processes for
learning and teaching resources
CLOE: Cooperative Learning Object Exchange, http://cloe.on.ca/
About: CLOE is a collaboration between Ontario universities and colleges for the development, sharing,
and reuse of multimedia-rich learning resources through the CLOE Learning Object repository. Each
institution develops multimedia learning resources to address instructional challenges shared by the
other partners, or uses/adapts learning objects created at another partner institution to use at their own
institution.
Considerations: Peer review process for learning objects submitted to CLOE has the following
characteristics- Clear (consistent with author guidelines); Recognised (being a reviewer is institutionally
valued); Automated (neither onerous nor a disincentive for reviewer or reviewee; automation of certain
tasks should occur); Familiar (like other forms of scholarly peer review in the disciplines); Thorough
(capacity to evaluate the form and the content of a learning object). Evaluation Standards:
1. Quality of Content: The content must be professional, clear and accurate. The use of technology must
be appropriate for the content and the learning object should be provided with academic references and
credits to the creators.
2. Effectiveness as a teaching/learning tool: The learning object should be able to function as a learning
tool in different types of learning environments. The pre-requisite knowledge or skills have to be
identified and the author must provide evidence that the learning object enhances student learning. The
learning object must have clear learning objectives and the target learners must be identified. There
must be clear instructions on how to use the learning model. It must also provide an opportunity for
students to get feedback within or outside the learning object.
3. Ease of use: The learning object must have easy to use navigation tools and user control. The author
must indicate if the learning object can be accessed by learners with various needs. Technical
requirements to run the learning object must be stated.
Protocols: The CLOE peer review process requires the involvement of two kinds of reviewers:
instructional design experts and subject matter experts. The CLOE peer review process involves an
instructional designer and two subject matter experts.
Instructional design experts evaluate the degree to which the LO is likely to meet its instructional goals
as described by the author(s) and defined in the CLOE Guidelines for Authors. Subject matter experts
are responsible for examining the LO for the validity and quality of the content, for factual information
and for the overall contribution of the LO towards student learning. See Appendix 1 & 2 for A model of
the CLOE Peer Review process and the Peer Review evaluation form.
EducaNext: http://www.educanext.org/ubp
About: EducaNext is a service supporting the creation and sharing of knowledge for Higher Education.
It is open to any member of the academic or research community.
Considerations: EducaNext co-ordinates peer review of online content on two levels – formal and
informal. The formal peer review is conducted in a classic way by inviting subject matter experts to
assess learning resources in terms of three major aspects: content, usability and instructional design.
The informal peer review is conducted with the users of learning resources, be they faculty members or
students. They are required to describe their perceptions and experiences of a learning resource right
after they have used it. Their descriptions are based on a tripartite inquiry on the technical, pedagogical,
and organisational aspects.
Protocols: Large-scale resource entities like complete online courses undergo a more comprehensive
review process. A team of evaluators with different backgrounds, including experienced instructional
designers, media designers, domain experts, programmers, and end users, are invited to take part in a
forum to analyse and discuss the strengths and weaknesses of online courses.
EducaNext Catalogue Managers are responsible for screening the quality of metadata descriptions of
learning resources. In their role of catalogue administrators they are able to remove inappropriate
descriptions.
The Health Education Assets Library (HEAL): http://www.healcentral.org/index.jsp
109
About: HEAL is a free digital resource repository designed to meet the needs of health sciences
educators. HEAL promotes the preservation and exchange of useful educational assets while respecting
ownership and privacy.
Considerations: All items submitted by authors to HEAL undergo the peer review process
HEAL employs a rigorous review process, which follows the traditional scholarly model of peer review.
Two unbiased reviewers who are content experts and/or experienced users of instructional technology
systematically examine the quality and usefulness of the submitted resource use a standardised
instrument to rate each item in terms of a set of review criteria, including:
•
•
•
•
Preparation and content quality
Effectiveness of the material
Presentation and ease of use
Significance
Protocols: Following standard academic practice, resources are either rejected, accepted conditionally,
or accepted as-is. Reviewers can also accept outstanding resources “with acclamation.” In addition,
reviewers are specifically asked to rate both the quantity and quality of descriptive information (i.e.,
“metadata”) that authors provide.
Reviewers are faculty at accredited institutions of higher education that have expertise in one of the
basic or clinical sciences. Notably, they are “peer users” of instructional technology (not necessarily
“peer developers”) and have demonstrated excellence in teaching. The resources are not blinded for
review and reviewer comments are made available to the author after the review.
Upon acceptance, resources are permanently published in the HEAL Reviewed Collection. Authors
receive a certificate of publication describing the resource, the review process, and date of acceptance.
Resources can be submitted for peer review using an online Submissions Form (login for this is
required).
Intute: http://www.intute.ac.uk/policy.html
About: Intute is a free online service providing access to the very best web resources for education and
research. All material is evaluated and selected by a network of subject specialists to create the Intute
database
Considerations: Intute aims to inform both content users and content suppliers about the nature,
extent, and accessibility of the collections.
The Intute Collections and Cataloguing Management (CCM) Working Group consist of staff from across
the Intute service with experience in collections management and/or cataloguing.
A central and over-arching collections policy for the Intute service provides each Intute Subject Group
with a basis for their subject-specific collection policy.
Intute employs a network of subject specialists to find, evaluate and catalogue the best of the Web.
The Intute collection is selective and only online resources that meet the agreed quality selection criteria
are included. The selection criteria are outlined in Appendix 6
JIME – Journal of Interactive Media Education http://jime.open.ac.uk/index.html
About: JIME is a publication that does not have a chronological concept of "issue". Instead, JIME
publishes articles for open peer review as they are received. Final versions of articles are published as
soon as they complete the Review Process
Considerations: In JIME's review environment:
1.
2.
3.
Authors have the right of reply.
Reviewers are named, accountable for their comments, and their contribution acknowledged.
The wider research community has the chance to shape a submission before publication.
Submissions will be critiqued by many more reviewers than is currently possible in the conventional
review process, and this will take place in a public rather than private forum. While JIME prefers all
comments to be signed, anonymous contributions are also permitted. Willingness of both authors and
reviewers to engage in this process underpins its success.
110
Protocols The review process for JIME submissions provided in Appendix 5. This review model has
three stages: preprint under private, open peer review, preprint under public, open peer review, and
publication. These are explained below.
Private open peer review...
Articles submitted to JIME are first reviewed by three reviewers who are named, and acknowledged for
their contribution to a review. They post their reviews as threaded comments to a private site. Reviewers
have the option of posting anonymously, but usually reviewers are happy to be named, and in JIME's
conversational review model, it helps to know to whom you are talking, and hence, how better to
interpret comments. Authors are encouraged to respond to these comments and reviewers in turn (who
may not necessarily agree with each other). This takes place during an agreed period when authors and
reviewers are able to respond.
Publication as a preprint for public, open peer review...
On the basis of the discussion, if the editor assigned to the submission judges it to be of sufficient
quality -- that is, broadly acceptable, pending changes based on the review discussion -- the submission
will then be published as a preprint for public open peer review, and announced to relevant communities
to invite their participation. The author-reviewer discussion provides the 'seed' for this second phase of
online review debate. This phase of open review will be closed after one month.
The editor will post to the discussion an editorial report summarising the most significant issues, and
specifying change requirements to the authors.
Following publication...
In conventional journals, the point of publication is the beginning of scholarly debate. JIME brings this
point forward by making submitted preprints accessible, and continues to support discussion about the
revised, published article. In addition, the most interesting review comments/exchanges are published
with the final version, providing readers with insight into the issues that arose during review, and
enabling them to build on those discussions.
Thus, authors can post links to publications to point to subsequent work. Readers can post comments
and links to point to work which has not been referenced, or did not exist when the article was written.
Authors, reviewers and anyone else who has subscribed to the article will receive email alerts to new
postings to its discussion forum.
The final publication will be freely accessible on the JIME site.
Jorum: http://www.jorum.ac.uk
About: Jorum is a JISC-funded collaborative venture in UK Higher and Further Education to collect and
share learning and teaching materials, allowing their reuse and repurposing, and standing as a national
statement of the importance of creating interoperable, sustainable materials. It is run jointly between the
EDINA and MIMAS national data centres.
Protocols: Metadata: Contributors are required to complete a few basic metadata fields available in the
system. The software system in addition creates some metadata automatically. To provide full quality
assurance of metadata, a team of information specialists from the JISC Intute service complete the
remainder of the fields for all objects deposited, apart from learning assets such as images, where
completion of full educational metadata would be difficult.
Technical: A Collection Procedures policy provides the details of certain checks that contributors are
required to undertake. The Jorum team reserves the right to withdraw materials from the repository after
publication for discussion with contributors, if they receive alerts or queries from other users unable to
play their materials and technical checks reveal problems with the materials. The Jorum team check
around 10% of materials as they are added to the repository and Intute cataloguers check that all
materials catalogued by them run correctly when they add metadata.
In addition, project or departmental teams can request that another stage in the Jorum workflow be
added for their group, to enable a pedagogical review stage within their own group.
The LAMS Community: http://www.lamscommunity.org
About: The LAMS Community provides an environment for using and sharing digital lessons under
open content licenses referred to as sequences.
Considerations: There is an informal peer review process in place, a rating feature for each sequence
– including the number of times the sequence has been downloaded, and comments to the author from
users
111
MERLOT: http://www.merlot.org
About: MERLOT is a user-centred, searchable collection of peer reviewed, higher education, online
learning materials created by registered members, and a set of faculty development support services.
Considerations: Faculty selected in accordance with MERLOT guidelines perform the peer review of
MERLOT learning resources. The peer review process is led by Editors and an Editorial Board.
Protocols: The Peer Review Process follows the model of peer review of scholarship, and includes:
1.
−
−
−
−
Developing Evaluation Standards
The MERLOT Editorial Boards provide leadership, tools, and training in developing evaluation
standards and processes.
The MERLOT Administrative Team provides the Editorial Boards with a framework of evaluation
criteria which is based on: Quality of Content; Potential Effectiveness as a Teaching Tool and Ease
of Use
All Peer Reviewers on each Discipline-specific Editorial Boards share and compare their
evaluations following the processes developed and the framework provided to create test cases.
These test cases are then used to develop evaluation guidelines/criteria that are applied to all
materials in the discipline.
Each Editorial Board establishes substantial inter-rater reliability in its evaluations before evaluation
procedures are implemented on the remaining materials in its discipline.
2.
Conducting Evaluations:
−
Editorial Boards decide on the process for selecting materials to be peer reviewed.
If a collection of materials is not large enough to adequately establish inter-rater reliability, its
Editorial Board works to expand the collection sufficiently.
Review teams typically use the following two stage review process:
−
−
Stage 1 Cursory Review to Identify Worthy Candidates:
•
•
•
•
The Editorial Board reviews its collection and "triages" materials as follows:
1. Definitely worth reviewing
2. Possibly worth reviewing
3. Not worth reviewing at this time
Peer Reviewers and Associate Editors report their cursory evaluation to their Editorial Boards.
Peer Reviewers can post Member Comments based on cursory evaluations as appropriate.
The Editorial Board compiles its list of worthy materials. Those deemed "definitely worth reviewing"
receive top priority in the review process.
Stage 2 Intensive Review of Worthy Candidates:
•
•
•
•
The Editorial Board assigns "worthy" materials to Editorial Board Members.
In some cases, the Editorial Board asks the author(s) of the material(s) for permission to review.
Two Peer Reviewers each use their Editorial Board's review procedures, forms, and evaluation
standards as they independently review the material.
Reviewers write peer review reports using the evaluation criteria for MERLOT learning materials as
a guideline and publish these individual reviews in their Editorial Board Workspace.
If there is any significant disparity in the two reviews, an Editor or Associate Editor assigns the
material to a third reviewer.
3.
Reporting Evaluations:
•
−
−
−
−
The Editor or Associate Editor reviews both individual reviews and creates an integrated or
Composite Peer Review Report.
The Editorial Board sends the Composite Peer Review Report to the author(s) for feedback and
permission to post the review on MERLOT.
The Editorial Board posts the Composite Peer Review Report on the MERLOT website.
Authors may ask the Editor to send two letters, to two individuals of their choice, summarising the
peer review process and including the Composite Peer Review Report.
112
Appendix H: Peer review criteria and standards
When a resource is submitted it will undergo an initial functionality review. After passing this
first level of functionality testing, 19 the resource is reviewed on the following criteria:
N.B. Normally a rating of “not at all” on any question by the reviewers will require that the
author provide additional information or revision of the resource before it is accepted.
Resource and Scope 20 :
Reviewer:
Quality of Content
1.
The content of the resource is accurate and up-to-date
2.
The use of technology is appropriate for this content
3.
The content is presented clearly and professionally
(spelling, grammar, etc.)
4.
Appropriate academic references, citations and/or links
are provided
5.
Credits to creators are provided
6.
Multimedia resource quality is good
7.
Documentation provides sufficient information to guide
users in using the resource, including suggestions for
adaptations or extensions
Not at all
Somewhat
Definitely
Effectiveness as a Teaching/Learning Tool
8.
There are clear learning objectives and/or statement of
purpose
9.
The resource meets the stated learning
objectives/purpose
10.
The target learners are clearly identified (academic
level addressed/technical ability/demographics)
11.
There are clear instructions for using the resource
12.
The technology helps learners to engage effectively
with the concept/skill/idea
13.
The resource provides an opportunity for learners to
obtain feedback within or outside the resource
14.
The author provides evidence that the resource
enhances student learning 21
15.
Pre-requisite knowledge/skills, if needed, are identified
16.
The resource stands alone and could be used in other
learning environments
19
Initial functionality testing will be conducted by the CE gatekeeper and will include checking to ensure
that links work, plug-ins are available, platform and browser compatibility are identified, the resource is
useful, adheres to copyright restrictions and adequate metadata has been provided, et cetera.
20
Scope of resource may be suitable for one learning session, several learning sessions, entire length of
program/course, a full program of study, or other as specified
21
Acceptable evidence could be anecdotal comments, student feedback questionnaires, or more formal
learning impact studies.
113
Quality of Content
Not at all
Somewhat
Definitely
Significance
17.
Resource offers an innovation in learning and teaching
18.
Resource contributes to the field
Ease of Use/Usability
19.
The resource is easy to use (i.e. navigation, user
control)
20.
The author indicates whether the resource is
accessible for learners with diverse needs
21.
Technical requirements for the resource are provided
22.
Application loads, launches and executes smoothly
(appears to have no bugs)
23.
Terms and conditions of use are specified
24.
The design of resource (or part of the resource) is
scalable
25.
The costs of implementation in other contexts are
identified
Overall rating of this resource:
π
π
π
π
Does not meet minimum standards – REJECT
Meets minimum standards, but there are concerns – ACCEPT WITH REVISIONS
Meets quality standard – ACCEPT
Excellent overall – ACCEPT WITH ACCLAMATION
Additional Comments:
Strengths of this resource:
Modifications required for acceptance:
Suggestions for future development:
114
Appendix I: Protocols for peer review
(Based on the HEAL peer review policy)
The protocols below detail a review process that follows from the technical check to be
completed by Carrick Exchange staff to confirm the functionality and integrity of the
contribution I.e. links work, plug-ins available and operable, platform compatibility, adheres to
copyright restrictions, adequate metadata provided etc.
1. Objectives of Review Process (Formal and Informal)
•
•
•
•
Assure the quality of CE collection
Systematise the appraisal of resource contributions (including experts)
Provide a review process that is nationally recognised as transparent, fair, valid and
reliable
Foster the recognition of CE as a dynamic, useful and accessible collection.
2. Roles of Chair of review panel, Reviewers, Carrick Exchange Staff
•
Chair of review panel
−
−
−
−
−
−
−
−
•
Reviewers
−
−
−
−
•
Notify panel chair within three days if contribution not in area of expertise, or
present a conflict of interest
22
Read guidelines prior to review activity
Review materials according to guidelines prior to the deadline, using the proforma
Forward review details to panel chair with a recommendation regarding
acceptance or rejection, provide comments and suggested modifications
Carrick Exchange Staff
−
−
−
−
−
−
22
Assume accountability for the peer review process
Make final decisions to accept or reject contributions
Identify three review panel members (subject specialist, educational specialist
and media or collegial specialist)
Convene review panel
Notify panel members when contribution can be located for review
Communicate with contributor/s regarding outcomes of review
Communicate with Carrick Exchange staff
Contribute to periodic evaluation of peer review process
Acknowledge receipt of contribution (this may be automated and CE staff need
only monitor the operations of the system)
Contact contributor if submission appears to infringe copyright or is unacceptable
for other reasons
Contact contributor if additional metadata or further information is required prior to
review
Notify panel chair when contribution is ready for review and identify location for
access
Issue certificates of acknowledgement (this may also be an automated system
requiring only a monitoring role)
Maintain a review database that includes prospective panel chairs and reviewers
Guidelines yet to be developed
115
3. Review Process and Procedures
•
Formal
−
−
−
−
−
−
−
−
•
Panel chairs and reviewers must have the following qualifications… 23
Prospective reviewers sign online up as members of CE and indicate their
qualifications and area of expertise
Carrick Exchange staff oversee the maintenance of the reviewers’ database
Contributors may recommend reviewers
Sets of items submitted for review will be regarded as a set. The review of the set
however is based on the systematic examination of each individual item
Material can be published previously in other outlets, and arrangements with
partner organisations can be made for joint review and acceptance to collections
The copyright of materials remains with the original copyright owner
The Carrick exchange review process, including review criteria, and names of
panel chairs and reviewers are displayed in the Carrick Exchange
Informal
−
−
−
−
−
Informal commentary can be guided by prompts to users such as:
Which category would you like to add your comment (a) friendly praise; (b)
constrictive critique; (c) how did you do that?; (d) how have you used this?; and
(e) how did it work for you?
“Member spotlights” can allow display of a cumulation of informal
acknowledgement
Submission of comment can be acknowledged as “Thank you for submitting your
comments. You are a true contributor to the Carrick Exchange”
Contributor can further reply to the question “how useful was this comment to
you?”
4. Review Criteria and Standards (refer to Appendix H)
23
Qualifications to be determined
116