life, leveling the playing ield for people, companies, and countries worldwide.
Education
for a
Digital World 2.0
Innovations in Education
Volume 1
Edited by
Sandy Hirtz & Kevin Kelly
The irst edition,
learning objects | dialogue | ePortfolios | engag
ODL | gaming | social networking | learning com
activism | praxis | relection | mobiles | wikis
blogs | LMS | media | virtual worlds | co-creatio
collaboration | learning | authentic | assessmen
workshops | Web 2.0
ISBN 0-7726-6488-9
9 780772 664884
7540006117
Education for a
Digital
World
2.0
Innovations in Education
Vol. 1
Published by Open School BC
Acknowledgements
We are unlocking formidable new capabilities that could lead to much more
exciting lives and glorious civilizations. – James Martin
Education for a Digital World 2.0: Innovations in Education, embodies an interactive and
collaborative model of writing and publishing that demonstrates the powerful opportunities
aforded by online technologies, innovative thinking and cooperation around the world.
I would like to express my gratitude to the 98 people from 13 countries (listed in the contributor
section), who collaborated on this project, provided encouragement, and assisted in the peer review
and copyediting process.
I am grateful for the friendship and support of Kevin Kelly who tirelessly helped me lead this
endeavor. A special thank you to Paul Beaufait and Rick Lavin for maintaining the wiki and
coordinating the ever growing list of contributors, and to June Kaminski for the cover art.
Sincerest appreciation and thanks to the Ministry of Education and Open School BC for inancial
and production support, and to Dennis Evans for maintaining enthusiasm as the book expanded,
burst its seams, and metamorphosed into two volumes.
Sandy Hirtz, Editor
2
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
If we teach today as we taught yesterday, we rob our children of tomorrow. –
John Dewey
he need to know the capital of Florida died when my phone learned the
answer. Rather, the students of tomorrow need to be able to think creatively;
they will need to learn on their own, adapt to new challenges and innovate onthe-ly. – Anthony Chivetta, high school student in Missouri
hese two quotes, separated by almost 100 years, speak to the importance and timeliness of
Education for a Digital World 2.0: Innovations in Education. he world around us is changing,
and what better way to prepare educators than to ask other educators to share their knowledge
and experiences? I was convinced of this as an author and peer reviewer of the irst edition, and
am grateful to Sandy Hirtz for inviting me to join her as an editor for this second edition. As
mentioned above, this entire process—this book—was truly collaborative, evidenced even by
the wiki URL: http://collaborativebook.wikispaces.com/. I hope to see many others emulate our
approach to co-creating knowledge.
I add my own thanks to all the participants. We would not have a second edition without the many
authors’ contributions. I also would like to thank the collaboration coordinators, Paul and Rick; the
peer reviewers, experts from around the globe in diferent areas of education and technology; and
the copy editors—all of whom volunteered time to make this efort more rigorous. Finally, thanks
go to Dennis, and everyone at the Ministry of Education and Open School BC for bringing our
work to the world.
Kevin Kelly, Editor
Copyright Permission
he print version of these volumes is © All rights reserved, and the digital version is CC-BY-NCND. Any part may be reproduced without permission but with attribution to the authors. For
derivative work, please write to the editors.
Cover design by June Kaminski
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
3
Education for a Digital World 2.0:
Innovations in Education Vol. 1
Foreword (Volume 1)
Emerging Technologies, Learning Management,
and Developing Content in a Digital World . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Tony Bates
Chapter Abstracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Part One: Emerging Technologies and Practices
Emerging Technologies in eLearning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Patricia Delich, Kevin Kelly, and Don McIntosh
Digital Interaction: Learning and Social Communication in the Information Age . . . . 71
Christopher Boyle and Gerald Wurf
Managing Your Digital Footprint: Ostriches v. Eagles . . . . . . . . . . . . . . . . . . . . 89
Julia Hengstler
Language Learning 2.0 – Innovative Approaches for Tomorrow’s Global Citizens. . . . 141
Sandy Hirtz, Sarah Elaine Eaton, with a contribution from Brittany Espe
Immersive Gameworlds for Worldwide Change. . . . . . . . . . . . . . . . . . . . . . . 167
Natasha Boskic
Media Education in India: A Peer Perspective. . . . . . . . . . . . . . . . . . . . . . . . 191
Ankuran Dutta and Anamika Ray
mLearning: Small Technologies – Massive Contributions . . . . . . . . . . . . . . . . . 213
Brown Onguko, Symon Ngatia and Susan Crichton
Part Two: Implementing and Managing eLearning
he Pearls and Perils of Digital Education . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Pheo Martin
Learning Management Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Don McIntosh with contributions from Kevin Kelly, Patricia Delich, and Randy Labonte
Stakeholder Issues for Adopting a LearningManagement System . . . . . . . . . . . . . 309
Joshua L. Mindel and Kevin Kelly
Towards an implementation of eLearning at a multi-campus university . . . . . . . . . 327
Azra Naseem, Shariq Khoja, Melaine D’Cruze, and Karim Wallani
Case Study of Implementation of a Learning Management System in India . . . . . . . 351
Nilay M. Yajnik
4
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part hree: Developing and Monitoring eLearning Content
Co-Creation of Content to Promote Learning, Activism and Advocacy . . . . . . . . . 363
June Kaminski and Amandah Hoogbruin
he OER Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
Paul G. West and Lorraine Victor
Open Licences for Educators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
Julien Hofman and Paul West
Quality in eLearning: Engaging Learners . . . . . . . . . . . . . . . . . . . . . . . . . . 483
Randy LaBonte
Learning Objects and Personal Learning Environments . . . . . . . . . . . . . . . . . . 511
Valerie Taylor, Richard S. Lavin, and Nellie Deutsch
ODL For Corporate Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537
Anil Prasad P
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
5
Stakeholder Issues for Adopting a Learning
Management System
Joshua L. Mindel and Kevin Kelly
he pace of academe is perhaps best measured by the 25 years it took to get the
overhead projectors out of the bowling alley and into the classroom. - Gilbert
& Green (1997, p. 25)
Introduction
Selecting a learning management system (LMS) for an organization is not an easy process,
especially if that organization has already been using one for several years. LMS vendors
beneit from many institutions’ fears of the pains of change and user backlash when faced
with changing systems. Beyond the iscal cost, an organization must also weigh the labor
costs required of the users and support personnel to make such a change. his decisionmaking process can take several years to complete, depending on the amount of planning
performed and the results of early pilot tests. herefore, this article looks at the decisionmaking process of enterprise-level academic technology adoption rather than simply
documenting the comparison of possible LMS solutions to the legacy system.
As San Francisco State University (SF State) went through its investigation to ind an
alternative LMS solution, it soon became apparent that diferent stakeholders asked
diferent questions relating to the same concepts. To organize these questions during the
investigation itself, the authors put them into a matrix that served as a helpful tool for
conducting a rigorous analysis. As new people joined the process, the matrix provided
evidence that the decision-making process considered their interests. More broadly, the
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
309
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
analysis described in this article is applicable to any organization in need of learning
infrastructure tools or systems, whether that organization is the obvious educational
institution or a training division within a business, government, non-proit, etc.
Ater providing the context in which SF State made its decision, the authors describe
the three stakeholder perception categories, hereater called “lenses,” used to make sure
everyone’s viewpoint was considered. he paper then investigates a number of issues
through each lens as part of the decision-making process. he paper concludes with
implications for future technology adoption decisions.
Institutional Context
San Francisco State University (SF State), a large, urban institution of higher education,
accommodates the learning needs of roughly 30,000 undergraduate and graduate
students. Roughly 80% of these students work half- or full-time, and many have additional
obligations, such as taking care of children or family members, that take time away from
their studies. he student body is also diverse; there is no dominant ethnic group on
campus and there are over 100 languages other than English spoken as the irst language
at home. he campus footprint is expanding to include several additional teaching and
learning locations throughout the San Francisco Bay Area. hese conditions necessitate an
atmosphere of lexible learning at SF State.
Since the mid-1990s, SF State faculty and students have used a learning management
system (LMS) to supplement face-to-face courses with online resources, activities, and
assessment strategies, and to facilitate hybrid courses (that are partly online) and fully
online courses. Ater careful analysis of the best LMS solutions available at the time in
1997, SF State made a transition from an earlier, vendor-based product to the Blackboard
LMS. While that transition constituted a signiicant change, it impacted a smaller portion
of the faculty and students than it would today. he Center for the Enhancement of
Teaching (CET) at SF State put a great deal of energy into the move to Blackboard (which
today is considered the legacy LMS). hese initiatives included both technology adoption
and efective learning workshops. By academic year 2000-2001, almost one-third of
SF State faculty and students used Blackboard, ranging from providing course-related
documents for face-to-face classes to conducting fully online courses.
he campus teaching and learning community faced serious LMS service quality issues during
the 2002 to 2003 period. hese involved a mix of operational and sotware quality issues
that SF State attempted to resolve under the contractual constraints of the California State
University (CSU) system-wide contract with the Blackboard LMS vendor. By December 2003,
the SF State Academic Technology team organized a series of focus groups to assess whether
these issues warranted campus exploration of alternative LMS solutions. Approximately 40
faculty members participated in these meetings, and the response was airmative.
he AT team identiied three options for how to provide LMS services to the university.
he irst option was to host the legacy LMS at the SF State campus, rather than paying the
vendor to host it as an application service provider. he campus did not seriously pursue
this solution, as it would only address issues with service and issues with the product
would still remain. he second option involved investigating other proprietary, vendorbased LMS products. hese vendor-based solutions were generally seen as disadvantageous
310
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
because SF State would remain captive to vendor responses to enhancements,
customizations, and bug ixes. he third option was to look at open source sotware (OSS)
LMS solutions. If the products had suicient feature sets and were supported by a sizable
community of users and developers, then an open source LMS solution could help SF
State overcome the captivity associated with commercial vendor solutions. Ater initial
investigations of equal numbers of commercial and open source products, the academic
technology staf concentrated on the third LMS option, which ultimately led to the Moodle
evaluation described in this article.
At the time (spring 2004), there were approximately 1,000 known Moodle sites and 10,000
registered users on the Moodle.org community website. hese metrics indicated an active
community of support, which relates to the product’s potential long-term viability and
support structure. Exploratory projects, both vendor-driven and open source, posed a
risk of waning interest, but Moodle appeared to have gained the critical mass necessary
to overcome this risk. From an administrative vantage point, a global community of users
(and developers) provides numerous tangible and intangible beneits. For example, at the
time of the decision to investigate Moodle further, the New Zealand Open Source Virtual
Learning Environment project successfully used Moodle with 30,000 learners. Since that
number was equivalent to the size of the entire SF State student body in spring 2004,
the AT group conidently anticipated that Moodle would scale for the SF State campus.
Within the CSU system, Humboldt State was another early investigator of Moodle for their
campus LMS. Both Humboldt State and SF State selected Moodle over Sakai, another OSS
product that held tremendous promise, because it was a more mature product at the time
of the investigation. Problems with the legacy LMS dictated that the campus could not risk
waiting to see if and how the Sakai project developed successfully and on-schedule. he
social constructionist philosophy behind Moodle was also a very attractive aspect from the
instructional technology vantage point shared by many in the AT group.
SF State successfully rolled out its exploratory Moodle LMS to approximately half the faculty
and student body between 2004 and 2006; both Blackboard and Moodle were supported
on the SF State campus during this time of parallel operation (ETAC, 2006). Efective fall
2007, Blackboard was completely phased out and Moodle became the sole LMS serving the
university community. he evaluation that was undertaken over the next several years is
described in this paper, and related to the theoretical literature on technology adoption.
he SF State director of academic technology (Beers, 2010) recently emphasized the
success of the Moodle LMS to the Academic Senate by citing the following student usage
statistics for fall 2009:
• 95% of 30,000 students and 90% of 1,600 instructors used Moodle for one or more
courses, through which those students:
▸ Completed 334,504 quizzes.
▸ Submitted 164,026 assignments.
▸ Posted 114,099 forum comments.
▸ Watched 20,767 streamed lectures captured in CourseStream system.
• 1,500 students also built electronic portfolios.
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
311
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
At times that success has presented challenges for the Academic Technology staf
responsible for maintaining Moodle and supporting its users. For example, during peak
periods such as inals week, there were as many as 16,000 quiz attempts in a twelve-hour
period—in addition to load generated by students uploading written assignments, posting
forum comments, and checking their grades. With each challenge, the SF State team has
moved the Moodle system forward with respect to scalability, usability, or both.
SF State has worked with other campuses within the CSU system to help them pilot, adopt,
implement, or maintain their own Moodle instances. In academic year 2010-2011, SF
State took a leadership role in the newly formed CSU Moodle Common Interest Group, a
consortium of Moodle campuses.
Moodle and Learning Management
here is a rich literature on technology adoption. Rogers (1962) produced the seminal
work, Difusion of Innovations, in which he identiies ive key perceptions that inluence
technology adoption: relative advantage, compatibility, complexity, trialability, and
observability. Numerous other studies (Ostlund, 1974; Tornatzky & Klein, 1982; Davis,
1989; Moore & Benbasat, 1991) have followed suit with reinements. Key studies from
LMS adoption literature (Harasim, Hiltz, Teles, & Turof, 1995; Anderson, Varnhagen, &
Campbell 1998; Wilson & Stacey, 2003) partially overlap with the technology adoption
literature since both sets involve adoption of information systems.
Based on previous experiences with LMS vendor tools and support, SF State determined
that lexibility and extensibility were critical considerations. Flexibility was loosely
deined as the ability to modify and ix the LMS, while extensibility was deined as the
ability to customize the LMS for current use as well as plan for unforeseen future needs.
Open source LMS solutions provide both options. Brandon Hall (2007, para. 3) described
Moodle as “the best-known open-source LMS.” When the academic technology team began
its investigation in 2004, Sakai was the best-known solution. However, early tests with
Sakai showed that it had not yet become feature-parallel to the legacy LMS. Rosen (2006,
para. 8) described Moodle as more lexible than vendor products, as well as Sakai:
So the diference is lexibility versus heaviness. As an Open Source
project Sakai will be able to take advantage of some of the power of
the open source paradigm, but its heavier architecture may mean that
it will always be closely compared to Blackboard and WebCT (while
Moodle moves ahead into fresher waters).
Stakeholder Lenses
Complexity in the mission-critical technology adoption decision at SF State emanates from
the LMS infrastructure attributes and from the broad set of stakeholders afected by the
infrastructure. For example, the LMS infrastructure has capabilities, limitations, risks, and
implications that need to be clearly understood. Diferent stakeholders may view the LMS
infrastructure diferently based on their ailiated roles.
How someone evaluates any given issue or criterion related to the technology adoption
decision partly depends on his or her vantage point in the organization, or the lens through
312
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
which he or she views the decision. As noted above, SF State identiied three primary
lenses through which technology decisions would have to be viewed: the teaching and
learning lens (primarily faculty); the technology management lens (primarily technology
support staf); and the university administration lens (administrative decision-making
staf). hese lenses are used to identify a wide range of concerns rather than to pigeonhole
any particular stakeholder into a single viewpoint. On the contrary, some stakeholders
will use multiple lenses to fulill their multiple responsibilities. For example, university
administrators and academic technology managers frequently come through the ranks of
faculty prior to assuming their university leadership or management responsibilities.
• he teaching and learning lens addresses the core purpose of the learning
management system—to help instructors and students improve learning. he
LMS inluences how learning activities are created, how the instructor facilitates
interactions, how the instructor provides learning resources, and inally how the
instructor assesses learning.
• he technology management lens relects the operational support required
to maintain the learning management system, such as providing training and
technical support for students and faculty, making the LMS universally accessible,
administering sotware components, evaluating and potentially modifying LMS
functionality, and inally interacting with the LMS vendor or community.
• he university administration lens relects the executive level view of the learning
management system within the university. his view includes strategic concerns
such as teaching and learning (overlapping with teaching and learning lens), policy
development, impact of LMS on matriculation, risk exposures, costs (e.g., total cost
of ownership, return on investment), and other high-level considerations.
hough these three lenses were developed and used independently by academic technology
practitioners at SF State, they are consistent with the literature on LMS adoption
decisions. Coates et al. (2005), for example, state that LMS adoption decisions deal with
interrelated educational, technological, and administrative issues that clearly map to the
three lenses identiied in this paper. Landon, Henderson, and Poulin (2006) asked MIT’s
peer institutions to answer questions about choosing an LMS. hey put the reported key
drivers into three categories—systems administration drivers, organizational drivers, and
pedagogical drivers—that mirror SF State’s lenses.
Evaluating an LMS through the Lenses
Once the Moodle evaluation gained momentum and included external activities, such as
the CSU Summit on Open Source LMS solutions in November 2005 and early discussions
of the research documented in this paper, the list of evaluation issues and three lenses
were made explicit for discussion purposes. his ex ante or in-progress consideration of
teaching and learning issues is apparently somewhat unusual in light of the predominantly
ex poste consideration that Coates et al. (2005) argue is given to many LMS adoption
decisions. Having said that, the characterization of issues in this paper beneits from ex
poste relection during the research that led to writing this paper.
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
313
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Issue by Issue
Each evaluation issue is described here, some in groups for conciseness. Given that speciic
issues will frequently look diferent based on the lens through which the stakeholder views
them, we characterize the issues with representative questions that a stakeholder would
ask. We do this to ensure that the analysis will be useful to other institutions seeking
guidance. Issues are not listed in order of importance since that depends on the stakeholder
view. Stakeholder prioritization of these issues is addressed explicitly in “Prioritization of
the Issues” below. Appendix A collects questions that diferent stakeholders might ask with
respect to each issue.
Accessibility
Accessibility has emerged as a key issue for CSU campuses, both in the classroom and for
online teaching and learning. he CSU Chancellor’s Oice issued Executive Order 926 to
express its responsibility related to accessibility. It then began the Accessible Technology
Initiative to help the campuses reach speciic accessibility goals. However, legal concerns
are not the primary motivation for addressing accessibility. Universal design for learning
principles dictate that everyone should have equal access to course materials and activities.
hese principles—“teach every student”—accommodate everyone, including people
who have disabilities, speak English as a second or third language, have limited Internet
connectivity, or have diferent learning preferences.
hrough the teaching and learning lens, as well as the university administration lens,
accessibility ensures that persons with disabilities can use all LMS resources. Speciic
questions that a teaching and learning stakeholder might ask include: Can instructors
and students with disabilities use the LMS to accomplish all teaching and learning tasks?
Do instructors and students have access to the necessary assistive technology tools and
information to make the LMS useful? Questions that a university administrator might ask
include: How can the university prevent lawsuits related to unequal access to LMS? What
types of accessibility information must be disseminated to the instructors and students
with disabilities?
To technology management stakeholders, the issue of accessibility can be interpreted
as one in which the technology is the focus. Representative questions include: Does the
LMS work with all browsers and platforms? Is it feasible to modify the system to improve
accessibility? Do campus labs have stations with the necessary assistive technology tools
and information to make the LMS useful?
During the investigation of alternative LMS solutions, SF State conducted an accessibility
study of Blackboard and Moodle. Staf from Academic Technology and the Disability
Programs and Resource Center chose not to rely solely on the results from a validation tool
such as Bobby or WebXACT. Using a scripted process, they asked students who regularly use
assistive technology, such as Dragon Naturally Speaking for speech recognition or the JAWS
screen reader, to complete common academic tasks within an LMS. he students performed
tasks related to content (e.g., download a ile), interaction (e.g., participate in a forum), and
assessment (e.g., take a quiz) in parallel courses established in both LMS solutions. While the
number of accessibility study participants was not statistically signiicant, the preliminary
results showed that Moodle was easier to use for assistive technology users.
314
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Availability of Support
For instructors using the LMS, the community of interest is the set of instructors using a
similar tool or teaching similar material with whom ideas and materials can potentially be
shared. For students, the community is likely within a given class, college, or the university
campus. So the key question for those viewing the LMS through the teaching and learning
lens is whether the LMS fosters a sense of community amongst peers. Is there anyone
available, either in-house or in an external support community, who can help users learn
the LMS and resolve issues if they arise?
For technology management stakeholders, the in-house questions to address include: Do
we have the staf and resources to provide 24/7 support for the LMS? What contingency
plans must we establish to deal with a server crash if no one is on campus? Can we train
existing staf with this LMS, or will we need to acquire new staf? What outside resources
are available to help when in-house resources require assistance? When considering
external community support: Can the community provide supplemental or backup
technical support for operations as well as modiications or extensions to the LMS?
Community members may be users or third-party service providers, and in the case of
an open source LMS such as Moodle, the community also includes those volunteers that
actively participate in the development and maintenance of the system.
University administrators share much with the teaching and learning and technology
management stakeholders here. What is the appropriate number of staf members to
support the teaching and technology aspects of mission-critical LMS use, based on number
of actual users on campus? Can the university competently support the LMS? Are there
related or similar universities using a comparable LMS with whom we can establish virtual
communities for faculty (if not students) that span campuses? Is there support of any type
(vendor, community, other) to which staf can turn when they need additional resources? Is
there the budget to leverage available support? How will an open source LMS be supported
since there is no single vendor “standing behind” the LMS product?
Worldwide, there were more than 24,000 active Moodle sites in 175 countries (Moodle,
2007b) when the legacy LMS contract was set to expire, so the potential for external
community support was there. By 2011, that number had doubled to over 49,000 sites in
212 countries. Closer to home, San Francisco State engaged with other CSU campuses
and several Northern California community colleges to create the Digital Teaching
and Learning Consortium (DTLC). he DTLC acted as an umbrella organization that
helped coordinate common academic technology projects, starting with collaboration
around Moodle and electronic portfolios. Each stakeholder group derived beneit from
these eforts. For example, when SF State shared its print-based training materials with a
community college, the community college then used them to create podcasts, or audio
versions of the materials, which also beneited SF State faculty. SF State and another CSU
campus also collaborated on grant-funded work on Moodle projects. his local community
provided another layer of support beyond the global Moodle community. In 2010-11,
the CSU system formed a more oicial Moodle consortium, the CSU Moodle Common
Interest Group, in which SF State has played a leadership role.
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
315
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Customizability, Extensibility
hrough the teaching and learning lens, the customizability and extensibility issues can
be understood through the following line of questioning: Is it possible for instructors and
students to suggest changes or new functionalities to help accomplish instructional goals?
To what extent can the LMS be customized to suit the heterogeneous needs of the campus
learning community? For example, can the online content in an engineering course be
organized diferently than the writing portfolios in an English writing course? Can the
LMS be linked to other teaching and learning tools, such as the online library system?
For technology management stakeholders, these are representative questions: Can
programmers or system administrators make appropriate changes to sotware to meet
educational needs expressed by instructors and students as requirements evolve? Will
customizations of open source create a branch that breaks too far away from the core,
isolating the university from external support?
University administrators may have a more strategic and risk-oriented view of LMS
customizability and extensibility. For example, in anticipation that campus needs will
evolve over time, how much lexibility and freedom does the campus have in enhancing the
LMS to serve unanticipated needs?
In terms of risk assessment, there are upsides and downsides to local modiications to the
LMS. Beneits include being able to tailor the LMS to local requirements, plus there are
likely motivational factors that will positively inluence staf retention. However, there are
potential downsides to this capability too. Downsides include the potential inability to
meet unreasonable expectations and the risk of customizing the local LMS so much that
the university can no longer use future updates to the system.
Quality, Reliability, Scalability, Security, and Stability
For teaching and learning stakeholders, is the LMS reliable so that instructors can be
assured that students will have access to course materials at all times? Can large classes be
conducted in the LMS as easily as small classes? Are there features that ease block uploading
or downloading of information? Are there adequate information security controls in place
to ensure that suitable privacy is maintained for student data as well as for instructor data?
From the perspective of teaching and learning, quality is addressed by ease of use (which is
treated in the next subsection) and reliability. Stability is also addressed by reliability.
From the perspective of technology management, will the LMS be up and available 99.9%
(or whatever availability measure is targeted)? If the system is hosted on campus, how oten
should the academic technology team require maintenance windows? What are the most
stable underlying platform and operating systems? Can the LMS technology and support
organization scale to serve the needs of a campus of our size, and can it expand if we grow?
Does the LMS design require load balancing to handle high demand? Do certain LMS
features or functions fail with high volume use? At what point should the application, data,
and database layers be given their own server(s)? How can the academic technology team
make the system scalable if adding more hardware is not an available option? What are the
symptoms of inability to scale? (e.g., Frequent system crashes? Slow system performance?
Inadequate response to instructor or student users? Resource issues?) What security
questions do the team need to ask the vendor if it is hosting the LMS? How susceptible
316
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
are the LMS and supporting infrastructure systems to security breaches? Is there adequate
support to rapidly report and address security vulnerabilities?
For university administrators: Does the product relect the high standards of our
organization? Will instructors and students feel empowered and pleased to use the LMS?
Can the LMS technology and support organization scale to serve the needs of a campus
of our size? Will the LMS sotware require additional costs for hardware to make it truly
scalable? What eforts need to be coordinated among various centralized and decentralized
technology staf (e.g., lab managers, college technology staf) to promote good security
practices across the campus? Will the LMS help us comply with all suitable privacy
and safeguard regulations? If the LMS does not meet stability expectations, how will
administration address vocalized concerns?
In 2005 and 2006, SF State undertook scalability tests for the Moodle LMS that
demonstrated successful operation with almost 10,000 students, each of whom were
enrolled in at least one course. A list of other institutions that are successfully running
large Moodle installations is available at the Moodle website (Moodle, 2007a). SF State
migrated completely to Moodle as of fall 2007. During the fall 2007 term, Nearly twothirds of the 1600 total faculty (~66.50%) and more than three-quarters of 30,000 total
students (78.33%) used it for one or more classes. Overall, this equated to use in 1,525 out
of 11,000 total class sections (13.85%).
Long-Term Viability
For teachers and learners: Will we need to learn yet another LMS again in the near term, or
will this system stick around for a long enough period to warrant time investment?
For the technology management team: Will we need to initiate yet another LMS
investigation process again because this product does not have market sustainability?
Long-term viability is very important for minimizing disruption to campus teaching and
learning, and avoiding wasting valuable academic technology resources on evaluating and
deploying a new system.
For university administrators: Will the campus soon have to go through the investigation
process again because the LMS product may disappear? Long-term viability helps in
academic technology infrastructure planning and budgeting. It also aids in keeping
academic technology staf motivated, since they are maintaining and building relevant
market skills.
Blackboard announced its intention to acquire and merge with its competitor, WebCT,
on October 12, 2005—the same day the CSU system was facilitating an LMS summit in
Southern California to assist campuses considering the need to change products. he
merger/acquisition demonstrated that even the second largest LMS provider could not
maintain long-term viability in the market. Since that time, Blackboard has also acquired
Angel, another leading LMS vendor, and the two largest Web conferencing solutions
(Elluminate and Wimba) as well. Mature open source products like Moodle and Sakai
may actually have greater long-term viability than several commercial products, as their
communities or segments therein could maintain the code independently regardless of
potential acquisition-like activity.
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
317
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Functionality
For teachers and learners: Do the LMS features allow instructors and students to perform
tasks as they would like, or must instructors and students modify tasks to it system
limitations? To what extent does the LMS provide a set of tools that enables instructors to
establish a vibrant and efective online community outside the classroom? In addition to the
rudimentary ile upload and download capabilities, are there advanced community features
such as forums, wikis, instant messaging, RSS feeds, and gradebooks? Is there consistency?
For example, does restricting access to a selected resource restrict access in all situations?
For technology managers: How easy is it to develop and implement faculty and student
training to use the technical features of the LMS? To what extent do the features meet or
exceed requirements expressed by instructors? If not, then is there suicient capability to
add on features to provide such functionality? To what extent is there LMS support to add
functionality as requirements evolve?
For university administrators: Can any of the features be used to provide institution-level
reporting for accreditation or for generating reports useful to administrators of the LMS
and executive management in the university?
At SF State, Academic Technology regularly conducts interviews with faculty users, as well
as faculty who have stopped using the LMS, to determine which features meet or do not
meet teaching and learning needs. It can
Feasibility of Integration
his issue has to do with how well the LMS will integrate with a university’s other online
information services. For teachers and learners: Are students automatically registered
in LMS course shells when they register for a course through the oicial university
registration system, or must they separately register for an LMS course? Are course shells
automatically created for professors, or must instructors request them? Are there linkages
between the LMS content that professors post and the library’s electronic databases?
For technology managers: What level of efort is required to integrate the LMS with student
management databases, campus authentication processes, or other academic technology
solutions? How easy is it to complete each of these integrations? What level of efort is
required to implement university policies (e.g., if a student drops a course)? Does the LMS
leverage current and standards-based technology to ease integration today and in the future?
For university administrators: Can the LMS be integrated with student management
databases, campus authentication processes, or other academic technology solutions? If
so, can the campus perform these integrations in-house or does it need to hire outside
consultants? Can information from the LMS be made available and other reporting tools used
to administer the university? Can LMS integration be used to implement university policies?
Early in the adoption process, SF State integrated the Moodle LMS with the Student
Information Management System (SIMS) database to replicate the daily updates that were
being done with the legacy LMS. his kept the Moodle student enrollment synchronized
with university class enrollments throughout the semester as students added and dropped
classes. Prior to this integration, students needed to explicitly enroll in each Moodle course
shell associated with an SF State class. While this integration is now considered essential,
318
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
it was not as common in 2005 and 2006. Additional integration steps have been completed
since then, such as linking Moodle with the College of Behavioral and Social Sciences’
Digital Information Virtual Archive (http://diva.sfsu.edu); a commercial lecture capture
solution (Echo360); a plagiarism detection service (Turnitin); .and a library services feature
that allows faculty to select journals and books for students to consume as part of a class.
Ease of Use
For teachers and learners: Is the LMS environment intuitive enough for instructors and
students to complete tasks with little or no training?
For technology managers: Is the LMS written in a common language for which it will be
easy to ind qualiied programmers? Is the LMS easy to install and maintain? Is there a
graphic user interface (GUI) for administrators? What resources need to be allocated to
training users?
For university administrators: How accessible is the aggregated data that is collected? Is
it easy to extract and summarize logs (e.g., usage levels) and other data for return-oninvestment analysis? What is the cost of training users and technology staf?
Open Source or Proprietary?
Strictly speaking, this is more of an issue for the technology management and university
administrators than it is for the teaching and learning stakeholders since the university
administration lens will capture the users’ view. Users primarily focus on the information
services availed by the LMS rather than on the underlying technology used to provide that
information service.
In practice, though, some faculty may pay attention to whether the underlying product
is open source or proprietary. his may be because the open source LMS provides for the
possibility of technical faculty suggesting modules or features to add. Another reason is
that the open source vs. proprietary debate sometimes becomes an ideological one. A third
reason is that faculty may recognize the relatively low procurement and licensing fees for
the LMS sotware and seek reallocation of those funds to teaching enhancement.
For technology managers: Can the open source LMS be used “right out of the box” (i.e.,
without much modiication)? Does the current technical team have the skill sets to support
the system? If the university is currently paying the vendor to host the legacy LMS, is it
possible to pay a diferent group to host the open source alternative? How can the campus
best take advantage of the control gained by using open source?
For university administrators: What are the beneits and hidden costs associated with any
open source product? Recognizing that the cost structure of procuring and maintaining an
open source product is diferent than that of a proprietary product, how should the campus
reallocate the money?
Prioritization of the Issues
he previous section described the evaluation issues, but neglected any prioritization of
them; this was intentional, since each set of stakeholders would likely prioritize them
diferently. In this section, we clarify the bias of each lens and then prioritize the issues
based on this bias. It is up to the university to seek the appropriate balance between the
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
319
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
three lenses when making the ultimate decision to adopt an LMS. To highlight the bias, we
raise two questions for each stakeholder:
1. What are the beneits of viewing the LMS adoption decision predominantly from the
perspective of your own lens?
2. What pitfalls may arise if concerns of either or both of the other two stakeholders
dominate the decision?
Teaching and Learning
Teachers and learners will say that their lens should take priority because the university
exists to serve teaching and learning, and technology management and university
administration are overhead that exist to serve teaching and learning. herefore, basing
the decision predominantly on teaching and learning will ensure that the LMS furthers the
core mission of the university.
If the other perspectives are allowed to dominate, the technology management
stakeholders may focus too heavily on interesting technology (i.e., bells and whistles) or
technology that is easy to support, without meeting teaching and learning needs. And
university administration may focus too much on costs or vendor relationships.
Prioritizing the evaluation issues into three levels:
• Key issues for teachers and learners will be accessibility, ease of use, functionality,
quality, reliability, scalability, security, stability, and support.
• Less important but necessary issues to be considered are availability of support
(community or otherwise), customizability, extensibility, integration with other
online information services at the university, and long-term viability.
• Whether the product is open source or proprietary will be the least important
consideration for teachers and learners.
Technology Management
To the irst question above, technology management stakeholders will respond that the
LMS is mission-critical infrastructure that needs to be managed properly to support
educational needs. Paying attention to technology management needs will ensure that this
service can be provided reliably.
Technology management staf have the concern that a predominantly teaching and
learning focus may not address what is technologically feasible or what its within
university guidelines. Technology management staf may also have the concern that the
university administration lens could focus too heavily on short-term cost and not enough
on total cost of ownership or impact on the tech management team.
Prioritizing the evaluation issues into three levels:
• Technology management staf ’s key issues are accessibility, ease of use, availability
of support, customizability, extensibility, integration with other online information
services at the university, functionality, quality, reliability, scalability, security,
stability, support.
320
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
• It will be less important but still necessary to consider the long-term viability of the
product and whether it is open source or proprietary.
• No issues are irrelevant to the technology managers.
University Administration
University administration will likely conclude that serving the teaching and learning
community is key, but that cost and strategic direction of the university are also important.
However, administration may also have the concern that a strictly teaching and learning
focus may not relect technology management requirements or university cost constraints.
And from an administration standpoint, the technology management lens is important
but not all-inclusive, as it may not take into account university strategic plans or
operational requirements. Technology managers may pull the campus in directions it is
not prepared to travel.
Prioritizing the evaluation issues into three levels:
• University administration and technology management staf will share many key
issues including accessibility, ease of use, customizability, extensibility, integration
with other online information services at the university, functionality, long-term
viability, quality, reliability, scalability, security, stability, support.
• he university administration will also need to consider whether or not support is
available and whether the product is open source or proprietary.
• No issues are irrelevant for university administration.
Evaluation Against the Rogers Dimensions
We previously characterized each of the SF State evaluation issues through three separate
lenses: teaching and learning, technology management, and university administration.
With that in mind, we now demonstrate that these are consistent with – but more detailed
and speciic to an LMS – the ive technology dimensions proposed in the seminal work
(Rogers, 1962): relative advantage, compatibility, complexity, trialability, and observability.
We do this to clarify that the SF State issues are useful in practice and consistent with prior
theoretical work.
Relative Advantage refers to the perceived beneits of a technology over earlier technologies
or ideas. Many of the SF State evaluation issues address aspects of relative advantage:
accessibility, community support, customizability, ease of use, extensibility, functionality,
feasibility of integrating LMS with other online information services at the university,
quality, reliability, scalability, security, and support.
Compatibility refers to consistency with practices and norms. Accessibility is an aspect of
compatibility in that SF State must serve people with disabilities. Community support is
an aspect of compatibility because SF State has a mission that includes interactions with
the SF Bay area. One could argue that many of the other SF State evaluation criteria (e.g.,
ease of use, quality) support compatibility, but the reasoning behind that is captured by the
relative advantage construct.
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
321
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Complexity refers to ease of use or learning a new technology. Ease of use and support
required obviously address this, but so do customizability and extensibility since they can
be aided or hindered by complexity. Another grouping of issues, centered around quality
(quality, reliability, scalability, security, stability), also relates to complexity in that more
complex systems will hinder technology support staf ’s ability to achieve high levels of
these criteria.
Trialability refers to the having the opportunity to experiment with a technology before
committing to it. his is indirectly addressed by the open source vs. proprietary product
discussion, since the former typically entails a lower upfront sotware cost. When a demo
version of a vendor-centric LMS is available, that partially addresses trialability. But it is
very unlikely that the extent of trialability of a vendor-centric LMS will ever reach the level
of an open source system. SF State’s operational evaluation of an LMS over a three-year
period was only possible because it was open source.
Observability refers to the extent to which features and beneits are visible. Over a very
short period of time, ease of use and functionality address this. Over an extended period
of time, most of the SF State evaluation issues address observability. For example, the
extent to which an LMS is accessible, customizable, or scalable can be observed by the
stakeholders over a period of time.
hough the speciic comparative analysis of Moodle vs. Blackboard is beyond the scope of
this paper, one such analysis has been undertaken through the teaching and learning lens.
he Educational Technology Advisory Committee (ETAC) at SF State serves as the liaison
between faculty and other stakeholders on campus. his committee produced a report
(ETAC, 2006) that included the application of Rogers’ ive attributes to the two systems.
Cultural Changes in Teaching and Learning
Not all cultural shits on a university campus are positive, of course. Coates et al. (2005)
argued that one of the more persuasive reasons for LMS adoptions in general is to provide
university administrators with “a means of regulating and packaging pedagogical activities
by ofering templates that assure order and neatness, and facilitate the control of quality.”
Further, Smart and Meyer (2005) described challenges associated with changing from one
LMS to another. If an organization has acculturated itself around one system, the users will
not want to lose all their work or any familiar functionality in moving to another.
he academic technology team at SF State used the aforementioned focus groups to
determine what functionality faculty would not want to lose when moving to another
LMS. Once the campus started Moodle pilot tests, the same team worked with a computer
science graduate student to create a conversion utility so instructors would not lose the
work they had done in the legacy system. As it turned out, at least half of the faculty
members chose to start over from scratch, rather than convert their legacy LMS course
to Moodle. To capitalize on this period of change, the academic technology team asked
faculty to think about changing the way they used the LMS to supplement their teaching.
Namely, they asked faculty to take this mentality a step further by going beyond using the
LMS solely for document delivery. Many faculty members began including interaction or
assessment activities to supplement their face-to-face instruction.
322
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
During the transition from Moodle pilot tests to scalability tests and exclusive use of
Moodle as the campus LMS, the number of transition faculty was closely matched by
faculty who had never used an LMS at all. Some had heard from their peers that Moodle,
a learning-centered environment, was easier to use than the tool-centered legacy system.
Since there were fewer barriers to entry than the legacy LMS, these novices to online
coursework wanted to add new elements to their teaching. O’Quinn and Corry (2002)
identiied concern about workload as the second highest factor keeping traditional
classroom faculty from engaging in online work. With reports that Moodle took one-third
the time to set up, the threshold had become low enough for these novice users to enter the
world of online teaching.
Surprisingly, one of the most frequently used activities by both veteran and novice
instructors did not even exist in the legacy LMS. Wikis had become a frequently utilized
Moodle activity, ater discussion forums, assignments, and quizzes. Instructors were able
to facilitate several variations of group work, such as the entire class generating a literature
review, or small teams posting their project summary (Mindel & Verma, 2006). One novice
faculty member reported wanting to use wikis again, even ater having diiculties related
to (a) lost formatting when copying and pasting from Microsot Word and (b) having MLA
citations become hyperlinks due to the use of square brackets, a wiki convention.
Conclusion and Implications
Organizations that are in the middle of, or just beginning, the decision-making process
about adopting a new or diferent learning management system will ind using the
stakeholder lenses to be helpful. hrough this approach, San Francisco State University
was able to identify the issues that required further research, to address previously unmet
needs, and to generate greater buy-in among the interested parties. With everyone on
board from the beginning, the transition between LMS solutions was very successful. To
maintain this level of success, the campus continued to engage people from the three lens
categories as new decisions were made about upgrades, feature additions, and integrations
with other applications. Further, the campus can use the same process if and when it must
make other adoption decisions for other technologies, such as electronic portfolios, online
meeting spaces, or online course evaluations. To do so may require adding or subtracting a
couple categories from the original matrix, but by and large the process will remain intact.
Since every institution is diferent, the authors recommend using the matrix of questions
as a starting point. Organizations should ask stakeholder groups to modify the questions
regarding each issue and to add issues, if necessary. Answering the questions themselves
can be accomplished in diferent ways. he project leaders can facilitate a face-toface process, such as a series of large meetings or focus groups. hey can set up an
asynchronous activity, such as a wiki, discussion forum, or online qualitative survey. For
best results, organizations may want to use some combination of the two.
Acknowledgement
he authors would like to thank groups involved in the multi-year LMS evaluation at SF
State—Academic Technology (formerly Center for the Enhancement of Teaching) for
performing the work related to establishing the pilots and following studies, hosting and
maintaining the Moodle LMS, supporting the faculty and student users, and conducting
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
323
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
the campus evaluation; the faculty-led Educational Technology Advisory Committee for
reviewing the evaluation results at regular intervals and providing valuable feedback; the
Disability Programs and Resource Center for assistance with accessibility study; the early
adopter faculty members who helped move the project forward through word-of-mouth
recommendations; the Instructional Technologies Department and a Usability Studies student
from UC Berkeley for assistance with the heuristic evaluation; and the administrators who
supported the LMS investigation when things worked and when they did not.
Authors
Joshua L. Mindel, Ph.D., Assistant Professor of Information Systems, College of Business,
San Francisco State University, 1600 Holloway Avenue, San Francisco, CA 94132. Email:
jmindel@alumni.cmu.edu
Kevin Kelly, Ed.D., Manager, Online Teaching and Learning/Media Distribution and
Support—Academic Technology, San Francisco State University, 1600 Holloway Avenue,
San Francisco, CA 94132. Email: kkelly@sfsu.edu
324
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
References
Anderson, T., Varnhagen, S., and Campbell, K. (1998). Faculty adoption of teaching
and learning technologies: Contrasting earlier adopters and mainstream faculty. he
Canadian Journal of Higher Education, 2-3, 71-98.
Beers, M. (2010, February). he role of academic technology in assisting the online course
evaluation policy and the online teaching policy. Presentation given to the San Francisco
State University Academic Senate, San Francisco, CA.
Black, E. W., Beck, D., Dawson, K., Jinks, S., & DiPietro, M. (2007, Mar/Apr). he other
side of the LMS: Considering implementation and use in the adoption of an LMS in
online and blended learning environments. TechTrends, 51(2), 35-39, 53.
Coates, H., James, R., and Baldwin, G. (2005). A critical examination of the efects of
learning management systems on university teaching and learning. Tertiary Education
and Management, 11(1), 19-36.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use and user acceptance of
information technology. MIS Quarterly, 13(3), 319-339.
ETAC. (2006). Viability of an Exclusive iLearn-based LMS Environment. Educational
Technology Advisory Committee, San Francisco State University, retrieved on 13 April
2007 http://www.sfsu.edu/~etac/ETAC_Report-iLearn-Bb-inal.pdf.
Fisher, S. (2006). Open technologies and resources for the humanities - and cooperative
consequences. Arts and Humanities in Higher Education, 5, 127-145.
Gilbert, S. W., Green, K. C. (1997). Moving information technology into the classroom. In
Linda Enghagen (Ed.), Technology and higher education (pp. 25-56). Washington, DC:
National Education Association.
Hall, B. (2007, March). Five open-source learning management systems. Chief Learning
Oicer. Retrieved from http://www.clomedia.com/content/templates/clo_article.
asp?articleid=1714&zoneid=190
Harasim, L., Hiltz, S. R., Teles, L., & Turof, M. (1995). Learning networks: A ield guide to
teaching and learning on-line. Cambridge, MA: he MIT Press.
Landon, B., Henderson, T., & Poulin, R. (2006, July 19). Massachusetts Institute of
Technology & Edutools: Peer Comparison of Course/Learning Management Systems,
Course Materials Life Cycle, and Related Costs. Boulder, CO: Western Cooperative for
Educational Telecommunications (WCET).
Mindel, J. L., & Verma, S. (2006). Wikis for teaching and learning. Communications of the
Association for Information Systems, 18, 1-23.
Moodle. (2007a). Moodle Documentation: Installations 10000 plus. Retrieved April 13,
2007, from Moodle.org: http://docs.moodle.org/en/Installations_10000_plus
EDUCATION FOR A DIGITAL WORLD 2.0 VOL.1
325
Part Two: Implementing and Managing eLearning
Stakeholder Issues for Adopting a LearningManagement System
Moodle. (2007b). Moodle Sites. Retrieved April 13, 2007, from Moodle.org, http://moodle.
org/sites/
Moore, G. C. & Benbasat, I. (1991). Development of an instrument to measure the
perceived characteristics of adopting an information technology innovation.
Information Systems Research, 2(3), 192-222.
O’Quinn, L. & Corry, M. (2002). Factors that deter faculty from participating in distance
education. Online Journal of Distance Learning Administration, 5(4). Retrieved from
http://www.westga.edu/%7Edistance/ojdla/winter54/Quinn54.htm
Ostlund, L. E. (1974). Perceived innovation attributes as predictors of innovativeness.
Journal of Consumer Research, 1(2), 23-29.
Rogers, E. (1962). Difusion of Innovations. New York, NY: Free Press.
Rosen, Z. (2006, Jan). Comparison of Sakai vs. Moodle. Educause CIO Archives. Retrieved
04 May 2007 from http://listserv.educause.edu/cgi-bin/wa.exe?A2=ind0601&L=cio&T=
0&F=&S=&P=20617
Smart, K. A. & Meyer, K. A. (2005). Changing learning management systems: Lessons
learned. Educause Quarterly, 28(2), 68-70.
Tornatzky, L. G. & Klein, K. J. (1982). Innovation characteristics and innovation adoptionimplementation: A meta-analysis of indings. IEEE Transactions on Engineering
Management, 29(1), 28-46.
Wilson, G. & Stacey, E. (2003). Online interaction impacts on learning: Teaching the
teachers to teach online. In G.Crisp, D.hiele, I.Scholten, S.Barker & J.Baron (Eds),
Interact, integrate, impact: Proceedings of the 20th annual conference of the australasian
society for computers in learning in tertiary education (pp. 541-551). Adelaide, 7-10
December 2003.
326
EDUCATION FOR A DIGITAL WORLD 2.0 VOL. 1