Systemic Practice and Action Research, Vol. 17, No.

5, October 2004 (
C 2004)

DOI: 10.1007/s11213-004-5789-7

Systemic Evaluation Methodology: The Emergence of

Social Learning From Environmental ICT Prototypes
Sandrine Simon1
Received February 3, 2004; accepted June 2, 2004

This paper investigates why and how systems approaches can help in evaluating the
design of new Information and Communication Technologies (ICTs) as social learn-
ing platforms. It focuses on the prototypes created by the research project Virtualis,
whose objective is to promote social learning on environmental concepts and practices
amongst a variety of stakeholders. The paper presents the principles of systems thinking
and practice that did help in formulating such evaluation processes. It illustrates how
both a peer systemic evaluation process (within the research team) and a participatory
evaluation process (involving potential future users of the ICTs) were carried out.

KEY WORDS: peer evaluation; participatory technology assessment; systems thin-

king; ICTs.

1. INTRODUCTION
The formulation of this systemic evaluation methodology took place in the con-
text of the Virtualis research project.2 This project involved a multidisciplinary
team of Information and Communication Technology (ICT) specialists, “learning
experts,” and environmentalists, all interested in exploring how ICTs can help a
variety of stakeholders coming from different backgrounds in understanding the
natural environment and how human activities can affect it. We constructed var-
ious ICT prototypes aimed not only at improving environmental awareness and
stakeholders’ involvement in environmental decision making but also at facilitat-
ing changes of (environmental) practices in order to promote more sustainable
ones. The premise of Virtualis was a special interest in ICTs as “democratic,
non-exclusive learning platforms”—that is, platforms that have the potential to

1 Department of Systems, Faculty of Technology, Open University, Walton Hall, Milton Keynes, MK7
6AA United Kingdom; e-mail: s.m.simon@open.ac.uk.
2 The Virtualis project, or “Social Learning on the Environment with Interactive Information and Com-

munication Technologies,” was funded by the EU and lasted from Sept 2001 to April 2004. For more
detail, check http://systems.open.ac.uk/page.cfm?pageid=sustdevptV

471

1094-429X/02/1000-0471/0 
C 2004 Plenum Publishing Corporation
472 Simon

give a voice to groups of people who are normally ignored, either for social or for
political reasons, or because they are considered as “non-experts.” We believe that
“environmental knowledge” is multifaceted and that learning platforms that wel-
come and merge a diversity of knowledges on environmental issues and sustainable
practices are needed.
Here, we describe the learning processes that, we hope, will take place when
people use these ICTs as “Social Learning” platforms: the objective of these ICTs
is not only for people to learn individually, but also to use an interactive tool
as a way to share their knowledge. This project puts a special emphasis on the
communication and exchange that can take place between expert and “non-expert”
stakeholders.
The evaluation tool described can help us identify whether the objectives of
these ICTs have been met and, consequently, what makes a good interactive ICT,
the use from which can generate social transformational learning.
Various types of methodologies can be used when evaluating ICT tools. In
this paper, we explain why developing a systemic methodology is of particular
relevance when working on ICT prototypes that are focused on social learning
processes. We also present the two main dimensions of this methodology (a peer
evaluation process taking place within the research team and a participatory eval-
uation process involving future users) and how it was implemented while the
Virtualis water prototypes were being developed by the Cranfield Virtualis team.
Interestingly, these evaluation exercises highlighted the fact that social learning
was potentially going to take place when the ICTs would be used but also pre-
cisely when the ICTs were being constructed, within the very team of virtualis
partners.

2. EVALUATING SOCIAL LEARNING: WHY FOCUS

ON A SYSTEMS APPROACH?
2.1. Social Learning on Environmental Issues
Learning takes place in different ways for different people living in differ-
ent contexts; it extends well beyond the boundaries of traditional “educational
institutions.”
In the area of environmental decision making and debates, stakeholders, in-
cluding policymakers, have realized that the construction of environmental knowl-
edge cannot be limited to collecting data constructed by “environmental experts.”
Many stakeholders, considered as nonexperts, also have a respectable knowledge
of environmental practical problems and could suggest a plethora of potential “best
sustainable practices” to each other.
In this paper, we are interested in the broad notion of “learning” on envi-
ronmental issues, both in conceptual terms (for instance, we need to understand
Systemic Evaluation Methodology 473

better the notions of uncertainties and complexities related to the functioning of

the natural environment and the way in which we impact upon it) and in practi-
cal terms (for instance, what does respecting sustainable yields means in practical
terms). Recently, agencies such as the UN, or the EU, have realized that they would
benefit from the participation of stakeholders in environmental policy processes.
The consequent popularity of the notion of stakeholders’ participation has been
taking place, in parallel to a growing scepticism regarding scientific expertise.
This, remarkably, opened the door of environmental debates and decision making
to more of the “nonexperts, more practical, stakeholders.”
This resulted in not only revolutionizing the way in which we think about the
natural environment but also the way(s) in which we learn about it. “Environmental
learning” extended in terms of its content, its learners, its teachers, and its outcomes.
Understandably, it has thus become one area of focus for what is now described
as “social learning.”
In this paper, we argue that evaluating whether social learning is taking
place in the context of environmental issues, and when using ICTs such as the
ones developed in the Virtualis project, can be greatly helped by using systems
thinking.

2.2. A Systems Approach to Social Learning

Systems thinking considers the various agents interacting in the world as
“systems.” It provides a multidimensional framework in which information from
different disciplines and domains can be integrated without being forced into one-
dimensional mapping. As Flood explains, “Systemic thinking explores things as
wholes and is highly relevant because the world exhibits qualities of wholeness.
Life events appear to be distinct in space and time, but they are all interconnected.
They can be made sense of in a meaningful way only in the knowledge that
our actions contribute to patterns of interrelated actions” (Flood, 1999, p. 13). In
what follows, we use the main systems concepts defined in Box 1 to explain our
perspectives on social learning and environmental issues.
Numerous authors have been working on the analysis and the operationaliza-
tion of sustainability in systemic terms (Clayton and Radcliffe, 1997; Hoekstra,
1998; Marten, 2001; Newson, 1997; Stowell, 1997). In this paper, we present
the concept of sustainability from a “human ecology” or “ecological economics”
(Martinez-Alier, 2002) angle: ecological and human systems are interconnected.
To understand sustainability therefore requires some understanding of the behavior
of human and ecological systems and how they dynamically interrelate.
We understand the set of activities of the Virtualis project as a complex system
dominated by one main objective: social learning—focused both on the improve-
ment of stakeholders’ environmental awareness and societal practical changes
in natural resources management. The main boundary of our system reflects this
474 Simon

Box 1. A Few Key Systems Terms

System: A part of reality conceived as a coherent whole of interacting entities. An open

system is connected to, and interacts with, its environment. A closed system does not
take in from, or give out to, its environment. In practice, close systems do not exist. The
world can be thought of as a large and complex system, which contains subsystems such
as ecological, economic, and human social systems.
Emergence: Systems have properties that the systems components do not have.
An emergent property of a system is that which is not readily explainable with reference
to subcomponents.
Hierarchical control: Hierarchies are levels of relative complexity within a system, and
hierarchical control refers to the imposition of new functional relationships by each level
on the detailed dynamics of the level below. Controls can be positive (where certain actions
are promoted) or negative (where certain actions are constrained).
Boundary: The real or abstract delineation between a system and its environment.
Feedback loop: An iterating chain of causal connections. With negative feedback loops,
change is effected in a direction that makes further changes less likely. With positive
feedback loops, change is effected in a direction that makes further changes more likely.
Complex systems: Complex systems generate outcomes that depend on numerous in-
teractions. For instance, global socioeconomic and environmental systems are complex
systems.
Complex adaptive systems: They interact with their environment and change in response
to environmental change.
Sources: Clayton and Radcliffe (1997) and Hoekstra (1998)

particular focus on social learning. The representation of social learning as a system

reflects its dynamic, complex and adaptive characteristics.
Social Learning, within Virtualis, is both a set of processes, a means and an
end. The premise of our project is that Social Learning processes can improve
stakeholders’ awareness and participation in environmental deliberation and deci-
sion making and therefore contribute to practical change in environmental man-
agement as well as institutional change. In other words, social learning can not
only contribute to the operationalization of sustainability; it also encourages more
social learning—hence fuelling a positive feedback loop centred on governance,
participatory processes, and collaborative learning. The main tool used for this
social learning process to take place is the suite of Virtualis prototypes.
The emergent properties of the system as a whole can only be identified
in the long run—when the effectiveness of the prototypes will be testable once
they have been implemented by a wider audience than our evaluation samples.
Nevertheless, some emergent properties relating to subsystems can already be
analyzed and contribute to the tuning of the prototypes. The delineation of the
subsystems represented in Fig. 1 resulted from the ways in which we identified
areas and criteria of evaluation of the Virtualis prototypes, in lines with the original
objectives of our project.
Systemic Evaluation Methodology 475

Fig. 1. Activities, application domains, and areas of interest in the Virtualis project.

This systems diagram presents six main systems of interest focused on the
prototypes:
– their users, be they experts and nonexperts;
– their design;
– their policy dimensions;
– the changes that their use will, we hope, generate;
– the environmental domains they cover; and
– the learning content and strategy they encompass.
The bigger, overall, system, “social learning,” illustrates the fact that the
learning that is taking place when the prototypes are being used goes beyond the
mere use of the tool: it extends to the way in which the learning effects changes in
society, in policy design, in participatory processes, or in the improvement of the
prototypes throughout time.
One can identify the interactions taking place between the components of
each of these systems. For instance, within the system “Environmental themes
covered by the prototypes,” it is interesting to examine the links between the
information provided within the prototypes, the way in which stakeholders collab-
oratively define an environmental problem or issue, and the process through which
a stakeholder becomes sensitive to an issue, whether this issue is directly related
to his/her daily life or not.
476 Simon

Looking at these three components can help us understand a great deal about
the meaning of environmental information and the perception of “environmen-
tal problems” in a context of social learning. These three components are also
related to the other components of this system—for instance, the various ways
of improving stakeholders’ motivation to learn and construct meanings about the
environment can be strongly motivated by the wish or need to improve the quality
of the environment, or by new policy measures.

2.3. The Systemic Evaluation of Social Learning Processes Taking Place

in ICT Platforms
Information and Communication Technologies can be evaluated in various
ways: some evaluation methods do, for instance, focus on Quality Assessment,
or on the ways in which information is being presented or stored (OECD, 2000).
In this paper we evaluate the ICT prototypes created by the Virtualis project in
terms of the learning processes and outcomes they generate, based on the as-
sumption that these outcomes are to be systemic in nature. The Virtualis ICTs
will be used by various types of stakeholders to investigate their impacts on the
environment as well as various options for environmental practices. This will be
done through the use of an interactive ICT platform that will serve as a “learning
facilitator.”
The four types of prototypes3 that have been developed are
– the personal barometers—they allow users to identify their personal envi-
ronmental impacts
– the scenario generators—these explore how various individual impacts are
combined at the level of a region, for instance
– the multiplayer games—they explore deliberative processes and how both
knowledge and different perspectives can be shared
– the virtual reality supports—they help the user explore a situation as a
whole.
In the context of this evaluation, we are particularly interested in the area
of “team learning” as explored by systems thinkers. The aim of team learning
is to achieve alignment in people’s thoughts and energies: even if people do not
agree on everything, they can collaboratively construct a more common balanced
understanding of a situation. Through communication, and the focus on a common
direction (in our case the pursuit of sustainable practices and lifestyles), the learning
process can create resonance so that the whole team achieves an understanding of an
issue that is more than the sum of the team members’ understanding. If the learners

3 For more information on these prototypes, visit http://neptune.c3ed.uvsq.fr/virtualist/project

description.php.
Systemic Evaluation Methodology 477

have got conflicting objectives and aspirations, however, the qualities valued in the
learning process, such as empowerment, may actually increase conflict, yielding
a whole team that achieves less than the sum that its members can, if they were to
work in isolation.
Team learning in Senge’s outlook (Senge, 1990) is best pursued through
methods of dialogue. In this way, people make a genuine attempt to appreciate
matters of concern through the eyes of people who raise the concerns. People
learn from this by expanding their understanding of circumstances that prevail.
Systems thinking can help to bring together people’s mental models in a shared
systemic language, generating team learning and understanding, and a shared sense
of purpose.
Within team learning, one type of open learning (the most common) is partic-
ipatory. It can be described as the freedom to speak one’s mind and to state one’s
view and is reflected through interactive processes in ICTs, as well as debates
with other stakeholders. It can encourage wide modes of involvement in decision
making. Another generative learning comes from reflective openness: this entails
challenging one’s own thinking. It necessitates surfacing assumptions that shape
our views and then subjecting the assumptions to open criticism.
One important challenge of the Virtualis ICTs is to demonstrate that team
learning can empower their users, be they environmental experts or not. One way
to do so is by generating a common sense of purpose in which they focus energy
in a meaningful way. Systems thinking may empower people by enabling them
to begin to appreciate rather than be confused by the interrelated nature of the
world and how this might cast insights into their experiences. In order to put these
principles into practice, systems thinkers have been keen to promote “interactive
planning.” Interactive planning builds on the premise that obstruction to change sits
mainly in the minds of participants, rather than separately “out there” in the problem
context. Interactive planning is a form of scenario building methodology that offers
in tangible form helpful guidelines about realistic intrinsic desires and shared
vision. The Virtualis ICTs have been developed in view of helping stakeholders
understand better the impacts they have on the environment and how they could,
individually or collectively, change their practices and lifestyles. A change in their
way of thinking and behavior would reflect the outcome of a successful learning
process.
We have chosen to develop our systemic evaluation with these principles in
mind, the objective being to check whether the ICT platforms developed in the
project genuinely allow team learning and interactive planning to take place in an
empowering learning environment. In order to do so, we have focused on
– examining three types of systemic interactions (between learning stake-
holders, environmental applications and between learning and change) in
a peer evaluation and
478 Simon

– allowing main evaluation concerns to emerge through a participatory eval-

uation process
as described in the next two sections.

3. LONG-TERM PEER EVALUATION: FOCUS ON THREE TYPES

OF INTERACTIONS
In this section, we describe the evaluation that some of us (the “learning ex-
perts”), within the Virtualis team, carried out on the ICTs being developed by our
ICT colleagues. We did this over a long period of time which, we initially hoped,
would help us adopt a systemic approach by allowing us to take time to appreciate
all the components at play and how they were related to each other. The objective
was to have a systematic approach and to communicate our feedback to the techni-
cians in view of recommending some refinements in the design of the prototypes.
Figure 2 illustrates the three types of interaction we examined when helping
our research partners design the prototypes. We evaluated the storyboards and the
draft versions of the prototypes over many months.
From a systemic perspective, we wanted to ensure that the prototypes were
designed in such a way that they could

Fig. 2. Focus on three types of interactions.

Systemic Evaluation Methodology 479

– help users understand the links between various environmental domains

and problems
– be used as a platform for deliberative, interactive, reflective learning
amongst learners
– facilitate the users’ transformational learning experience.
We developed an evaluation questionnaire that we and other members of
the team could use, while the prototypes were being designed and from which
a series of recommendations for improvement could be made. The parts of this
questionnaire that are relevant to each of these types of links are presented in the
next paragraphs. We used these questions to evaluate the prototypes focused on
the water and agriculture themes. The recommendations were exchanged within
the Virtualis team via e-mails.

3.1. The Links Between Environmental Domains

The social learning processes that relate to these links could be described as an
iterative learning process (Fig. 3). The prototypes are intended to not only provide
information but also invite people to deliberate in order to collaboratively con-
struct meanings and exchange knowledge on environmental issues. They provide
simulation tools that can help people understand the impacts of their individual or

Fig. 3. Social learning on the environmental domains.

480 Simon

collective actions (environmental practices), hence affecting the way in which they
initially perceived the environmental issue. Uncertainty and surprise also entered
the “equation” and contributed to the practical testing of the original understanding.
The social learning processes based on the environmental domains explored
in the prototypes are therefore made of a “mixture” of practical experience, per-
ceptions, and understanding of an environmental issue, provision of “scientific
information,” and uncertainty and surprise. By mixture, what I really mean is
that these components are linked to each other through the deliberation processes
facilitated by the prototypes.
The evaluation questions related to the links between environmental domains,
as well as their relevance from a systemic perspective, are presented in Table I.
These questions could be used and adapted by anybody designing a questionnaire
aimed at evaluating, in a systemic way, ICTs focused on environmental issues.
More questions could be asked, of course. We found that explaining why asking
these questions was relevant (right column of the table) proved to be an important
part of the design of the questionnaire as well as of the learning process involved
in identifying “what makes evaluation systemic.”
The outcome of our evaluation of the Virtualis water and agricultural proto-
types can be presented as a series of recommendations that are valid to anybody
trying to develop the same type of ICTs. They relate to the following themes:
– Systemic visualization of interrelated environmental issues: Illustrating
the links between environmental domains and issues should be helped by
the creative innovative representational systems provided by new ICTs.
Virtual reality, for instance, can help the user explore an environmental
domain and get an overview of how different issues and aspects of this
domains relate to each other.
– Links between geographical scales: The progression in learning between
the various types of Virtualis prototypes (e.g. between the personal barom-
eter, focused on individual environmental impacts, and the scenario gener-
ator, focused on aggregated impacts at the society level) allows the users to
understand the context of their impacts and which effects they have, once
combined with other stakeholders’ impacts.
– Links between the specific issue and the broader context: The instructions
given (text, videos, audio, animations) and the Virtual Reality tool in par-
ticular, should clarify the context within which the environmental issue is
being explored. We felt that this contextualization was potentially missing
from the prototypes we evaluated and that the integration of some hyper-
links to sites that either help understand the background or context better
could be a good starting point to correct this shortcoming.
– Social learning processes should allow users to understand the links be-
tween environmental domains through the perceptions of stakeholders
Systemic Evaluation Methodology 481

Table I. Evaluation Questions Focused on the Links Between Environmental Domains

Issues covered—is the Here the evaluator identifies whether the prototype offers a reductionist
information provided view of environmental domains or whether it allows the users to
correct, up to date, appreciate the broader, systemic meaning of the issues presented in
systemic? the prototype.
Are the issues relevant to The links this evaluation question is intended to highlight are the links
the users? between the representation of reality in the ICT and the real life
experience of the user—how is the prototype highlighting such a
link?
Are the environmental “Contextualizing” is a crucial component of systemic approaches. In
issues covered order to appreciate the systemicity of an issue, one can be helped by
contextualized? exploring the boundaries that delineate the issue from its “external
environment.”
Concepts used—how do This evaluation question relates to the choice of concepts used to
they affect the users’ animate the curiosity and interest of the users on the subject. The
perceptions of the issue question is how do these concepts bridge the reality of the issues and
at stake? the perceptions of these issues by the users?
Are meanings clearly This evaluation question is intended to assess whether the understanding
explained? of what defines the issues and domains explored is both clear and shared.
The prototypes should help the users to share their understandings of
an issue.
Is the information on the The prototypes are intended to address current “priority issues.” We
environmental theme also need to ensure that the ways in which people’s perceptions,
precise and up to date? understanding, and sensitivity to certain issues evolve with time are
taken into consideration.
Are users likely to be This evolution of understanding and ways of addressing an issue can,
motivated to reuse the in part, emerge from using the prototype and through embracing an
prototypes? iterative mode of learning. The question is whether the prototype
can be used as an inviting means for people to make sense of various
environmental issues and to improve their understanding of
them—which might ideally include them being able to appreciate
the systemicity of environmental issues.
Are notions such as The environmental domain must be presented in a way that is not
complexity and simplistic nor reductionist. The prototypes must help in addressing
uncertainty related to the complex characteristics of the issue discussed, as well as the
the environmental uncertainties attached to it. The social learning process is intended
themes addressed in the to acknowledge and face both complexity and uncertainty as
prototypes? important characteristics of environmental issues to be taken into
consideration both in “environmental information” and in
environmental decision making processes.
Is the presentation of the The use of ICTs allows one to represent issues in alternative, creative
issues clear? Does it ways—we therefore believe in the importance of making the best
affect the users’ out of these ICTs capabilities in view of both animating the users’
perceptions of the curiosity and opening up their perspectives on the issue.
issues?
Is the learning structure The learning path designed through the prototypes must help the user
presented in the in developing a systemic cognitive representation of the
prototype clear and environmental issue.
understandable?
482 Simon

Table I. Continued

Does the prototype provide The advantage of using hyperlinks is that the users can concentrate on
hyperlinks to useful urls one main domain, while appreciating the ways in which they are
that contain information linked to other domains.
on these issues?
Does the prototype provide These might represent the four domains in different ways and this
hyperlinks to other types might help the users understand better the links between different
of similar ICT tools? types of environmental domains.
Is more than one Although the prototypes focus on one environmental domain and
environmental domain hyperlinks can be used to highlight the links with other issues, the
covered by the facilitation carried out within the prototype needs to explicitly make
prototype? it clear that, for instance environmental domains focused on in
different prototypes or suites of prototypes (for instance water and
agriculture) are very closely interrelated.

(from the same or from different countries) on these interconnected issues.

The facilitation questions (or “learning triggers”) used in the prototypes
could allow the users to express their views and perspectives.
– The policy relevance of the prototypes is clearly focused on the sustainable
management of resources: the links between environmental issues and
domains is therefore also made in the context of “sustainable policies”
and practices. The examples of sustainable practices provided through
the prototypes should come from users as well as from the experts who
designed the prototypes if a real social learning process and exchange is
to take place.

When designing and constructing such prototypes, one difficult question to

tackle relates to the stakeholders’ perceptions of how environmental issues and
domains interrelate. Here, allowing the participants to share their views is necessary
and so is the facilitation of the “collaborative construction of meanings” and of the
process of capturing the outcomes and evolution of the debates. They both require
careful moderation.

3.2. The Links Between Learning Stakeholders

Our second focus of evaluation was the ways in which the prototypes’ users
deliberate, are empowered to do so, and how they learn from each other. Here again,
the social learning process takes place throughout time and can be described as
another spiral within which three main components (consultation, deliberation,
and participatory technology assessment) interrelate (Fig. 4).
The questions we focused on, in our evaluation questionnaire, are listed in
Table II.
Systemic Evaluation Methodology 483

Fig. 4. Social learning on the construction of participatory ICTs.

Here again, these questions can constitute the basis of other evaluation ques-
tionnaires of ICTs focused on social learning and participation. Indeed, debating
with other researchers like yourselves, about which other questions might be seen
as relevant in the context of systemic evaluation of participatory process would be
a social learning experience in itself.
The various ways in which social learning is to be promoted in the Virtualis
prototypes are through people’s sharing of knowledge and experience, deliberation,
and collaborative tasks. Certain skills have to be developed through the social
learning process. Thus, meaningful interaction of the form argued by Cobb (1996)
requires some negotiation of meanings, probing one another’s understanding, in
attempt to generalize meaning across different experiences. For social learning
to take place, the prototypes must therefore be tailored appropriately and certain
characteristics of ICTs have to be valued—for instance, the ability of people to use
these tools at a distance and to work on interactive tools. Hypermedia have in fact
been described by many as “enablers of social learning” (Jonassen, 1993, 1996)
in that they can help generate a shift from electronically presenting information to
providing support for the learner in constructing knowledge and deriving meaning.

3.3. The Links Between Learning and Change

Social learning processes promoted through the Virtualis prototypes are also
intended to provide platforms to what is most commonly known as stakeholders’
484 Simon

Table II. Evaluation Questions Focused on the Links Between Stakeholders

Relevance of the subjects Often, people disagree on environmental issues; have different
covered and discussed objectives, interests, or constraints. For them to become interested
for a varied audience? in learning together and from each other, not only the facilitation of
these deliberations must be carefully designed but also people must
be interested in the subject to start with. The prototype must
highlight how relevant the issue is to the users.
Clear instructions People from different backgrounds, professions, views on an
concerning the environmental issue, do not communicate as often as would benefit
involvement and society. Learning from conflicts is something we are not used to
participation of the value and therefore people are not used to deliberating and learning
users in collaborative from each other in a participatory, nonexclusive way. The
learning? instructions included in the ICTs on how users can deliberate,
negotiate, collaborate, must be genuinely facilitating. These
instructions can really be the key to successful social learning.
Clear learning structures? Facilitating social learning processes is one thing, helping the
participants in capturing the learning from it is another. Whether
they are linear or not, the learning paths designed in the prototypes
must help the users in realizing that various learning steps have
been “climbed” and that something (learning and potentially
practical change) is emerging from this process.
Hyperlink toward other Social learning refers to what is being learnt about but also how
learning ICT learning is taking place. In terms of learning, participation and
platforms? collaboration comes into play not only regarding the exchange of
various views on an issue, or the exchange of information and
knowledge, but also when people are prepared to learn differently,
through different processes. Social learning can therefore also take
place when it comes to “learning to learn.”
Possibility of using the The type of participation that Virtualis is encouraging could involve
ICT learning tool at a people who live far from each other. This is one of the most
distance? interesting characteristics of ICTs that such prototypes should make
the most of.
Identification of targeted This should help the prototype designers in building some flexibility
users? (in terms of presentation of the material, type of help provided,
examples given, type of learning feedback provided, etc.) and in
helping participants in communicating better too.
Relevance of the modes There are different ways of learning about a same issue. To retain the
of learning for a varied motivation and attention of all types of participants, the modes of
audience of users? learning must be varied enough and appropriate to different types of
tastes and needs.
Promotion of autonomous For social learning to take place, people must trust each other and also
learning? their facilitator. The trust must not only focus on what is learnt and
how, but also on people’s belief that them becoming autonomous
learners is one of the main outcomes of this social learning process.
Bias taken into account in The existence of bias in the way in which issues and/or perspectives
collaborative learning? are sometimes presented in society has to be addressed in the
context of social learning process, not only in order to install an
atmosphere of trust but also to demonstrate a rigorous exploration
of knowledge and meanings.
Systemic Evaluation Methodology 485

Table II. Continued

Communication promoted Communication is a key component of social learning. The prototypes

amongst stakeholders? must allow the participants to communicate, deliberate with each
other.
Emotional involvement of The differences in views, attitudes, etc. toward a subject can be
the users? expressed through emotions. One must bear in mind the fact that,
the more motivated to learn a user is, the more he or she will “feel”
about this issue. Encouraging participation and deliberation implies
that the deliberation platform has been designed in such a way that
people can express these feelings. Often, it is when feelings can
finally “get out” than people realize the importance of an issue.
Participation of expert as One of the characteristics of the Virtualis prototypes is that they
well as nonexpert promote social learning as an “enabling process” through platforms
stakeholders? that build bridges between experts and nonexperts. Not only
participants must be enabled to contact experts, but their
communication must be facilitated by the platforms and skilled
moderators.
Users enabled to input The communication and deliberation promoted through the prototypes
their knowledge? ought to not only focus on information and issues provided through
the prototypes but also on information and knowledge inputted by
participants themselves. The prototypes must be used not only as an
enabling platform but also as a recipient to new information to be
stored and to work on.
Instructions given in If collaborative social learning is to be promoted through such
different languages? platforms, using internet facilities, then participants from different
countries should ideally be able to communicate in their own
language and have translation of what other participants reply.

participation in environmental decision making. One of the premises of the project

is that people will not be motivated to embrace new environmental laws, aware-
ness, and sensitivity to environmental issues, and more environmentally friendly
practices unless they can be involved in the deliberation processes, and acquire a
sense of ownership of the issues at stake. The Virtualis prototypes are therefore
designed as enablers as well as facilitators toward practical societal change. Here
again, the system we are looking at and want to evaluate is made of various com-
ponents that are interrelated as described in Fig. 5. Social Learning results from
deliberation, linking perception and practical change, and evolutionary thinking
in systems of governance and policy priorities.
The evaluation questions focused on such interactions were formulated as
described in Table III.
The prototypes explore the practical context within which changes resulting
from the learning experience offered in the prototypes might emerge—thus, the
water prototypes explain the importance of the Water Framework Directive, or the
new approaches to catchment management, for instance.
486 Simon

Table III. Questions Focused on the Links Between Social Learning and Societal Change

Context of issues Often, stakeholders do not realize that a particular issue is actually
discussed in prototype practically relevant to them. Articulating this relevance to them and
and context within helping them understand in which context the ICT platform is used
which prototype used and useful then becomes crucial.
and developed?
Clear expression of Most users will want to know why they are doing what they are doing
objectives of the in the prototypes—the learning and practical outcomes (changes in
prototypes, context, practices) constitute the major motivators for the users.
and expected
outcomes?
Is the user motivated to go Maintaining the motivation of the users all along is crucial—otherwise
all through the learning the social learning process will be incomplete and meaningless. This
experience until he can be done both by articulating what has been learnt and how, plus
sees a result (reflected what is going to be explored next and also by reminding the context
through changes of within which this exercise is being carried out, i.e. why it is useful to
practice)? them.
Do the instructions help A component of “Social learning” lies in helping the participants
users realize when they become autonomous learners. The instructions must help them first
have learnt and how? by articulating such learning steps and outcomes, and then by
helping the participants in reflecting on learning and practices.
Gains in knowledge from Reflecting capacities developed throughout the learning experience
learning experience must help the users in articulating what was learnt, what changed
when using the through the learning processes. Not only the users should be able to
prototypes? reflect on this, but they should also be encouraged in articulating this
clearly.
Links between prior Various pedagogic strands have been developed to describe “Social
knowledge and learning” processes. One that we particularly value derives from
practice and outcomes experiential and “students”-centred learning: it is important that the
of learning? users realize (through the design and learning processes facilitated
in the prototypes) that their prior knowledge and experience is being
valued and genuinely contributes to the social learning process. If
too much comes from the “machine,” . . . we are back to an “expert
versus nonexpert” learning traditional process.
Common, collaborative Social learning is not only about exchanging knowledge and
construction of information but also about learning and working together. From a
meanings? social learning process, collective outcomes should emerge—the
construction of meanings, the concerted decisions to change a way
of managing a resource . . . The role of the facilitator/moderator in
the prototype becomes then absolutely crucial.
Change of attitudes This is (potentially) one outcome of the social learning process, one
toward the subject that the moderator must help the users realize. The prototypes could
explored as a result of provide information for people to be better equipped to actually
learning through using change and overcome the barriers in place that are preventing then
the prototype? from doing so.
Systemic Evaluation Methodology 487

Fig. 5. Social Learning: Toward practical societal changes.

The information explored in the prototypes concerning alternative options to

environmental management can help users initiate a change in the way in which
they “do” things. Users should be able to share their own knowledge concerning
alternative practices. Also, changing practices is not only related to having more
information on how to do things differently; it might be related to feeling empow-
ered because part of a group of stakeholders who have decided to change the way
in which they do things, for instance. By building bridges between stakeholders
who are not used to learn from each other in an equitable way, the prototypes could
also encourage the building of bridges between institutions or actors who typically
have the power to initiate change or ensure that change takes place. Explanations
on how the environmental issues explored in the prototypes relate to current, up
to date, environmental action are crucial for users to understand the importance of
what they are doing.
A further consideration relates to time: how long does it take for learning
processes to generate practical changes? What is needed, during this period of time,
to keep people interested in changing their way of life? The design of prototypes
ought to be considered over a long period of time: their maintenance and the way
in which learning experiences are combined in a real continuum will both help in
ensuring that the prototypes, as learning platforms, can progressively constitute
real stepping stones to action.
These were the types of issues that we thought were important to take into
consideration in an evaluation of our prototypes based on transformational learn-
ing. Here again, other evaluation questions could be thought of in the context of
other similar projects.
488 Simon

This long-term evaluation process was carried out within the temporal and
logistical constraints of the project: the teams that focused on the construction of
the ICTs and those that focused on the pedagogic evaluation of these “tools” had
different ways of learning and working, different time constraints,4 and often used
different “languages.” And so, having carried out this type of peer evaluation, we
conclude that, regrettably, and even though it did respect some important systemic
principles, it was a rather linear, one-sided process. Above all, we learnt that the
systemic dimension we concentrated on had mainly focused on the design, content,
and future use of the prototypes but not enough on the interactions taking place
between the members of our own research team during the construction of the
prototypes. This was partly due to the fact that evaluating our own social learning
processes had become a sensitive issue:
– The coordination of the project proved to be difficult, not only because
people came from different disciplines and had different tasks to focus on
but also because, like in any European Research project, many partners
were involved (11 in our case, with at least 2 people involved per host part-
ner institution) and asked to provide deliverables that would complement
each other but which ended up doing so only at the end of the project’s life.
– Evaluating our own social learning process and the difficulties involved
when a genuine social learning process is taking place amongst people
coming from different disciplines and different cultures proved to become
a tricky exercise, from a political perspective. A critique of our learning
process was directly synonymous with a critique of our ability to do what
we actually researched on and . . . preached. We did carry out our evaluation
but made sure that it was presented as an outcome of our learning process,
something positive to build on—our funders, them, were keen to hear that
various evaluation processes had been taken place throughout the project
and that iterative modifications of the ICTs had helped in generating tools
that were “nearly perfect.”
– The timing related to the delivery of the material to evaluate, thus, did not
allow for any substantial iteration and modification, therefore making the
evaluation exercise appear as being quite linear.
If more time had been allocated to do so, the ICT evaluation process would have,
in fact, greatly benefited from having the prototypes designers involved in taking
part in the design of the evaluation questions. In effect, we wondered whether
the improvement of the prototypes wouldn’t have been more effective if we had
facilitated the collective design of the evaluation questionnaire with the designer

4 The evaluation team was dependent on the production of the tools it was to evaluate and the “ongoing”

recommendations ended up becoming “end-of-process” recommendations—there was no time left to

make the iterations and changes recommended by the evaluation team.
Systemic Evaluation Methodology 489

team—rather than asking that team to fill the evaluation questionnaire once it
had been written. We attempted to address this shortcoming in our evaluation
methodology in a participatory evaluation session we carried out with potential
future users, as described in the next section.

4. PARTICIPATORY TECHNOLOGY ASSESSMENT—ITERATIVE,

SHORT-TERM EVALUATION PROCESSES
While the Virtualis team worked toward promoting social learning processes
when stakeholders come to using the prototypes, it is also interested in observing
the social learning processes that are taking place while the prototypes are being
constructed. Indeed, one of the important characteristics of the project is to en-
courage the members of the team to converse with members of the public as well
as experts, all potential future users, to ensure that the design of the prototype is,
in itself, participatory and respectful of future users’ needs and expectations. It is
with this perspective in mind, and in view of complementing the peer evaluation
process, that the OU team organized an evaluation session with a group of our
Open University tutors coming from a range of backgrounds and perspectives,
social classes and ages (and who, therefore, tend to be more representative of the
general public than other, more traditional university tutors).
In order to carry out a participatory evaluation process, we varied the types
of activities the tutors went through and ensured that the day was made of a
mixture of individual tasks, group work (in pairs and groups of five, as well as
a few collective brainstorming sessions), sharing of perspectives, exploring the
prototypes, and filling the evaluation questionnaire online.

4.1. Participatory Evaluation Techniques

The main differences with this participatory evaluation session (as opposed
to the peer evaluation) were that
– the evaluators were numerous and their perspectives on the prototypes, but
also on the evaluation exercise, were varied
– through this evaluation session, we were facilitating the emergence of
unexpected ideas and outcomes, both on the prototypes and also on the
notion of ICT evaluation.
The principles of participatory evaluation are, to a very large extent, in com-
plete agreement with the understanding of the notion of social learning in the
Virtualis project: while “nonexperts” can learn from experts (in this context on
learning processes, structures, platforms . . . ), “experts” also have much to gain
from the knowledge, common practical sense, experiences, . . . of nonexperts. So,
leaving the task of evaluating the ICT prototypes to “learning experts” only would
490 Simon

be, from a social learning perspective, a mistake. What works and what doesn’t,
from a point of view of learning, depends on the learner and the facilitator. The
Open University makes a point of concentrating on student-centred experiential
learning, where the “teacher” is a facilitator. As a complement to participatory eval-
uation processes carried out in the context of other technological developments
(such as those focused on Genetically Modified Organisms, Joss and Bellucci,
2002), the OU Virtualis team therefore focused its participatory evaluation pro-
cesses on enabling certain recommendations and collective outcomes to emerge,
hence empowering the “users” by allowing them to take part in the design of the
(communication) technology itself.
The session therefore focused on
– honouring people’s prior knowledge on how ICTs can promote social learn-
ing and why;
– identifying what evaluation criteria seem to be of particular importance in
a broad brainstorming session organised around three themes
– the relevance of ICTs to improve people’s awareness and participation
in environmental debates and decision-making processes
– the relevance of ICTs to motivate learning processes
– ICTs, social learning and change
– discovering the prototypes in small groups
– filling the evaluation questionnaires
– discussing about the questionnaires and
– identifying ways in which the prototypes could be best disseminated and
used.
The evaluation of the prototypes constituted, in fact, a platform for broader
reflections and recommendations on how ICTs can help promote social learning.
A variety of recommendations emerged from the session.

4.2. The Outcomes

In the first brainstorming session, focused on participation and ICTs, the
tutors concentrated first on discussing the notion of participation. It was first ac-
knowledged that “participation” facilitated by such ICTs could range from getting
informed, being more environmentally aware, to changing one’s behavior as a re-
sult of “transformational learning.” But the differences in levels of knowledge and
awareness were presented as having the potential to result in interactions amongst
participants that are very different from what we normally imagine in participa-
tory processes, as explained in this quote (from one of the tutor): “Participation has
many facets. It can be dangerous if people take part from a position of ignorance. It
can increase ownership of decisions and could cause disillusionment if participants
Systemic Evaluation Methodology 491

have different power levels.” Ultimately, what emerged from this first discussion
is that “the question [of whether ICTs can facilitate participation] is impossible
to answer. The technological tools are neutral: what needs to be evaluated is how
they are used—we can only evaluate the actual usage of tools.” The discussion
then focused around three areas. First, tutors were keen to address the question
of “participation around what issues?.” On the one hand, people acknowledged
that the use of ICTs can help in providing up to date information, from a range of
sources. But they expressed concerns regarding the choice of information provided
to people. Would people trust this information? How relevant would this informa-
tion be to people if they don’t choose it? Will the presentation of information in
ICTs make it appear as unimportant because part of a “game?” And, beside, would
the use of such tools in the arena of decision making dictate what type of issues,
selected, “frozen” by decision makers, would be discussed? Would the choice of
issues discussed be updated, maintained frequently enough?
This led the participants to discussing who would take part in using such ICTs
platforms. The fact that these tools are targeted at the “general public” generated
more scepticism than enthusiasm in the group, as expressed by one of them:
“Are this form of participation and choice of issues equivalent to ‘the others’
telling us (the general public) what to do—while commercial organizations are still
not changing their understanding of environmental issues nor their environmental
practices?.” Beside the question of “who are these tools for?,” the question of
“who can they realistically be for?” was raised. While the issue of accessibility
was regarded as less and less of an obstacle, at least in European countries, the
questions of who would be motivated to use such tools and how to bring these
issues to the attention of a wide audience was discussed at length. As was stressed,
local (environmental) authorities websites—including the ones with consultation
platforms—do exist but it doesn’t mean that people take part in using them. It was
emphasized that, ideally, a maximum user coverage, including corporations, was
needed for an effective use of these prototypes. In other words, the use of these ICTs
by a self-selected audience who would, anyway, be interested in these issues, would
somehow defeat their object—they would “preach to the already converted.” Much
more of a challenge would be to direct the attention of a wide range of people to the
existence of these tools by, for instance, using other media (radio, TV, magazines,
etc.). Interestingly, many tutors emphasized the importance of the “human factor”
in promoting the use of such ICTs. Thus, “young people might come to use these
ICTs by browsing or by ‘word of mouth’ between their peers and school pupils,
excited by the use of ICTs, will help older generations in getting familiar with and
interested in them.” So the non-ICT aspect of the initial introduction to the tools
was presented as having a major importance, somehow beyond the way in which
the ICTs were designed. The variety of media encompassed in ICTs was therefore
presented as having a positive impact on their users. But it was recognized that
users had to be ICT literate in order to make the participation processes more
492 Simon

effective— or else, the navigation tools included in the ICTs, the way in which
the information is presented and the interactive tools integrated in them, have to
genuinely enable people to comfortably take part in the social learning process.
The tutors spontaneously divided the second brainstorming, focused on the
relevance of ICTs to facilitate learning processes, in two parts:
a) The first point focused on “motivation ingredients” for learning. From a design
perspective, tutors highlighted the fact that the learning instructions, the facilita-
tion within the ICTs, need to address the issue of “retention of knowledge and
understanding”—and therefore reflective and self-assessment questions need to
be included. The temporal dimension of the learning processes was also related
to the design of the ICTs in that the tutors stressed the fact that the ICT users need
to have time to follow the learning through each prototype. Beside, and amazingly
so, the principal focus of discussion became “interaction, interaction, interaction”
as the best motivation to learn. The main advantage of using ICTs as learning
processes was explained by the fact that these facilitate “learning from others.”
“The interaction component is the crucial motivator for deeper learning.” The use
of ICTs for doing so was presented as particularly interesting because still novel to
the users: this means of learning is progressively breaking down barriers for long
put between education and leisure and the learning experiences can be varied with
the user being in control of his/ her own learning path.
b) The second point covered in this brainstorming session was on the characteristics
of ICTs that would help in promoting participation and interactive learning. Many
advantages in using ICTs as learning platforms were highlighted (they take out
some of the tiresome reading; they offer all sorts of multimedia potential (sound,
sight, interactive environments, 3D representations of information, etc.). They were
seen by the group as generally giving learning more scope. However, the tutors
also highlighted the fact that, in order to make the best use of these new tools,
people must be trained to use them, like they would use a new language.
In the third brainstorming session, we addressed the issue of transformational
change—i.e. how the learning processes experienced in such ICTs could result in
practical changes in the users’ ways of life, for instance. The main change that
the group highlighted was the fact that “ICTs will certainly change the way in
which we socially interact.” But whether these changes are for the better or for the
worse depends on the overall structured framework and societal strategy within
which these changes are taking place. As one tutor stressed “affordable housing
is high on the agenda in the UK, but what about helping with making computers
affordable to more people?.” This would help bring new learning opportunities to
previously excluded groups (disabled, people for whom the cost of higher education
is prohibitive, etc.). Beside, the use of ICTs as educational/learning tools will
have to be introduced carefully since, whether these ICTs are supposed to be
participatory or not, “current evidence appears to suggest that the use of ICTs in
education reinforces societal divisions.”
Systemic Evaluation Methodology 493

An adaptation phase (the one we are currently experiencing, at least in some

European countries) during which people will “re-skill” in these new forms of
communication will take place. So, to some extent, the changes experienced as
a result of using ICT tools such as the ones developed by Virtualis might ini-
tially be mainly focused on the methods of interaction—beside the environmen-
tal issues explored within these tools. During this phase, another main change,
maybe less obvious, will probably be related to the raise in awareness on cer-
tain issues. And that’s a start. But in terms of linking the participation pro-
cesses facilitated in such ICTs to the actual decision-policy-making processes,
the group was not overenthusiastic: “Do they really want your contribution to
a poll or will they then use the poll results to justify decisions of powerful ed-
ucation or manipulation?.” Overall, therefore, the changes expected from the
use of such ICTs were regarded as . . . deriving from the use of these tools in
itself!

4.3. Discovering the Prototypes

The objective of these brainstorming sessions was to deepen the group’s un-
derstanding of the context within which the prototypes created in Virtualis have
emerged. The mistake, from an evaluation perspective, would have been to present
the prototypes first—and, in the process, make tutors believe that they were rep-
resentative of any good ICT learning platform. These brainstorming sessions con-
stituted an interesting insight to us on what final users would expect from our
prototypes and what they would be sceptical about even before starting using
them. Having gone through these discussions, the tutors then went on discover-
ing one of the prototypes (the water “personal barometer”) in depth. The water
personal barometer5 allows its users to calculate his/her water consumption and
“water shadow”—i.e. the area of water needed, in the region where the user lives,
to meet his/her water needs. This tool can be viewed as a platform for reflection
on how to use water resources better.
People worked in pairs to discover the water prototype constructed by Cran-
field. We thought that working in this way would allow them to help each other
feel more confident about the discovery of the technological tool and also that it
would allow them to express themselves freely.
The prototype had been installed on a laptop placed between the two people.
People explored the prototype on screen and were also provided with the users’
manual that complements the software. For a bit more than an hour, people navi-
gated through it and had the opportunity to share their views within their pair. They
were asked to write down their first reactions as well as the criteria that would, in
their view, help one evaluate such ICT tools.

5 For more information, check http://neptune.c3ed.uvsq.fr/virtualist/water/personal barometer.php.

494 Simon

The prototype worked beautifully and the navigation was faultless. The com-
ments therefore focused more on the content, assumptions, guidance, and presen-
tation of the material.
Interestingly, most comments were focused on the fact that there was a crucial
need for the contextualization of both the content of the learning and the learning
tool itself. In other words, the reason(s) why any stakeholder might be interested in
using this tools, how they would be invited to do so, why they would be interested in
focusing on an environmental domain such as water when they perceive that there is
enough water as it is in a country like the UK, etc. . . . needed to be presented, if not
in the prototypes themselves, at least in a “learning support” that would introduce
them. Using participants’ knowledge, as opposed to mere data and information,
was also highlighted as one of the main issues.
The users also found the prototypes relatively dry and it is noticeable that,
even amongst older generations, the expectations for up to date graphics and sounds
in such ICT tools has become very high.

4.4. Participatory Evaluation Process: Some Interesting Lessons

This method of evaluation of the prototypes opened up our understanding of
what the final users of such tools would need and expect. Interestingly, while our
peer evaluation was focused on the structure of learning processes and on whether
and how these would facilitate social learning, this participatory evaluation process
helped us in stepping back and evaluating these tools in a broader context. This
context includes the motivation for using such tool, the training needed by many
to feel comfortable in using such tools, a better understanding of the domains
covered in the ICT tool, and the creation of an atmosphere of trust within which
the learning could take place.
These series of evaluation comments and recommendations are being used to
design an online learning support,6 very much based on these contextualization
characteristics, aimed at complementing the four Virtualis water prototypes.
At the end of the day, the “human side” of technology all depends on how
humanly its use and design are integrated within society.

5. CONCLUSION
Evaluating ICTs from a systemic perspective can mean different things.
One fundamental principle of evaluation is to carefully reflect on the con-
text within which these tools are going to be used, whom by, why, and to what
extent these ICTs can offer what other tools apparently cannot. In this paper, we

6 For more information, check http://systems.open.ac.uk/page.cfm?pageid=ShortCourse

Systemic Evaluation Methodology 495

examined how to approach the issue of “systemic evaluation” of ICT prototypes

that have been designed to promote social learning. We observed how the social
learning could happen within the research team and also between the research
team and potential future users of these prototypes. The peer and the participa-
tory evaluation processes appeared to complement each other in different ways.
The peer evaluation examined the ways in which the design of the ICTs took ac-
count of the interactions between environmental domains, learning stakeholders,
and learning and change, and was carried out in the form of discussions with our
research partners and questionnaires designed by the “learning experts” team and
filled by the ICT designers as well as the learning experts. The outcomes of this
evaluation were somehow “timid” to be as constructively critical as we would have
allowed ourselves to be had we had enough time to rectify what needed rectifying
within the ICTs. On the contrary, a series of provocative recommendations came
out of the participatory process carried out with external future potential users,
highlighting the fact that, even if these three types of interactions were thoroughly
explored in the prototypes, the contextualization of their use was missing. To
some extent, thus, the evaluation of the prototypes as platforms for social learning
would gain from really concentrating on the evaluation of the social learning con-
text within which the prototypes were designed, constructed, and intended to be
used.
Above all, the message of this paper is that a systems approach to evaluation
can help in approaching an issue from various crucial angles, the links between
which is more important that their sum.

ACKNOWLEDGMENTS
Acknowledgments to James Fisher and Susan Ballard who contributed to the
Virtualis research project at the OU, as well as to Wendy Fisher who helped with
the organization of the evaluation session in Leeds.

REFERENCES
Clayton, A. M. H., and Radcliffe, N. J. (1997). Sustainability. A Systems Approach, Earthscan, London.
Cobb, P. (1996). Where is mind? A Coordination of sociocultural and cognitive perspectives. In
Fosnot, C. T. (ed.), Constructivism: Theory, perspective and practice. Teachers College, Columbia
University, New York.
Flood, R. L. (1999). Rethinking the Fifth Discipline. Learning With the Unknowable, Routledge,
London.
Hoekstra, A. Y. (1998). Perspectives on Water. An Integrated Model-Based Exploration of the Future,
International books, Utrecht, The Netherlands.
Jonassen, D. H. (1996). Scaffolding diagnostic reasoning in case-based learning environments. J.
Comput. High. Educ. 8(1), 48–68.
496 Simon

Jonassen, D., Mayes, T., and McAleese, A. (1993). A manifesto for a constructivist approach to uses of
technology in higher education. In Duffy, T. M., Lowyck, J., and Jonassen, D. H. (eds.), Designing
Environments for Constructivist Learning Springer-Verlag, Berlin, pp. 231–247.
Joss, S., and Bellucci, S. (eds.). (2002). Participatory Technology Assessment—European Perspectives,
Centre for the Study of Democracy, London.
Marten, G. (2001). Human Ecology. Basic Concepts for Sustainable Development, Earthscan, London.
Martinez-Alier, J. (2002). The Environmentalism of the Poor. A Study of Ecological Conflict and
Valuation, Edward Elgar, Cheltenham, UK.
Newson, M. (1997). Land, Water and Development. Sustainable Management of River Basin Systems,
2nd edn. Routledge, London.
OECD Information Technology Handbook. ICTs, E-commerce and the information economy (2000).
Organisation for Economic Co-Operation and Development, Paris.
Senge, P. (1990). The Fifth Discipline: The Art and Practice of the Learning Organisation, Century,
London.
Stowell, F. A., Ison, R. L., Armson, R., Holloway, J., Jackson, S., and McRobb, S. (1997). Systems for
Sustainability. People, Organizations and Environments. Plenum, New York.