Design and Evaluation of a Collaborative Telelearning
Activity aimed at Teacher Training
Barbara Wasson
University of Bergen, Department of Information Science
Abstract: This paper describes VisArt, a collaborative telelearning
scenario aimed at teacher training. Students at three educational
institutions in Norway collaborated at a distance through TeamWave
Workplace to design a learning activity for some subject of their choice.
At the University of Bergen, VisArt was part of a graduate course
assignment where the students had to participate in VisArt and write a
report reflecting on their participation from a theoretical perspective.
Salomon’s ideas on genuine interdependence and Gutwin et al.’s notions
of awareness formed the foundation for the student’s theoretical reflection.
This paper describes the VisArt scenario, the evaluation studies being
carried out within the DoCTA project, and summarises the student’s own
theoretical reflections.
Keywords: teacher professional development, evaluation, groupware
Introduction
In the Spring 1999 pedagogical information science course at the University of Bergen
(see http://www.ifi.uib.no/staff/barbara/courses/pivar99.html), one of the two semester
assignments involved participation in VisArt, a collaborative telelearning activity. In
VisArt, distributed teams of students collaborate over the internet in a common design
activity. VisArt is one scenario in the DoCTA (http://www.ifi.uib.no/projects/docta)
research project that is lead by the author. The collaborative telelearning assignment
required the students to not only participate in the VisArt activity, but evaluate their own
participation from a theoretical perspective.
The paper proceeds by describing the DoCTA project and the theoretical framework
within which the semester assignment is situated. Then VisArt and the collaborative
telelearning assignment are described and its evaluation discussed. The paper concludes
with a summary of some of the student’s reflections on the VisArt scenario.
Project DoCTA
DoCTA (Design and use of Collaborative Telelearning Artefacts) is a multidisciplinary
research project administered and co-ordinated by the Department of Information Science
at the University of Bergen. The project is funded by KUF's ITU (IT in education)
programme, and is a collaboration with HiNT (Nord-Trøndelag College), HSH (Stord /
Haugesund College) and Telenor Research & Development, Kjeller. The 16 researchers
and graduate students involved have various backgrounds including computer science,
psychology, sociology and education.
The project focuses on the design and use of technological artefacts to support
collaborative telelearning aimed at teacher training. The research is not limited to only
studying these artefacts per se, but includes social, cultural, pedagogical and
psychological aspects of the entire process in which these artefacts are an integral part.
This means that we both provide and study virtual learning environments that are being
deployed to students organised in geographically distributed teams. The main research
focus is reflected in both the theoretical and methodological approach chosen in the
project. The theoretical, or conceptual approach, is rooted in a sociocultural perspective
that emphasises an understanding of language, culture and other aspects of social setting.
The methodology is influenced by ethnographic studies, favouring naturalistic and
qualitative research methods.
Various scenarios utilising the Internet are used to engage the students in collaborative
learning activities. Through participation, teachers gain experience with not only
collaborative learning, but with collaborative telelearning, and the design of textual or
visual artefacts. Each collaborative learning activity has been designed to place a strong
emphasis on active engagement through both "hands-on" practical experience and
explicit reflection on the collaboration process. Experiences can be transferred back to
their own schools, and ideas about collaboration can be integrated into their everyday
teaching.
Four different collaborative telelearning scenarios are studied. The first scenario,
PedInfo, is a pilot study for analysing the use of TeamWave Workplace (TW,
http://www.teamwave.com) for collaborative activities in a graduate university course at
the University of Bergen (UiB). The next two scenarios, IDEELS and Demeter, involve
European inter-cultural simulations where the goal is to design a textual artefact (such as
a treaty or policy statement). A fourth scenario, VisArt, which has been designed and
developed explicitly for use between the three educational partners (UiB, HiNT and
Stord), has the goal of designing a visual artefact to be used in teaching a subject of
choice. In the remainder of this paper we focus on the VisArt scenario.
Theoretical Foundations
The theoretical perspectives that provide inspiration and guidance for this research come
from: computer supported collaborative learning (CSCL), in particular Salomon’s (1992)
work on genuine interdependence; computer supported collaborative work (CSCW), in
particular Gutwin et al.’s (1995) ideas on awareness; coordination science (Malone &
Crowston, 1994); sociocultural perspectives (Wertsch, del Río & Alvarez, 1995) on
learning and thinking; and, the emerging notion of distributed learning communities.
Computer supported collaborative learning (CSCL) is an emerging paradigm
(Koschmann, 1996) for research in educational technology that focuses on the use of
information and communications technology (ICT) as a mediational tool within
collaborative methods (e.g., peer learning and tutoring, reciprocal teaching, project- or
problem-based learning, simulations, games) of learning. It is an approach to ICT in
education that emphasises an understanding of language, culture and other aspects of the
social setting (Scott, Cole & Engel, 1992). Its intellectual heritage can be found in social
constructivism (Doise, 1990), the Soviet cultural-historical psychology (e.g., Vygotsky
(1978), Leontiev (1978), Davydov (1988)) and situated cognition (Suchman, 1987; Lave,
1988).
CSCL research involves naturalistic observations being used in an exploratory fashion to
permit a more complete understanding of this instructional mode. The emphasis is on the
process and not so much focused on outcome. Evaluations often result in descriptive
studies which focus on artefacts that support or are produced by teams of learners and
usually contain participant accounts of their own work. The influence of CSCL research
on DoCTA is evident in the type of research questions we ask and in the choice of
conceptual framework for organising our evaluations.
Guribye & Wasson (1999) describe the underlying conceptual framework adopted in
DoCTA as an integration of three different, although closely interrelated approaches:
activity theory (Leontev, 1978, Engeström 1987), distributed cognition (Hutchins, 1995),
and situated action (Suchman, 1987, Lave, 1988, Mantovani, 1996). All three approaches
underscore the need to look at real activities in real situations (Nardi, 1996, our italics),
and always, in some way, include the context in studies of human activity. One of the
goals of this research is to argue that, together, these approaches make up a rich
framework for describing, evaluating and analysing collaborative telelearning scenarios.
Salomon (1992) is concerned with effective collaboration and he argues that
collaboration will only be effective if there is genuine interdependence between the
collaborating students. Genuine interdependence is described as 1) the necessity to share
information, meanings conceptions and conclusion, 2) the necessity for division of labour
into complementary roles, and 3) the need for joint thinking in explicit terms. In project
DoCTA, Salomon’s ideas influence the design of the collaborative tasks given to the
students — effort is placed in designing tasks that create genuine interdependence
between the students.
Coordination theory (Malone & Crowston, 1994) provides a means for specifying
(inter)dependencies between, and among, actors, goals, activities, and resources by
identifying a dependency type (e.g., shared resource) and a coordination process (e.g.,
group decision-making) for managing the dependency. In their work, coordination is
defined as managing dependencies between activities, hence they have focused on
dependence between activities. Wasson & Bourdeau (1997) report that viewing
collaborative telelearning from a coordination theory perspective offers a means of
understanding the inter-relationships between actors and entities and how these
relationships can and should be supported. Adopting Salomon’s ideas about genuine
interdependence and a coordination science approach, they modelled (inter)dependencies
between actors in collaborative telelearning scenarios and they have extended the
definition of coordination to be managing dependencies between activities (Malone &
Crowston, 1994) and supporting (inter)dependencies between actors. Wasson (1997,
1998) proposes a set of actor (inter)dependencies and related coordination processes for
collaborative telelearning. The importance of this work for DoCTA lies in making sure
the technological environment within which the students carry out their tasks provides
mechanisms that make coordination as effortless as possible. Otherwise coordination
issues can become a bottleneck that hinders students from carrying out their individual
and collaborative work.
Gutwin et al. (1995) present a framework of awareness for collaborative learning which
comprises four types of awareness: social awareness, task awareness, concept awareness
and workspace awareness. Social awareness refers to awareness of the social connections
within the group or team (e.g., What role will I take in the team, what roles will the other
members take). Task awareness concerns awareness about how to complete the common
task (e.g., What do we know about the task, how much time do we have, what steps must
we take to complete the task). With concept awareness the individual student is aware of
how a particular activity or piece of knowledge fits into his or her own existing
knowledge. Workspace awareness is an up-to-the-minute knowledge of the other
students’ interactions with the workspace (e.g., what are they doing, where are they, what
has been done so far, what is left to do). They suggest that social awareness, being interpersonal, is best supported implicitly by providing communication opportunities for team
members to negotiate their roles. Task and concept awareness are often supported by
providing explicit scaffolds to assist with organisation and helping stay on task.
Workplace awareness is maintained by providing "tracking information such as other
learners’ location in the shared workspace, their actions, the interaction history and their
intentions (Gutwin et al., 1995, p. 147).
The final area providing inspiration is summarised as follows "…a distributed
collaborative learning community is a ‘place’ that is created by the individual students
through their individual and collective actions, …The designers’ role is to support the
students’ work of creating that community, and in such a way that the computer systems
become integrated parts of the students’ activity (Fjuk, 1998, p. 70)". Furthermore, Fjuk
(1995) concludes that collaborative telelearning applications need to have both a
mediating role between the individual learner and the peer-students and between the
individual learner and her learning tasks. Thus, collaborative telelearning can be
understood as a medium for inter-human interactions and articulation of individual work.
As designers of DoCTA learning tasks and the technological support environments, we
must keep in mind that our role is to provide a supporting environment that makes
coordination, communication and collaboration as transparent as possible (Bourdeau &
Wasson, 1997) enabling students to create their own learning community. The
environment must also support both individual and collaborative work.
VisArt
The VisArt scenario involved students taking courses at three educational institutions
(University of Bergen, Høgskolen I Nord-Trøndelag, Stord/Haugesund Høgskolen) in
Norway. Teams comprised of 3 students, 1 student from each institute, collaborated to
design a learning activity in TW. There were no opportunities for the teams to meet faceto-face. TeamWaveWorkplace (TW) was used as the main information and
communication technology. The VisArt activity took place during February and March
1999 and provided an opportunity for the DoCTA project to study an authentic
collaborative telelearning activity. One week of training in using the TW tool and in
collaboration proceeded three weeks of design activity. Figures 1 and 2 give examples of
the TW working room and designed learning room designed by Team07.
Participation at the University of Bergen
The University of Bergen students participated in VisArt through the pedagogical
information science course given in the Spring of 1999. The first semester assignment on
collaborative telelearning involved participating in the VisArt scenario. Participation
alone, however, was not enough. Since this is a graduate course and two of the topics the
students were learning about were collaborative learning and telelearning, the VisArt
scenario provided an opportunity for both participation in collaborative telelearning and
an opportunity for theoretical reflection. A semester assignment was designed to cover
both opportunities.
The students were given literature on CSCL (including Salomon (1992) & Gutwin et al.
(1995)) and were asked to organise a team effort thinking about Salomon’s definition of
genuine interdependence (sharing information, division of labour, and joint thinking).
They were to participate in the VisArt design activity for 1 month and then were to write
an individual report on the experience.
Figure 1 Team 7’s working room in TeamWave
Figure 2 Team 7’s room for learning about polar bears
Participant’s Profile
The learner’s participating in the scenario had different backgrounds, ranged in age from
23 to 68 years and many had family responsibilities and full time jobs. The University of
Bergen students were taking a graduate course in pedagogical information science and
were learning about collaborative telelearning. They were a blend of pedagogical
information science graduate students (with a teacher’s background) and information
science graduate students (with a social science background). The students at Stord
Teacher’s College, were senior undergraduate students training to be teachers who were
taking a distance learning course on pedagogical information science and were also
learning about collaborative learning. Finally, the Nord-Trøndelag students were taking
an undergraduate introduction course to uses of technology in learning and had a choice
to participate in the scenario to learn about a telelearning application.
Evaluation of VisArt
The evaluation of the VisArt scenario is being carried out on several levels and from
several perspectives. In addition to the student’s own theoretical reflections (described in
the next section), VisArt is being evaluated as part of the DoCTA project. Eight Master’s
theses are being carried out. These theses include usability studies of TW, looking at the
efficiency of TW from a qualitative perspective using the data logs generated by TW,
performing a formative evaluation of how to support collaborative design activities,
seeing how TW supports coordination, how to design training and assistance in a
collaborative telelearning setting, and several activity theory studies of how students,
instructors and facilitators organise their work. Table 1 summarises the research
questions of the 8 graduate students. Numerous data and data collection techniques have
been used including questionnaires, semi-structured interviews, log files produced
automatically by TW, email send among team members, chat files saved by the teams,
documents and artefacts produced by the teams, participant observations, video taping of
activities, self-evaluations and the participants own interpretation of the experience.
The learners’ experiences
A summary of the learners’ experiences as described in their individual reports is given in
the following. In particular, attention is paid to how their group met Salomon’s criteria
for genuine interdependence and how TW supported Gutwin’s ideas on awareness.
In general, the students were very satisfied with TW. As one student writes
"An important side with TeamWave is that one can work both
asynchronously and synchronously. … For example one can use the
shared whiteboard synchronously when the users are online at the same
time and write on it together, but it is also possible to use the whiteboard
asynchoronously when the different users log on at different times and
work individually on tasks on the whiteboard. …That it supports both
forms of work makes the program package flexible and accessible at all
times."
Several students wrote that the successful use of TW was not just tied to the ease of use,
rather, that it is used in an activity that meets Salomon’s requirements. As one student
succinctly put it
"I think that a requirement for successful use of TW is that the participants
are motivated and have mindful engagement and that the tool is used for
something meaningful."
Table 1 Research Questions for Master’s theses related to the VisArt scenario
Domain/theoretical
framework
Research Question
Usability of TeamWave Workplace
Usability — effectiveness
Can students working in collaboration reach their team goals with
TW?
Usability — efficiency
To what extent do the tools enable students to meet their task?
Usability - satisfaction
Are students satisfied of TW tools?
Efficiency from a qualitative perspective
Comparison of anticipated use and
actual use of the tools
Research methodology
What are the differences between the anticipated use of the tools
and the actual use of the tools? Which are the differences of tool
use when alone and when others in the room? How are the rooms
used?
What implications do the use of the tools have for electronic datacollection?
Formative evaluation
Tailoring for instructional design
How to support pedagogical room design in TW? How do students
use TW tools to design a room for teaching?
User centred design
What is to be improved?
Coordination theory
Supporting coordination
How to support coordination in a collaborative telelearning
environment?
User centred design
What kind of coordination do students do?
Activity theory
Organisation of work — student
perspective
How do the students organise their work?
How instructors and facilitators organise their work
Organisation of work — Instructor`s
perspective
Design of training and
assistance
Assistance design/ improvement
How effective is the training designed for collaborating and
learning to use TW? How to improve help and assistance?
Sharing of information
The majority of the groups had a heterogeneous makeup with the group members having
different backgrounds. As one group said, this meant that they had different
preconceptions and different experiences with collaboration. They said, that
"according to Salomon it is exactly these differences that makes
collaboration work…to use each others competence and pull something
useful of these competencies through collaboration."
When it came to TW supporting the sharing of information, it was possible to do so both
synchronously and asynchronously. One particular group worked mostly synchronously
(with respect to sharing and they meant this was time-consuming. Often they would meet
at a pre-arranged time and have little to say to one another and this was a disadvantage as
it required that they bind themselves to meeting at a particular time. It would have been
"better if we could have worked a little bit more asynchronously, and
insead given each other feedback, for example by using a PostIt, when we
were logged on to work with one of our own tasks. TW provides several
tools to support this asynchoronous communication…but our use of this
functioned badly".information)
Division of labor
With regard to division of labor, one of the students said the following:
"TW was a good support for division of labour in those cases where it was
not necessary to be in our common work area at the same time. If one
person started on a task, then left the work area, the next person could
come in an continue working on the task since there was a common work
area."
Joint thinking
One of the teams said that the requirement for joint thinking in explicit terms that can be
examined, changed, and elaborated upon by peers was supported in TW through the Chat,
Postit and Brainstorm tools. For them
"This particular need was satisfied through the many online joint
discussions between team members. The fact that team members had the
opportunity to express their ideas, conceptions, meanings, etc., in clear
and observable terms (by means of Postit and Chat artifacts), enables each
one of us to analyze, expand on, or change these as they were being
developed."
Awareness
The students were in general agreement that TW supported many of the aspects of
awareness as identified by Gutwin et al. (1995). As one student points out, it is not
TeamWave Workplace alone that supported social awareness, but the combination with
some of the training activities that were designed for learning about TW and about each
other that supported social awareness. He says
"The social awareness tied to expected roles is first and foremost tied to
personal relationships. I feel that the training activities we carried out in
TeamWave were important for this. Through informal chats we were able
to each get a feeling of what we could expect of each other and what type
of interaction and tone we would have between us. … It was never
explicitly said that "I am … and you are …", but over time we picked up
aspects of each others personaility and found our own roles."
According to the students TW supported workspace awareness by several techniques.
The Information Areas provided awareness about the other users presently connected to
the server, and indicated who was in the same room. The coloured cursor on the white
board made it easy to follow what each of the other users who are in the same room are
doing, and if another user is typing on the Whiteboard, PostIt, or Brainstormer, you see
what they are typing immediately. One of the other students meant that TW’s Chat tool
"provides for some of the social aspects of collaboration. We used it for
informal discussions around different themes, both scientific and social.
Through the common mileu that a room in TeamWave enables, both task
and conceptual awareness are fulfilled."
Another student said that
"The opportunities provided by TeamWave Workplace in dealing with the
issues in social awareness were generally poor. Support for interactivity in
information exchange and negotiation of roles, which is fundamental
given the interpersonal nature of social awareness, was limited to artifacts
such as Chat and Postit…In the case of concept and task awareness, the
requirement within CSCL environments is for explicit artifacts that can be
used as scaffolds to help students with task organisation, providde them
with concentration aids and step-by-step guidelines for learning tasks.
TeamWave’s support for both concept and task awareness is provided by
tools such as ToDoList and Concept Map (for task organisation, and
Message Board (For structured message capabilities)."
Conclusions
It is early in the evaluation of VisArt to make many conclusions. From a sociocultural
research perspective, however, the student’s own reflections are a very important part of
evaluation and as illustrated in the previous sections they demonstrated an ability to
reflect theoretically on practice. The research reports they submitted in the course
contained comments and reflections that were both thoughtful and insightful and will lead
to improvements in future versions of the scenario. In general our first impressions of our
research data indicate that the students were positive to both the distributed collaboration
and to the groupware tool TeamWave Workplace. Results from the Master’s theses
evaluations will be available from late 1999 and at that point more concrete conclusions
will be made.
Acknowledgments
DoCTA is funded by The Norwegian Ministry of Education, Research and Church
Affairs (KUF) under their Information Technology in Education (ITU) programme. The
author would like to thank all those involved in the VisArt scenario: Anders Mørch,
Arnstein Eidsmo, Glenn Munkevold, Lars Vavik, Eskil Andreassen, Rune Baggetun,
Frode Guribye, Hege Higraff, Øyvind Meistad, Trond Pedersen, Kurt Rysjedal, Helge
Underhaug, and Jo Wake.
Bibliography
Bourdeau, J. & Wasson, B. (1997). Orchestrating collaboration in collaborative
telelearning. In B. du Boulay & R. Mizoguchi (Eds.) Proceedings of the 8th World
Conference on Artificial Intelligence in Education, 565-567. Amsterdam: IOS Press.
Davydov, V. (1988). Learning activity: The main problems needing further research.
Activity Theory, 1(1-2), 29-36.
Doise, W. 1990. The development of individual competencies through social interaction.
In H.C. Foot, M.J. Morgan & R.H. Shute (Eds.) Children helping children. Chichester: J.
Wiley and Sons. 43-64.
Engeström, Y (1987). Learning By Expanding: An activity-theoretical approach to
developmental research. Helsinki: Orienta-Konsultit Oy
Fjuk, A. (1995).Towards an Analytical Framework for CSCdistanceL. In J.L. Schnase &
E.L. Cunnius (Eds.) Proceedings of CSCL'95, 130-134. Mahwah, NJ: Lawrence Erlbaum
Associates.
Fjuk, A. (1998). Computer support for distributed collaborative learning. Ph.D thesis,
Dept. of Informatics, University of Oslo.
Guribye, F. & Wasson, B. (1999). Evaluating collaborative telelearning scenarios: A
sociocultural perspective. To appear at EdMedia’99.
Hutchins, E. (1995).Cognition in the Wild. Cambridge, MA: MIT Press.
Koschmann, T. (1996). Paradigm shifts and instructional technology: An introduction. In
T. Koschmann (Ed.) CSCL: Theory and practice of an emerging paradigm, 1-23.
Mahwah, NJ: Lawrence Erlbaum Associates.
Lave, J. (1988). Cognition in practice. Cambridge: Cambridge University Press.
Leont’ev, A. N. (1978).Activity, Consciousness, Personality. Englewood Cliffs,NJ:
Prentice Hall.
Malone, T. & Crowston, K. (1994). The Interdisciplinary study of coordination. ACM
Computing Surveys, 26(1), 87-119.
Mantovani, G (1996).New Communication Environments: From Everyday to Virtual.
London:Taylor & Francis Ltd.
Nardi, B. A. (1996).Studying Context: A comparison of activity theory, situated action
models and distributed cognition. In Nardi, B. (Ed.) Context and Consciousness: Activity
Theory and Human-computer Interaction. Cambridge, MA: MIT Press
Salomon, G. (1992). What does the design of effective CSCL require and how do we
study its effects? SIGCUE Outlook, Special Issue on CSCL, 21(3), 62-68.
Suchman, L. (1987). Plans and situated action. The problems of human-machine
communication. Cambridge: Cambridge University Press.
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological
processes. (M. Cole, V. John-Steiner, S. Scribner & E. Souberman, (Eds. and
Translators). Cambridge, MA: Harvard University Press.
Wasson, B. (1997). Developing a model of (inter)dependencies for collaborative
telelearning. LICEF Research Report, Télé-université, Montreal, Canada.
Wasson, B. (1998). Identifying Coordination Agents for Collaborative Telelearning,
International Journal of Artificial Intelligence in Education, 9, 275-299.
Wasson, B. & Bourdeau, J. (1998). Modelling actor (inter)dependence in collaborative
telelearning. Full paper to appear in Proceedings of Ed-Media’98, Freiburg, Germany.
Wertsch, J. V., del Río, P. & Alvarez, A. (1995).Sociocultural studies: history, action and
mediation. In Wertsch, J. V., del Río, P. & Alvarez, A. Sociocultural Studies of Mind.
Cambridge University Press.
Author’s address
Barbara Wasson (barbara.wasson@ifi.uib.no)
Department of Information Science; University of Bergen; P.O. Box 7800; N-5020
Bergen, Norway. Tel. +47 55 58 41 20. Fax +47 55 58 4107.