Conferencia Educación, Cognición, TIC, S PDF

IADIS International Conference on Cognition and Exploratory Learning in
Digital Age 2012

The IADIS International Conference on Cognition and Exploratory Learning in Digital Age was held in
Madrid, Spain, October, 19- 21, 2012.
The IADIS CELDA 2012 Conference intention was to address the main issues concerned with evolving
learning processes and supporting pedagogies and applications in the digital age. There had been
advances in both cognitive psychology and computing that have affected the educational arena. The
convergence of these two disciplines is increasing at a fast pace and affecting academia and professional
practice in many ways. Paradigms such as just-in-time learning, constructivism, student-centered learning
and collaborative approaches have emerged and are being supported by technological advancements
such as simulations, virtual reality and multi-agents systems. These developments have created both
opportunities and areas of serious concerns. This conference aimed to cover both technological as well as
pedagogical issues related to these developments. Main tracks have been identified. However innovative
contributions that do not easily fit into these areas will also are considered as long as they are directly
related to the overall theme of the conference – cognition and exploratory learning in the digital age.
The main tracks for submissions were: Acquisition of expertise, Assessing progress of learning in complex
domains, Assessment of exploratory learning approaches, Assessment of exploratory technologies,
Cognition in education, Collaborative learning, Educational psychology, Exploratory technologies (such as
simulations, VR, i-TV and so on), Just-in-time and Learning-on-Demand, Learner Communities and Peer-
Support, Learning Communities & Web Service Technologies, Pedagogical Issues Related with Learning
Objects, Learning Paradigms in Academia, Learning Paradigms in Corporate Sector, Life-long Learning,
Student-Centered Learning, Technology and mental models, Technology, learning and expertise and
Virtual University.
The IADIS CELDA 2012 Conference received 98 submissions from more than 24 countries. Each
submission was reviewed in a double-blind review process by at least two independent reviewers to
ensure quality and maintain high standards. Out of the papers submitted, 29 were accepted as full papers
for an acceptance rate of 30%; 17 were accepted as short papers and 3 were accepted as reflection
papers. Authors of the best published papers in the CELDA 2011 proceedings will be invited to publish
extended versions of their papers in a special issue of Computers in Human Behavior (an Elsevier journal,
ISSN: 0747-5632), and in an edited volume to be published by Springer.
In addition to the presentation of full papers, short papers and reflection papers, the conference also
includes a keynote presentation from internationally distinguished researchers. We would therefore like to
express our gratitude to Professor Norbert M. Seel, Department of Educational Science, University of
Freiburg, Germany, as the CELDA 2012 keynote speaker.
Keynote Lectures:
K1 - THE THREE LEARNING SCIENCES (BIOLOGICAL, ARTIFICIAL, HUMAN) by Professor Norbert M.

Seel, Department of Educational Science, University of Freiburg, Germany
Abstract
Learning is existential, and so its study must be complex and interdisciplinary. Over the past centuries,
researchers from different fields have developed many theories to explain how humans and animals learn
and behave, i.e., how they acquire, organize, and deploy knowledge and skills. Basically, learning is
defined as a relatively permanent change in behavior and/or in mental associations due to specific
experiences. Learning is a response to environmental requirements and different from biological
maturation, which, however, is a fundamental basis for learning. Beyond psychology and biology,
disciplines such as anthropology, sociology, and education focused on the topic of human learning in the
course of the past centuries. However, one of the most important innovations for research on learning
resulted from the emerging computer sciences and their focus on machine learning. Machine learning
usuallyrefers to changes in systems that perform tasks associated with artificial intelligence (AI). Many
techniques in machine learning are derived from the efforts of psychologists to make their theories of
animal and humanlearning more precise through computational models. Conversely, it seems that the
concepts and techniques being explored in the field of machine learning also illuminate certain aspects of
the biology of learning. Accordingly, closely related to machine learning is also the study of human and
animal learning in psychology, neuroscience, and related fields. In my presentation I will focus on the
biological foundations of learning, mainly discussed in terms of the interplay between assimilation and
accommodation that correspond the basic functions of schemas and mental models. Second, I will focus
on cumulative or incremental learning as an example to demonstrate basic correspondences between
theories of human and artificial learning. Finally, I will discuss practical implications for interdisciplinary
research on human and artificial learning.
The Proceedings were both published in Book (ISBN: 978-989-8533-12-8) and CD-ROM (ISBN: 978-989-
8533-13-5) by IADIS Press.
Program Committee:
Conference Chair:
- Pedro Isaías, Universidade Aberta (Portuguese Open University), Portugal
Program Co-Chairs:
- Demetrios G Sampson, University of Piraeus and CERTH, Greece

- J. Michael Spector, Florida State University, USA
- Dirk Ifenthaler, Albert-Ludwigs-Universität Freiburg, Germany
Committee Members:
Ah-choo Koo, Multimedia Univerisity, Malaysia

Alke Martens, University of Education Schwaebisch Gmund, Germany
Andreas Holzinger, Medical University Graz, Austria
Anthony Y. H. Liao, Asia University, Taiwan
Aparna Lalingkar, International Institute Of Information Technology, India
Ari Korhonen, Aalto University School of Science and Technology, Finland
Arif Altun, Hacettepe University, Turkey
Baltasar Fernandez, Universidad Complutense De Madrid, Spain
Barbara Lockee, Virginiatech, Usa
Carla Falsetti, University Politecnica Delle Marche, Italy
Carol H.C Chu, Soochow University, Taiwan
ChanMin Kim, The University Of Georgia, USA
Charalampos Karagiannidis, University of Thessaly, Greece
Charoula Angeli-valanides, University Of Cyprus, Cyprus
Chen-chung Liu, National Central University,, Taiwan
Chien Sing Lee, Multimedia University, Malaysia
Chien Ching Lee, Nanyang Technological University, Singapore
Chih-hung Lai, National Dong Hwa University, Taiwan
Christophe Choquet, University Of Maine, France
Christophe Despres, University Of Maine,, France
Chung Hsien Lan, Toayuan Innovation Institute Of Technology,, Taiwan
Chun-Wang Wei, Far East University, Taiwan
Clark Quinn , Quinnovation.com, Usa
David Jonassen, University of Missouri, USA
Davinia Hernandez-leo, Universitat Pompeu Fabra, Spain
Dirk Ifenthaler, University of Mannheim, Germany
Dragan Gasevic, Athabasca University, Canada
Eshaa Alkhalifa, Royal University for Women, Bahrain
Eva Grundl, University Of Mannheim, Germany
Giuliana Dettori, National Research Council, Italy
Giuliana Guazzaroni, University Politecnica Delle Marche, Italy
Guangbing Yang, Athabasca University, Canada
Huei-tse Hou, National Taiwan University Of Science And Technology, Taiwan
I-fang Liu, National Central University, Taiwan
Ignacio Aedo, Universidad Carlos Iii De Madrid, Spain
Imran Zualkernan, American University Of Sharjah, United Arab Emirates
Jan Van Bruggen, Open University Of The Netherlands, Netherlands
Janet Tworek, University Of Colorado Anschutz, United States
Jarkko Suhonen, University of Eeastern Finland, Finland
Javier Melero, Universitat Pompeu Fabra, Spain
Javier Torrente Vigil, Universidad Complutense De Madrid, Spain
Jay Loftus, Schulich School of Medicine & Dentistry , Canada
Jie-Chi Yang, National Central University, Taiwan
Jirarat Sitthiworachart, Walailak University, Thailand
John Loewen, Vancouver Island University, Canada
Jon Dron, Athabasca University, Canada
Jonathan Mckeown, University of Tampa, United States
Jose Janssen, Open University Of The Netherlands, Netherlands
Joyce Pittman, Drexel University, Usa
Juan Felipe Calderon, Universidad Catolica De Chile, Chile
K. Robert Lai, Yuan Ze University, Taiwan
Katherine Sinitsa, Int. Res. Training Center For Info. Technologies A, Ukraine
Kathrin Figl, WU - Vienna University of Economics and Business, Austria
Kuo-Liang Ou, National Hsin-chu University Of Education, Taiwan
Kyparissia Papanikolaou, School Of Pedagogical And Technological Education, Greece
Lence Miloseva, Goce Delcev University , Macedonia
Luis Anido, University Of Vigo, Spain
Lynne Schrum, George Mason University, Usa
Maiga Chang, Athabasca University, Canada
Michael Verhaart, Eastern Institute Of Technology,, New Zealand
Mike Joy, University Of Warwick, United Kingdom
Ming-chi Liu, National Cheng Kung University, Taiwan
Ming-puu Chen, National Taiwan Normal University, Taiwan
Minjeong Kim, Dankook University, Korea, Republic Of
Minkyu Kim, UGA, United States
Monica Divitini, Norwegian University Of Science And Technology, Norway
Monica Landoni, University of Lugano, Switzerland
Natalia Padilla Zea, Universidad De Granada, Spain
Ni Chang, Indiana University South Bend, United States
Norlaila Hussainn, Massey University, New Zealand
Patricia Santos, Universitat Pompeu Fabra, Spain
Peter Sloep, Open University Of The Netherlands, Netherlands
Radu Vasiu, Politehnica University of Timisoara, Romania
Richard Huntrods, Athabasca University, Canada
Roger Hartley, Leeds University, United Kingdom
Russell Johnson, Massey University, New Zealand
Sabine Graf, Athabasca University, Canada
Sahana Murthy, IIT Bombay, India
Sanghoon Park, Northwestern State University, USA
Sarfo K. Frederick, University Of Education, Winneba. College Of Techn, Ghana
Sebastien Iksal, University Of Maine, France
Stephen Corich, Eastern Institute Of Technology, New Zealand
Steve Leung, Athabasca University, Canada
Su Luan Wong, Universiti Putra Malaysia, Malaysia
Takashi Onoyama, Hitachi Solutions,Ltd., Japan
Teoh Sian-hoon, Uitm Shah Alam, Malaysia
Thomas Lehmann, University Of Freiburg, Germany
Tiong Goh, Victoria University Of Wellington, New Zealand
Tobias Schmidt, University Of Freiburg, Germany
Tristan Johnson, Northeastern University, USA
Tzu-chien Liu, National Central University, Taiwan
Veronica Rossano, University Of Bari, Italy
Vincent Barre, University of Maine, France
Vive Kumar, Athabasca University, Canada
Yen-ting Lin, National Taiwan Normal University , Taiwan
Yi-chuan Jane Hsieh, Ching Yun University, Taiwan
Yoshitaka Sakurai, Tokyo Denki University, Japan
Yueh Min Huang, National Cheng-kung University, Taiwan
Yu-ju Lan, National Taiwan Normal University, Taiwan
ADDITIONAL REVIEWERS
Moushir El-Bishouty, Athabasca University, Canada
Ting-Wen Chang, Athabasca University, Canada
Richard Tortorella, Athabasca University, Canada
IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012)
REDEFINING PUBLIC POLICY EDUCATION THROUGH

AN EXPLORATORY DIGITAL CURRICULUM:
SINGAPORE’S STATECRAFT X
Ming Hui Shoon and Yam San Chee

National Institute of Education, Nanyang Technological University
1 Nanyang Walk, Singapore 637616
ABSTRACT
Given the current international context of instability and uncertainty, we were driven by the desire to utilize a digital
game to cut across the complexity of public policy, so as to educate our young with the experience and deep learning to
be appreciative, accountable and proactive citizens of a globalized world. Having developed a curriculum that synthesizes
technology, philosophy and pedagogy, we began our iterations of exploration with Singapore public schools. Through
our project, we wanted to share our idea that the learning of public policy, in the form of Singapore’s Social Studies
curriculum, was disadvantaged with the traditional, transmission-oriented mode of education, and had to be experienced
and performed. Data from the impactful dialogic sessions conducted showed that students experienced realistically the
problems faced in public policy decision making, and gained a sense of what governance, learnt performatively entailed.
KEYWORDS
Digital Games, Public Policy, Dialogic Learning, Performance, Citizenship, Governance
1. INTRODUCTION
We live today in a world beset with international complexity and danger. A panorama of the past five years
of international affairs yields the instability of governmental systems in some regions of the world. Regions
are affected not only by economic woes of excess, ignorance and the lack of political compromise, but also,
the uprising of the human spirit for freedom and transparency, spreading across the vastness of continents.
Human civilization has never been more interconnected, and has not ever had such a chance for global
influence, if not for the speed and development of technology. This is especially so in areas like mobile
communications and social media, avenues favored among the young, globalized citizens of a new
prospective world order. Governments trying to harness this relentless technology in development to better
manage their states would realize that this would not be viable without addressing the new needs of education
for the young, in the area of public policy, through a digital game.
With the trans-continental mobility of people, ideas and digital influences, the young, globalized citizens
of this new world order, spurred by the ability to communicate their aspirations through social media, would
not want to be bystanders in the shaping of their countries’ futures. Globalization itself serves as an agent for
obligatory education reform, favoring curricula supporting global interdependence (Hershock, Mason &
Hawkins, 2007). Hence, more so in Asia than in other regions, to facilitate their participatory rigor,
educational development in areas like the Social Sciences might gradually shift to being a student-centered
endeavor, aided by a means to engage in social learning. This should be the case specific at the level of
Secondary education – the ages of 13 to 16, and involving the common subject long used to ‘teach’ concepts
of governance, Social Studies, an amalgamation of History, Geography, Political Science, Economics and
Sociology (Ministry of Education, 2012).
3
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. TECHNOLOGY’S ROLE IN 21ST CENTURY PUBLIC POLICY

EDUCATION
In a human resource dependent nation-state like Singapore, a hub connecting Western and Eastern interests,
every learned student matters in his or her contribution towards nation building. As Singapore moves to
become a society of further inclusiveness, the citizenry has to be educated away from the transmission-based
mode of learning. This is strikingly so in the subject of Social Studies, taking into consideration Social
Studies’ limitations in simply telling the students about the ‘facts’ of governance without assenting for
performance. Allowing for performance would bridge the gap between the theories and principles of
governance, and the actuality of enacting governance, enabling in the students a deeper plane of
understanding and awareness.
In this paper, we share our solution to addressing the needs of a student-centered, digital Social Studies
curriculum so to as encourage a forward-looking vision of preparing the young of Singapore as global
citizenry in their responses to public policy initiatives, both local and internationalized. We will discuss how
Statecraft X, a mobile game-based learning curriculum, represents our contextualized ideals of necessity and
philosophy of education. Following this, by examining selective instances, namely the utterances of the
students during the dialogic sessions, we will explain how Statecraft X provided the impetus for students to
think and enact public policy without having first learnt the ‘factual content’ found traditionally in their
Social Studies textbooks, and how this led to their incremental understanding of what governance entails.
2.1 Literature Review

Much of our motivation and desire to engage in the Statecraft X project was contingent with our
understanding of a variety of literature on learning. Video games, or more appropriately in today’s mobile
context, digital games, have immense potential as learning tools. Games are multidimensional, challenging
and are designed to allow for learning to take place. They alter the belief that thinking is an activity that takes
place in solitude and instead, identifies how the combination of emotion and pleasure derived from play are
key to thinking and learning (Gee, 2007).
In developing as well as developed societies that prize inclusiveness, approaches to learning have to
factor in the growth of creative differences in individuals. This must be supported by the need for experience
in environments, in the Deweyan sense of educational pragmatism, where consequence, action and thinking
in learning come together without a mechanical, predictable end result (Elkjaer, 2009). To the extent that this
is so, learning cannot and should not be regarded as an individual activity, and must take place in social
contexts, taking in consideration that constructs of what we know, and how we make meanings of them are
situated in communities (Wenger, 2009).
Learning should not take place with an emphasis on the capacity of the students’ capabilities, but rather,
the action being enacted, so as to ensure learning through performance, which is reflexive in nature
(Baumann, 1989). A performance oriented model of learning is advantaged when compared to the human
information processing model, which regards learners largely as storage machines, as performance leads to
the creation of new meanings and understandings of existential occurrences only possible by a biological
being (Chee, in press). That, in these perspectives explained, is why Statecraft X accentuates digital contexts
and socialized learning with opportunities for performance.
2.2 Game-Based Learning through Statecraft X

Technology knows no boundaries, and in an ever-evolving educational landscape, the digital divide would
render students who are not allowed to learn together with technological savvy in a much disadvantaged
position. Statecraft X, being an innovative game-based learning endeavor, bridges this gap. It provides not
only a platform, but a digital realm, a “magic circle,” well insulated from the distractions of the real world
(Nielsen, Smith and Tosca, 2008) to allow for students to conduct personal, yet socially connected
experiments on performing governance. The game is run on Apple iPhones as an installed application. The
portability and island-wide availability of 3G, as well as Wi-Fi connections enhances the mobile playability
of Statecraft X. Functionally, the game is akin to a miniaturized version of a Massive Multiplayer Online
4
Role-Playing Game or MMORPG for twenty plus players, where game play is only possible with teams of
participants, and players navigate through town, world map and specific in-game action menus on their
iPhone screens, like the town map shown in Figure 1.
Figure 1. The town map where players have access to buildings and their functions.
Figure 2. The world map where a player has entered another town and has a range of options.
Statecraft X is set in the fantasy kingdom of Velar, which stands without a ruler after the death of a wise
and capable king. Over the course of three weeks, students play in groups of four or five, where they
represent factions fighting for control, influence and power over Velar, and to control the capital city
Topezios. The four factions are Dragon, Pegasus, Phoenix and Griffin. Within each faction, each player starts
off as a governor of a town. Statecraft is innovative in giving each player the option to vary and manipulate
their town’s development pertaining to issues like housing, healthcare, defense and the acquisition and
production of resources. Adding on to this flexibility is the complexity of adjustable tax rates, workers’
wages and even limitations on the skill level and trainability of workers and citizens who have decided to
make a particular player’s town home, seen in Figure 3.
5
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 3. An in-game menu where players modify immigration policy.

The social aspect of control is also not forsaken as players have a say over the holding of ideological
rallies and beneficial multicultural events. Beyond the governorship of each individual town, players, at both
a personal and factional level, have to make constant decisions on the organizing of diplomacy, conflict,
influence and control over artificial intelligence controlled, and also, player controlled towns. The governors
of the towns in Velar also meet with the random and sometimes multiple event occurrences of bandit attacks,
epidemics and finally, towards the end of the game, an onslaught by a neighboring country, Salfreda, which
tries to seize control of the entire kingdom. The random events that occur add to the challenge of putting in
place strategic and decisive policies, and are reflective of the unpredictable nature of real world governance.
At all phases of the game, students’ progress in their factions is captured daily in the form of economic and
happiness graphs, reminding the students of the realities of having to balance societal well-being with
economic prosperity (Chee, Gwee & Tan, 2011).
2.3 A Curriculum for Digital Empowerment

Philosophically, Statecraft X’s curriculum design was influenced by the strength of educational and social
theories. The learning of public policy and governance cannot take place through forced, textually limiting
transmission-based materials like textbooks as the process of social science education is lost without a form
of unhindered development, such as that put forth in Dewey’s (Dewey, 1916; 1980) exposition on naturalism.
Essentially, public policy, in the sense of democratic education, is composed of decisional input from a
learned mass of citizenry, and would naturally involve conflicts of inquiry. Hence, Statecraft X allows for
students to replicate these conflicts of decisions, and the responsibility over the lives of their townspeople in
the game, which would lead to a deeper and more inclusive learning experience (Dewey, 1938; 2008). A
platform with performative potential is incomplete without a binding method for students to negotiate the
meanings of governance they have just put in practice. Thus, the Statecraft X curriculum gives students the
opportunity to engage in dialogic sessions, where meaningful, context shaped utterances in the form of inter-
student dialogues with one another, facilitated by the teacher, embody the backward and forward creation of
the renewed understanding of governance in practice (Bakhtin, 1981).
Pedagogically, the game is driven by the Performance-Play-Dialog model of game-based learning (Chee,
2007; 2011), which emphasizes the feedback loop where the performance of governance is enjoyed as a form
of play – a naturalistic development thought to originate in one’s first experience with learning, further
developed through meaningful dialog, and repeated as beneficial practice. The students are encouraged to be
inquisitive and experimental in their game-based performances, and to vocalize their thoughts in teacher-
facilitated dialogic sessions which complement the digital platform. Learning is not isolated in individuals
but takes place socially as students, in order to secure success in the governance of their towns, have to
collaborate and cooperate to enhance the workability of their choices made.
6
2.4 Method
Statecraft X comprises of Game-based Learning as curriculum interventions in Social Studies classes with
fifteen year old students. The objective of enacting the interventions was, as a research project, to enhance
classroom teaching and learning in the upper Secondary subject of Social Studies, and investigate its
applicability in Singapore’s context of education. This paper focuses on the curriculum interventions, and
dialogic sessions that took place in two public schools at the Secondary level, involving six classes and four
teachers, in the first half of 2012. The two schools volunteered to be involved in the research project as they
were keen to have their teachers acquire new methods of facilitation, while exposing their students to digital
learning.
The first school had the flexibility to experience two cycles of intervention, and the second school only
had the time in their school curriculum to fit in one cycle of intervention. The six classes comprised of four
classes from one school and two classes from the other, making up approximately one hundred and twenty
students. These students were from the Express stream, which forms the broad mass of students in Singapore
Secondary schools. The teachers involved are named teachers A, B, C and D in respect of their anonymity in
our study. Each Statecraft X class had approximately twenty students, and each curriculum intervention took
place over three weeks, where students loaned iPhones from the research team and played the game. Within
the intervention period of three weeks, each class underwent four dialogic sessions lasting sixty minutes
each, which is the largest block of time allocated for Social Studies in these two schools. Audio-visual
equipment was set up to record the interactions between the students and the teacher’s facilitation. The
interpretations in this paper are based on the qualitative data of students’ utterances captured from the
dialogic sessions, which were then transcribed and coded categorically based on grounded theory (Corbin &
Strauss, 2008).
2.5 Results and Discussion

The richness of utterances captured in meaningful dialogue is often underestimated in its qualitative value. In
fact, students’ development in their understanding of governance became apparent when we investigated the
utterances captured through video recordings during each of the four dialogic sessions carried out in the
schools that collaborated with us. Taking a grounded stance, we discovered that these fifteen year old
students, through their playing of Statecraft X, were able to reflect and express through their dialogue that
they have understood, with significance to their level of appreciation, accountability and pro-activeness in
four areas of public policy, without any exposure to conventional Social Studies materials. These four areas
were: the realities of governance, social policy, economic policy and foreign policy. Examples were
numerous and for the purpose of this paper, we will be highlighting selective instances of developmental
realization among these students.
2.5.1 Appreciating the Realities of Governance
Generally, dialogue from the dialogic sessions was evident in showing a raised awareness in students’
appreciation of how it was a difficult task to govern. When teacher A challenged her students to make a
choice between putting a greater emphasis on the economic development or a country’s defense needs, she
was met with a diversity of responses, such as “It is important to be prepared in both defense and the
economy,” and “If you have the money you can make people happy and you also have stability.” Some
students realized that it was more important to have a balanced development, while others took an economy-
first perspective. Fundamentally, it became clear to students that they had to be accountable for their choices
as governors. In another session, one of teacher A’s students, having explored the various towns in Velar,
remarked that: “When you have a big country and you have an order, it will take some time for the order to
be passed down.” Without having been told about the difficulty of effective communication in big countries,
this student comprehended the challenge on his own. Teacher C asked students if leaders could neglect their
people and there was unanimity in their response that this was a possibility. Students, through this
recognition, were aware that government leadership can and will go wrong from time to time.
Students uncovered the paradox of governance when teacher C inquired about the students’ understanding
of the importance of citizenry, and one of her students remarked that citizens were not only needed to provide
for defense, but they could also extractively give a governor and his or her town money. Governance, to this
7
ISBN: 978-989-8533-12-8 © 2012 IADIS
student here was not an altruistic activity but one of situational advantages. This student realized that
governments, in ensuring the survivability of their nation, as well as their own positions as leaders, could not
always take a selfless stand, and when the opportunity arose, had to balance themselves with some form of
gain, possibly like the monetary gain that governors in Statecraft X experience. Whenever governors in
Statecraft X take actions using their Action Points (AP), they would have to wait an hour for their APs to be
replenished. Without making this overtly known to the students so as to enhance the self-exploratory element
of the game, students recognized for themselves this limitation and in Teacher D’s class, admitted that “the
one hour gap between different plays show that we may not be able to give people what they want as fast as
they expect it.” Statecraft X, to this student gives the “opportunity to think hard about the actions you take,
you can strategize” and “gives us the timing to plan, real life also cannot plan like that” which is indicative
again of the realities of governance.
2.5.2 Understanding Social Policy with Accountability
Interestingly, responses to social policy made up the bulk of students’ meaningful dialogue, which shows that
despite their age, they are accountable to their social surroundings, both within the game and beyond.
Perhaps, due to their close proximity to society itself as middle class students from public schools, this was
their primary area of focus. Teacher A referred to an in-game event where an epidemic hit the towns of Velar
and many students found their citizens dying from this epidemic. Students responded that some of their
factions were well prepared for this event, as some of them had prioritized the building of a healing center,
while others were not prepared as they were too preoccupied in the building of their barracks when the
epidemic hit. With further dialogue, students were appreciative of the effects that the ignorance of
governmental action could bring to society. In the game, governors can travel to other towns where they can
hold rallies or even attempt to take over the town by force. When a student from teacher B’s class attempted
to conquer another student’s town forcefully, he was met with failure. This student had made his attempt with
the assumption that the use of force was the only known way to conquer another town, and had his
assumption proven inaccurate, which became a learning point when he tried to do so. Responding to this
episode, a student remarked to the victim that the aggressor “tried to take over your town but people drove
him out.” Reflective of reality, the responding student realized that the strength of citizens’ belief in their
governors and the hope that it brought can overcome invasive political maneuvers.
Of course, the reverse could happen and it certainly did. In one of teacher C’s class, a student was
preoccupied with conducting rallies purely to “increase people’s trust so that the person can take over others’
town” and elaborating on that “increase people’s trust in the way of governance leading to people trusting
you more and the people want you to govern instead of using force and violence.” This student saw social
policy in connection with diplomacy in the expansion of governmental influence, but at the same time
recognized the potential for soft power application to conquer without violence. Placing a strong emphasis on
the citizenry’s belief, a student from teacher C’s class exclaimed that “if there is no belief in the governor the
town cannot function.” Students from both classes therefore understood that for societies to function
harmoniously, and for the citizenry to provide the best of their talents and capabilities for continued growth
and prosperity, there had to be belief in governance, a reality of many political contexts. Teacher A quizzed
her students on their thoughts and feelings about the immigration of talents into a country, drawing on the
Statecraft X in-game function where governors could restrict the type of immigrants as well as the skill level
of the immigrants. This led to a peculiar, alternative pro-immigration response when a student chose not to
echo the other responses of anger but rather that foreign talents would allow for “the need to continue to
improve yourself.” Armed with dialogue from both sides of the immigration debate, the students emerged
from the dialogic session more appreciative of foreigners in their country and accountable for their criticisms.
2.5.3 Experience and Thinking about Economic Policy
The dialogic sessions revealed that despite the lack of ‘knowledge’ in economics, students could reflectively
suggest ideas relating to tax and labor policies. This would not be possible without invoking their higher-
order thinking abilities, and was due to them experiencing economy-related policies at first hand. When it
came to the management of workers, students from teacher A’s class were in favor of giving opportunities for
gradual salary increments and in their deployment, they stressed the need for not having excessive people
employed as to minimize the loss of revenue and profit.
8
In another session, students from teacher A’s class also observed, through their manipulation of the tax
rates in the game and the subsequent responses of the citizens’ happiness levels, that a reduction of taxes and
increases in the citizenry’s happiness as not something that occurred across the board. With concern to
managing a country’s reserves and budgeting, one of teacher C’s students explained: “save money in case the
food runs out, there are illnesses or war, where food prices will go up,” and “in times of war increase taxes so
that there is more money to train soldiers and high tech weapons.” This student was lucid about having strong
reserves to cater for times of crisis and the necessity of increasing taxes to suit a more immediate concern
threatening a country. The productivity of labor is another concept explored through students’ responses.
Teacher B questioned the students about how they trained workers specific to the water tower and was given
the reply that the workers would be “more efficient, have a higher productivity and more output of water.”
Students were proactively concerned about the running of their towns because they could, through play,
intervene as the government and shape their own polity.
2.5.4 Engaging in Foreign Policy with Pro-activeness
Statecraft X allowed for students to experience what it was like enacting and engaging in policy without
learning about the concepts first. This led to them showcasing their prowess in dialoguing about military
strategy, the act of diplomacy and also the interdependence of nations, in many instances with pro-active
engagements. In the aftermath of a series of Salfredan attacks, a student in teacher B’s class found that he had
lost five out of his seven towns to forces from Salfreda, and thinking about this situation led the student to
conclude that he was managing too many towns. This is in actuality, a common problem of big countries,
which have centralized governments and defending these countries’ borders is no easy feat.
Dialoging about military expansion and action resulted in students unveiling the various reasons for war,
and not to view war merely as an act of aggression. Students in teacher B’s class found that governors
declared war with one other to “get more resources, to get more land for the citizens,” in the words of one
student. Students also realized the importance a strong defense plays in deterring invasions and keeping the
citizenry happy. One of teacher B’s students found that her town’s wall defense was zero, and deduced that
as a result of this, there was “no stability, no safety and no sense of security” which led to the town’s people
being unhappy despite racial harmony functions being organized for their benefit and to keep them happy. In
teacher D’s class, the perils of a small, successful nation was brought out when a student raised the point that
as a town, representative of small countries in reality, “when you’re outstanding you’ll be the center of
attraction and there will be those who wish to harm you and those who wish to look out for you,” which
highlights the importance of necessary alliances as well as caution in diplomatic relations, and the dangers of
invasion.
3. CONCLUSION
In this paper, we argued that there is no better time than the present for education to emphasize the learning
of public policy, given the turbulence in international affairs and the growing influence which globalized
digital citizens have over matters of resolution. Being inspired for a noteworthy solution, we began the use of
exploratory technology, manifested in a digital game, Statecraft X, to allow students to experience, through
the PPD model of game-based learning, the complexity of public policy as a governor of a town in a fantasy
realm. Through the examination of our research data, notably the utterances from the dialogic sessions, we
found developmental evidence of students coming to terms with concepts of governance without having had
any formalized ‘teaching’ sessions during their Social Studies classes.
The curriculum interventions were successful, as identified in the categorization of four areas of public
policy concerns: the realities of governance, social policy, economic policy and foreign policy. Students were
appreciative of the difficulties of governance, could conduct social policy with accountability, run economic
policy with higher-order thinking and engage in foreign policy pro-actively by the end of each of the three
week cycles. Certainly, the implementation of Statecraft X in schools faced a number of challenges. As
Singapore schools ran on closely planned schedules, we were unable to run the curriculum together among
the schools at the same time, and among the students involved in the curriculum intervention, there were a
number who tended not to contribute sufficiently to the dialogic data in the sessions they had. However, these
concerns can be addressed as the teachers we worked with for this series of iterations are now experienced in
9
ISBN: 978-989-8533-12-8 © 2012 IADIS
both the game-based and dialogic pedagogy, and exhibit the increased ability to facilitate towards the
completeness of our intended scenarios. We are therefore confident, that further iterations of the Statecraft X
curriculum and its refinements from this series of iterations would allow for the curriculum to encompass a
wider range of concepts related not just to governance but that of a broader conceptual framework which
would allow students to emerge as learners ahead of the demands of the 21st century.
REFERENCES
Book
Bakhtin, M. M., 1981. The dialogic imagination: Four essays. University of Texas Press, Austin, Texas.
Baumann, R., 1989. Performance. In E. Barnouw, ed. 1989. International encyclopedia of communication. Oxford
University Press, New York.
Chee, Y. S., in press. Video Games for “Deep Learning”: Game-Based Learning as Performance in the Statecraft X
Curriculum. To appear in Lee, C. B. and Jonassen, D. H., eds. Fostering conceptual change with technology: Asian
perspectives. Cengage Learning, Singapore.
Corbin, J. and Strauss, A., 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded
Theory (3rd Edition). Sage, London.
Dewey, J., 1916; 1980. Democracy and education (Vol. 9, John Dewey: The Middle Works, 1899-1924). Southern Illinois
University Press, Carbondale, Illinois.
Dewey, J., 1938; 2008. Experience and education, Freedom and culture, Theory of valuation, and Essays (vol. 13, The
Later Works of John Dewey, 1925-1953). Southern Illinois University Press, Carbondale, Illinois.
Elkjaer, B., 2009. Pragmatism: A learning theory for the future. In Illeris, K, ed. 2009. Contemporary theories of
learning. Routledge, London, United Kingdom.
Gee, J. P., 2007. Good Video Games + Good Learning: Collected Essays on Video Games, Learning and Literacy. Peter
Lang, New York.
Hershock, P. D., Mason, M. and Hawkins, J. N., 2007. Changing Education: Leadership, Innovation and Development in
a Globalizing Asia Pacific. Comparative Education Research Centre, Hong Kong.
Ministry of Education, 2012. Combined Humanities GCE Ordinary Level (Syllabus 2192). Author, Singapore.
Egenfeldt-Nielsen et al., 2008. What is a game? In Nielsen, S. E., Smith, J. H. and Tosca, S. P., 2008. Understanding
video games: The essential introduction. Routledge, New York.
Wenger, E., 2009. A social theory of learning. In Illeris, K, ed. 2009. Contemporary theories of learning. Routledge,
London, United Kingdom.
Journal
Chee, Y. S., 2007. Embodiment, embeddedness and experience: Game-based learning and the construction of identity.
Research and Practice in Technology Enhanced Learning, 2(1), 3-10. Doi: 10.1142S/S1793206807000282
Chee Y. S., Gwee, S. and Tan, E. M., 2011. Learning as become citizens by enacting governorship in the Statecraft
curriculum: An evaluation of learning outcomes. International Journal of Gaming and Computer Mediated
Simulations, 3(2), 1-27.
Chee Y. S., 2011. Learning as becoming through performance, play and dialog: A model of game-based learning with the
game Legends of Alkhimia. Digital Culture and Education, 3:2, 98-122.
10
E-LEARNING 3.0 = E-LEARNING 2.0 + WEB 3.0?
Fehmida Hussain
Middlesex University, Dubai
ABSTRACT
Web 3.0, termed as the semantic web or the web of data is the transformed version of Web 2.0 with technologies and
functionalities such as intelligent collaborative filtering, cloud computing, big data, linked data, openness, interoperability
and smart mobility. If Web 2.0 is about social networking and mass collaboration between the creator and user, then the
Web 3.0 is referring to intelligent applications using natural language processing, machine-based learning and reasoning.
From the perspective of advancements in e-Learning, the Web 2.0 technologies have transformed the classroom and
converted a passive learner into an active participant in the learning process. This paper posits that the way both previous
generations of e-Learning (1.0 and 2.0) have emerged with the prevalent technologies in their kin Web versions (1.0 and
2.0, respectively), it can be argued that e-Learning 3.0 will provide all earlier generations’ capabilities enhanced with the
Web 3.0 technologies. Furthermore, in this paper, reviewing all the theories of learning and examining closely the theory
of connectivism (considered to be the theory of learning for the digital age), it is argued that since most of the
technologies that are to be a part of e-Learning 3.0 are addressed by these principles, a call for a new learning theory for
e-Learning 3.0 is not justified. Finally, a review of the secondary literature shows that there will be various challenges
and issues related to prevalence and adoption of e-Learning 3.0 technologies, for example increased privacy and security
risks, web accessibility, readiness of the users, requirement for further standardization of e-Learning technologies and
social issues in term of increase of the digital divide.
KEYWORDS
e-Learning 2.0, e-Learning 3.0, Web 2.0, Web 3.0, Learning theories, educational technology.
1. INTRODUCTION
E-Learning 3.0, is it the way ahead in educational technology or just another buzz word in the digital-spaces
and education? At a prestigious IEEE conference, an interesting paper was published titled “E-Learning 2.0
= e-Learning 1.0 + Web 2.0” (Ebner, 2007, p. 1235)? After five years of incredible technological
advancements, a reappraisal of the digital spaces is due; it is only fair to ask the question: Is E-Learning 3.0
= E-Learning 2.0 + Web 3.0? Exploring what entails e-Learning 3.0, there are three parts of this paper: (1)
How Web 3.0 technologies will be incorporated in the existing e-Learning 2.0? (2) Is there a need for a new
learning theory for e-Learning 3.0? (3) What are the challenges and issues related to adoption of e-Learning
3.0?
This paper first describes the three generations of Web, usually referred to as, the Web 1.0, Web 2.0 and
the recently used neologism, Web 3.0. After explaining how the educational technologies and the information
systems used in each are different, these are related to the concept of e-learning. E-Learning literature also
broadly distinguishes between e-Learning 1.0, e-Learning 2.0 and e-Learning 3.0; hence the paper draws a
parallel between the generations of the Web and e-Learning.
Next, in relation to e-Learning 3.0, as education researchers are questioning (Wheeler 2009b; Meichel,
2009) whether a new learning theory is required for e-Learning 3.0; so in the light of the existing learning
theories, a preliminary attempt is made to address this. This paper does not indulge in the debate of whether
connectivism is a theory or a phenomenon, nor does it go into responding to the critique that has been done
about it. The simple approach taken in this paper is to make the basis of the argument all the important
principles of the connectivist theory of learning and then examine how the technological shift maybe
supported by these principles.
11
ISBN: 978-989-8533-12-8 © 2012 IADIS
Nevertheless, with any advancement and more so, the technological advancement, there are associated
challenges in terms of technological, ethical and social issues which are briefly discussed and highlighted in
the end of the paper.
2. HOW WEB 3.0 TECHNOLOGIES CAN BE INCORPORATED IN THE

EXISTING E-LEARNING 2.0?
2.1 Evolution of the Web

This section briefly covers the three generations of the Web in terms of the capabilities, technologies and its
usage. The Web has evolved from the early days of the ENQUIRE project to the transformation of Web 3.0
(Berners-Lee et al., 2001; Berners-Lee 1995) Broadly speaking, where the Web 1.0 connects real people to
the world wide web (www), the Web 2.0 connects real people who use the www, the Web 3.0 will connect
the virtual representatives of the real people who use the www. So, it is believed that Web 1.0 is about
providing information, Web 2.0 is about overload of information and the Web 3.0 is about control of
information (Rego, 2011).
Web 1.0 is generally referred to as the “read-only Web” making content available online for viewing.
Authors of the web generally write what they want others to view and then publish it online. The reader can
visit these web sites and can contact the writer or publisher if contact information is available. There is no
direct link or communication between the two. Examples of these are static web sites and web pages created
using HTML. (Rubens et al., 2011)
The term Web 2.0 is usually associated with the O’Reilly Media 2.0 conference (O’Reilly, 2004), but was
actually used for the first time in early 1999. (DiNucci, 1999) As opposed to the Web 1.0 which is referred to
as the static web, Web 2.0 is considered as the dynamic web. The users can read, write and collaborate to a
certain extent. The latest technologies used on client side or server side in Web 2.0 are Ajax (Asynchronous
Javascript), XML (Extensible markup language), Adobe Flash, PHP, Per, Python, Flash and so on.
The technologies and concepts related to the Web 3.0, though still in the infancy stage, are advancing
quite rapidly. The Web 2.0 has given rise to silos of data being generated by social networking and there will
be a need to enable the utilization of this data. An astounding statistics by the Forrester Research (2006)
shows that 97% of the users never look beyond the top three search results when they are searching on the
internet. The main features of the Web 3.0 technologies which differentiate it from its earlier generation,
Web 2.0 are given as follows (Cho, 2008; Wheeler 2009a; Berners-Lee, 2001; Morville, 2005; Semweb,
2011):
• Intelligent/semantic Web: The term semantic web refers to the W3C’s vision on the Web of linked data
enabling people to create data and build vocabularies. .Simply put, semantic web is all about describing
things in a form that is understood by computers
• Openness and interoperability: This refers to openness in terms of application programming interfaces,
data formats, protocol and interoperability between devices and platforms.
• Global repository of data: This is the ability of information to be accessed across programs and across
the web.
• 3D Virtualization: Extensive use of 3D modeling and 3D spaces using services like second life and
personalized avatars connected to your devices.
• Distributed and Cloud Computing: The delivery of computing as a service rather a product.
AI and machine learning are the main driving force behind the Web 3.0. For example, in Web 2.0,
searching the word e-Learning on Google will give a plethora of unrelated hits, but the Web 3.0 will solve
this problem by providing context to searching for online information. As the Web 3.0 is also referred to as
the Semantic Web of Data (Berners-Lee Video), there will be huge datasets created, so the need of the time is
management of ‘Big Data’ and ‘Linked data’ (Fischetti, 2010). The Web 3.0 will make use of technologies
such as RDF (Resource Description Framework, SPARQL (Query Language for RDF), OWL (Ontology
Web Language and SKOS (Simple Knowledge Organization System) (W3CSW, 2009); these will help
structure information such that programs like web spiders and web-crawlers can search, discover, collect and
analyze information from the web (RDF, 2004). “If HTML and the Web made all the online documents look
12
like one huge book, RDF, schema and interface languages will make all the data in the world look like one
huge database”, (Berners-Lee, 1999).
2.2 Evolution of e-Learning

E-learning is usually understood as instruction delivered via a computer in teaching and learning. A number
of other terms are synonymously used with e-Learning for example, computer based training, online learning,
virtual learning, web-based learning and so on. The central idea is that all these refer to use of information
and communication technology that pertain to all educational activities either performed individually or in
groups, working online or offline, synchronously or asynchronously, via networked or standalone computers
or other electronic devices. (Romiszowski, 2004; Garrison and Anderson, 2003) The learner of the future is
totally digitalized. The often so-called digital natives, digital immigrants, net-generation, Generation @ are
some of the names given by researchers to the students of today. (Prensky, 2001; Tapscott, 1997; Pelevin &
Bromfield, 2002)
2.2.1 E-Learning 1.0
With the advent of the Web, the major change was to have content available online. The concept of “learning
objects” came into being which were used to create what is known as the learning (content) management
systems (LMS or LCMS) developed to support study management and course organization for learners. This
is considered more of a traditional, rather than a hierarchical way of learning where communication is mono-
directional. In this direct-transfer model, the instructor is the distributor of learning material in a media-rich
way and addresses learners through various communication channels. This era is usually referred to as e-
Learning 1.0.
Stephen Downes described the use of Web 2.0 technologies for teaching and learning as ‘e-Learning 2.0’
(2005). Web 2.0 for example has transformed the class room in terms of how it is not only socially but
collaboratively constructed by using wikis, blogs, podcasts, and other social web tools. Such tools demand
dynamic content generation which may comprise of reflections and conversation, hence requiring a
collaboration and interaction. (Richardson, 2009) This is a collaborative way of learning where
communication is multi-directional where knowledge may be socially constructed.
Education researchers are now quite freely using the term e-Learning 3.0 in various blogs and discussion
forums. (Walters, 2010; Moore, 2010, Wheeler, 2009a) Emergence of cloud computing and availability of
new technologies such as collaborative intelligent filtering, increased and reliable data storage capacity,
higher screen resolutions, multi gesture devices and 3D touch user interface is leading us into the next
generation of e-Learning. One of the big things of third generation of e-Learning will be the ubiquitous
access to learning resources with the use of mobile devices to virtually access anything, anytime and
anywhere. (Baird 2007; Wheeler 2009a) Technology people also suggest the use of AI and data mining for
building the e-Learning 3.0 systems which have the capability to sift and sort big data, in turn provide to the
learner a deeper and better understanding of the learning process itself. (Rubens et al., 2011) Further,
education researchers believe that the underlying concept of ‘anytime, anywhere and anybody’ will be
supported by ‘anyhow’ which will be provided by virtual 3D worlds such as use of Second Life and personal
avatars. (Baird 2007; Rego 2010) With well-established Web 2.0 technologies and moving ahead to Web 3.0,
research communities are talking about the personal learning environments (PLEs also referred to as mash-
ups). “Personalization is seen as the key approach to handle the plethora of information in today’s
knowledge-based society.” (Ebner et al., 2011, p. 22)
Though e-Learning 3.0 systems are not prevalent commercially, however researchers are proposing such
solutions as proof-of-concepts or working prototypes. One of the first online services to use semantic web
automatically and intelligently organizing information about users’ specific interests is Twine. (Spivack’s,
2010) Another example of a true e-learning 3.0 system is AHKME (Adaptive Hypermedia Knowledge
Management E-Learning Platform) an e-Learning Information System having learning requirements
compliant with a Web 3.0 philosophy (Rego, 2011).
13
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.3 Relationship between Web Evolution and E-Learning Evolution

Relating the generations of the Web with generations of e-Learning, if Web 1.0 is the read only web and Web
2.0 is the read/write web, then Web 3.0 is the read/write/collaborate web. E-Learning 1.0 is about providing
to the learner information access, whereas e-Learning 2.0 in addition to all e-Learning 1.0 capabilities
provides authoring and interacting capabilities to the learner. Furthermore, e-Learning 3.0, enabled and
enriched with Web 3.0 technologies will promote intelligently collaborative, rich 3D virtual learning
environments which will bring learners together for anytime, anywhere, anyhow learning experience,
utilizing the semantic capabilities to parse the global databases of knowledge. Hence it can be deduced that
e-Learning 3.0 will provide all earlier generations capabilities enhanced with the Web 3.0 technologies. The
relationship between the generations of the Web and e-Learning are summarized in Table 1.
Ebner (2007) while attempting to answer whether e-Learning 2.0 is just a sum of e-Learning 1.0 and Web
2.0 technologies posited that the human factor is also important, that is the readiness and acceptability of
users towards a new technology. Therefore, Ebner (ibid) revised his equation to be e-Learning 2.0 = f (e-
Learning 1.0, Web 2.0, human factor). In the context of this paper, when accepting Web 3.0 technologies for
e-Learning 3.0, the human factor will also play some role but possibly not a significant one because the
transition for the aforementioned ‘digital natives’ will be much quicker as compared to the earlier
generations, the ‘digital immigrants’.
Table 1. Relationship between generations of Web and e-Learning (adapted from Rego, 2011).
Web E-Learning
Version Concept Technologies Concept Technologies
1 Read-only or write HTML, HTTP,URL Content management, CBT, LMS, eBooks
only, security, web of Unidirectional activities VLEs,
documents
2 Read/write Dynamic web Blended learning, LCMS, social networks,
Social web technologies. ASP, content authoring, video conf ,VLEs,
AJAX, podcasts, SNS Bidirectional activities, Mashups
Multimedia content
3 Read/write/request/col RDF, XML, OWL, 3D, Learner-centered, PLEs, Social semantic
laborate big data, second life U-learning, knowledge web, second life,
linked data representation personal avatars
3. DO WE NEED A LEARNING THEORY FOR THE WEB 3.0?

Information on the fly and on the go has become a norm for the digital natives. (Oblinger & Oblinger, 2005;
Oblinger, 2005) One relevant question to ask here is, does all the advancement in technology signal a change
in the learning behavior, or do we need a new learning theory for the digital natives? Recently, education
researchers are also discussing whether connectionism is adequate as a theory in terms of its coverage of the
need for the digital age; Connectionism is so Web 2.0 (Miechel, 2009; Wheeler, 2009b). George Siemens’
Connectivist approach to learning in a digital age might actually be superseded by our need to re-
conceptualize the whole idea of what learning will mean - especially when we are immersed in a world of
ambient mobile pervasive communication where intelligent agents and filtering tools do our bidding for us.
Based on review of the literature of learning theories, this paper makes a preliminary attempt to answer this
question.
Learning theory refers to a framework that helps us think about how and why change (in learning) occurs
(Smith, 1999). A review of the literature shows that there are different orientations and approaches to
explaining how this process of learning takes place, for example, behaviorist, cognitivist, humanistic,
social/situational and the connectivist approaches to learning. Broadly speaking, in the education literature,
there is reference to four theories of learning namely Behaviorism, Cognitivism, Constructivism and
Connectivism. In Behaviorism knowledge is perceived as facts that can be transferred from teacher to student
(can be related to e-Learning 1.0) Cognitivism opens up the black box of the mind, considering the learner as
an information processor whereas Constructivism suggests that learners create knowledge as they try to make
14
meaning of their experiences. Connectivism, considered to be the learning theory of the digital age, according
to Siemens (2004) is, “a successor to behaviorism, cognitivism, and constructivism.” These theories of
learning are briefly described in Table 2, in terms of the view of the learning process, locus of learning and
purpose of education; Table 2 is adapted from Ashworth et al., (2004) adding a column to relate with the
connectivist approach (Siemens, 2004).
Table 2. Summary of learning theories (adapted from Ashworth et al., 2004; Seimens, 2004).
Theories of Learning
Behaviorist Cognitivist Constructivist Connectivist
Aspect
Learning Skinner, Pavlov, Bruner, Kohler, Bandura, Vgotsky Siemens, Downes
theorists Piaget
View of the Change in behavior Internal mental Construction of Connecting
learning processes meaning from specialized
process experience information sets
Locus of Stimuli in external Internal cognitive Internal Draw information
learning environment structuring construction of outside of our
reality by primary knowledge
individual
Purpose of Produce behavioral Develop capacity Construct Ability to
education change in desired and skills to learn knowledge synthesize and
direction better recognize
connections
Connectivism applies ideas from biological models of the brain to neural networks in machine learning;
stating its basic principles as follows (Siemens, 2004):
Learning and knowledge rests in diversity of opinions.
Learning is a process of connecting specialized nodes or information sources.
Learning may reside in non-human appliances.
Capacity to know more is more critical than what is currently known
Nurturing and maintaining connections is needed to facilitate continual learning.
Ability to see connections between fields, ideas, and concepts is a core skill.
Currency (accurate, up-to-date knowledge) is the intent of all connectivist learning activities.
This paper does not indulge in the debate of whether connectionism is a theory or a phenomenon or for
that matter does not attempt to counter some of the criticisms about it. The simple approach taken in this
paper is to examine the important principles of the connectivist theory of learning as stated by Siemens
(2004) and then look at the new technologies which will be introduced as a result of the advancements in the
web technologies, thus compare and relate which technological shift may be supported by the principles of
the connectionist learning theory; this is demonstrated in Table 3. Based on the relationship illustrated in
Table 3, the paper posits that the cutting edge technologies to be a part of e-Learning 3.0 are adequately
supported by the principles of the learning theory of connectivism. Hence a call for a new learning theory for
e-Learning 3.0 is probably not justified.
15
ISBN: 978-989-8533-12-8 © 2012 IADIS
Table 3. Web 3.0 technologies supported by basic principles of connectivism.
Web 3.0 technologies used in e-Learning 3.0 Basic principles of connectivism

Social semantic networks, openness and Learning and knowledge rests in diversity of
interoperability opinions.
Big data or global data repository, linked data, cloud Learning is a process of connecting specialized
computing, extended smart mobile technology nodes or information sources. Currency of
knowledge is important.
Machine learning, artificial intelligence, personal Learning may reside in non-human appliances.
avatars, 3D visualization and interaction
Semantic web, control of information Capacity to know more is more critical than what is
currently known
Semantic web, collaborative intelligent filtering Ability to see connections between fields, ideas, and
concepts is a core skill.
Semantic web, collaborative intelligent filtering Nurturing and maintaining connections is needed to
facilitate continual learning
4. CHALLENGES AND ISSUES RELATED TO E-LEARNING 3.0

A review of the secondary literature shows that various challenges and issues may arise due to prevalence
and adoption of e-Learning 3.0 technologies. In the MashUps of the interconnected world in the hyperspace,
there will be an increased risk of privacy of data due to difference in privacy laws from country to country.
Also, added security risks may be faced due to lack of server-side checks and excessive privileges. (Weippl,
and Ebner, 2008) Researchers are concerned that the ethical dilemmas will also be enhanced as due to the
nature of the semantic web, vastness, vagueness, uncertainty and inconsistency may add to privacy and loss
of control. (Alves, et al., 2011; Alkhateeb et al., 2010) Similar to other web-based applications, there is going
to be more of a challenge to provide accessibility with web content to people with special needs. There have
been some initiatives from the World Wide Web consortium on this (WAI, 2009) but much more work will
be required in this direction. There are researchers who are apprehensive about the use of such advanced
technologies. “There are teachers who are still struggling with Web 1.0 let aside the students. So the
questions or apprehensions on the other side are: are our students truly ready to be autonomous learners? Are
our teachers willing to give into technology? (Wheeler, 2009a)” As suggested earlier in section 2.3, the
human factor is definitely a variable in the equation; the impact should reduce as the users become more and
more comfortable with technology. Lack of standards is another concern.This is in relation to exchange of
data and content between systems. Standard that do exist will have to be enhanced for the future generation
of the web for example, sharable course object reference model (SCORM), IEEE learning technology
standards committee (LTSC), Instructional Management Systems project (IMS) and so on. Educational
technology standardization movement has become an important force, for example IMS Global Learning
Consortium (IMS, 2011), IEEE (IEEELOM, 202), Dublin Core (DC, 2010) which are basically working
towards standardization of educational technologies for e-Learning applications such as metadata, digital
repositories, and many more.
5. CONCLUSION
Keeping pace with the unprecedented innovations in technology, the education domain has also evolved. Use
of technology in education has had a significant impact on teaching and learning. E-Learning systems have
also evolved with the evolution of the Web and the new technologies will continuously enable the application
of learning and teaching theories into e-Learning practice. This paper explores how Web 3.0 technologies
will entail e-Learning 3.0; whether the need for a new learning theory is called for with the use of the Web
3.0 in educational technologies and what are the potential challenges and issues related to the advent of
e-Learning 3.0. The evolution of e-Learning (e-Learning 1.0, e-Learning 2.0 and e-Learning 3.0) is related to
16
the three generations of the Web (Web 1.0, Web 2.0 and Web 3.0). Web 2.0 and the associated technologies
are well established and accepted by the user and the prevalence of these in e-Learning is common. This
paper posits that, just like its predecessor, Web 3.0 technologies, once stable and well developed will further
transform the e-Learning discipline. However, it does not seem that there is a need to call for a new learning
theory as the theory of connectivism should be adequate. However, with the advent of any technology and its
adoption, use of Web 3.0 and e-Learning will come with a plethora of technological, social, legal and ethical
challenges. So the equation in the title can be modified to E-learning 3.0 = ƒ (E-learning 2.0, Web 2.0, other
factors and challenges), (here ƒ means a function of) . Advanced technologies will continue to play a central
role in the development and evolution of e-Learning; however it will do so more in the background providing
connections between knowledge; so technology is not merely an enabling tool in education, rather a driver of
change.
REFERENCES
Alkhateeb,F., AlMaghayreh, E., Aljawarneh, S., Muhsin, Z and A. Nsour. E-learning Tools and Technologies in
Education: A Perspective. E-learning, 2010.
Alves da Silva, N., Morais da Costa, G., Prior, M. and Rogerson, (2011). The Evolution of E-learning Management
Systems: An Ethical Approach. International Journal of Cyber Ethics in Education (IJCEE), 1(3):12–24.
Ashworth, F., Brennan, G., Egan, K., Hamilton, R. and Saenz, O (2004). Learning Theories and Higher Education.
Level3, Issue 2. Retrieved from http://arrow.dit.ie/cgi/ on March 01, 2012.
Baird, D. (2007). Learning 3.0: Mobile, Mobile, Mobile Barking Robot. Retrieved March 21, 2012 from
http://www.debaird.net/blendededunet/2007/02/learning_30_mob.html
Berners-Lee, T. (1995). Past, Present and Future, IEEE Computer, 29 (10), 69-77.
Berners-Lee. T, Hendler, J., and Lassila, O. (2001). The Semantic Web, Scientific American, 284(5), 34-43.
Berners-Lee Video. Berners-Lee, T. (2008). The Semantic Web of Data. Retrieved from
http://www.youtube.com/watch?v=HeUrEh-nqtU on March 27, 2012
Cho, A. (2008). What is Web 3.0? Suite101. Retrieved from http://www.suite101.com/internet on March 20, 2012.
DC. (2010). Dublin Core Metadat Initiative . Retrieved from http://dublincore.org on March 26, 2012.
DiNucci, D. (1999). Fragmented Future. Print 53 (4): 32. Retrieved from http://darcyd.com/fragmented_future.pdf on
March 26, 2012.
Downes, S. (2005). E-Learning 2.0. ACM, eLearn Magazine, October 2005(10).
Ebner, M. (2007). E-learning 2.0 = e-learning 1.0 + web 2.0? In The Second International Conference on Availability,
Reliability and Security, 2007. ARES 2007, held in Vienna, Austria. pp 1235 – 1239.
Ebner, M., Schön, S., Taraghi, B., Drachsler, H., & Tsang, P. (2011). First steps towards an integration of a Personal
Learning Environment at university level. In R. Kwan et al. (Eds.), ICT 2011, CCIS 177 (pp. 22–36), Springer-Verlag
Berlin.
Fischetti, M. (2010). The web turns 20. Linked data gives people power. Scientific American. Retrieved from
http://www.scientificamerican.com/article.cfm?id=berners-lee-linked-data on March 22, 2012.
Forrester Research, (2006).Search engine usage report, Retrieved http://www.seowritingjobs.com/organic-search-engine-
results-why-they-mean-more-work-for-seo-copywriters/ on March 2012.
Garrison, D. R. and Anderson, T. (2003). E–Learning in the 21st century: A framework for research and practice.
London: Routledge/Falmer.
IMS. (2011). IMS Specification. IMS Global Learning Consortium, Retrieved from
http://www.imsglobal.org/cc/index.html in March 2012.
Meichel Blog. Meichel, F., (2009). Qu'est-ce que la cognition - points de repères en sciences cognitives Qu'est-ce que la
cognition - points de repères en sciences cognitive. Retrieved March 20, 21012 from
http://florencemeichel.blogspot.com/2009/05/quest-ce-que-la-cogntion-points-de.html. (English translation found at
Wheeler, 2009b)
Moore, D. (2010) Web 2.0. Darcy Moore’s Blog. Retrieved March 21, 2012 from http://darcymoore.net/
Morville, P. (2005). Ambient Findability: O’Reilly Media.
O’Reilly, T. (2004).What is Web 2.0: Design patterns and business models for the next generation of software. Retrieved
from http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html in March 2012.
Oblinger, D. D., & Oblinger, J. L. (2005). Educating the Net Generation. Educause. Retrieved from
http://www.educause.edu/educatingthenetgen in March 2012.
17
ISBN: 978-989-8533-12-8 © 2012 IADIS
Oblinger, J. L. (2005). Is it age for IT: First steps Towards Understanding the Net Generation. Educating the Net
Generation, 2.1-1.5. Retrieved from http://www.educause.edu/ir/library/pdf/pub7101b.pdf in March 2012.
Prensky, M. (2001). Digital natives, Digital Immigrants, On the Horizon, 9(5), p. 1-6.
RDF, (2004). Resource Framework Description W3c Semantic Web. Retrieved from http://www.w3.org/RDF in March
2012.
Rego, H. Moreira, T., Morales, E. and Garcia. F. J. (2010). Metadata and Knowledge Management drivenWeb-based
Learning Information System towardsWeb/e-Learning 3.0. Int. Journal of Emerging Technologies in Learning (iJET),
5(2):36–44.
Richardson, W. (2009) “Blogs, Wikis, Podcasts and other Powerful Web Tools for the Classrooms” Cornwin Press
California.
Romiszowski, A. (2004). How’s the e-learning baby? Factors leading to success or failure of an educational technology
innovation, Educational Technology, 44(1).
Rubens, N., Kaplan, D., and Okamoto, T. (2011). E-Learning 3.0: anyone, anywhere, anytime, and AI. In International
Workshop on Social and Personal Computing for Web- Supported Learning Communities (SPeL 2011).
Semweb. (2011). Semantic Web Tutorial. W3Schools, Retrieved from http://www.w3schools.com/semweb/default.asp in
March 2012.
Siemens, G. (2004). Connectivism: A learning theory for the digital age. Retrieved from
http://www.elearnspace.org/Articles/connectivism.htm. on March 26. 2012.
Smith, M. K. (1999). Learning theory. The encyclopedia of informal education. Retrieved from www.infed.org/biblio/b-
learn.htm in March 2012.
Spivack, N. (2007). Nova Spivak Blog. Retrieved from
http://novaspivack.typepad.com/nova_spivacks_weblog/radar_networks/ in March 2012.
Spivack, N. (2010). Twine. Retrieved from http://www.novaspivack.com/?s=Twine On March 22, 2012.
Tapscott, D. (1997).Growing up digital: The Rise of the Net Generation. New York: McGrwa-Hill.
W3CSW. (2009). Semantic Web Activity W3C. Retrieved March 22, 2012 from http://www.w3.org/2001/sw/
WAI, (2009). Web Accessibility initiatives. Retrieved March 22, 2012 from http://www.w3.org/WAI/.
Waters, S. (2010). Sue Waters Blog, Retrieved March 21, 2012 from http://suewaters.com/
Weippl, E. and Ebner, M. (2008). Security Privacy Challenges in E-Learning 2.0. In C. Bonk et al. (Eds.), Proceedings of
World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2008 (pp. 4001-
4007). Chesapeake, VA: AACE.
Wheeler Blog. Wheeler, S. (2009b). Learning with e’s. Retrieved March 20, 2012 from http://steve-
wheeler.blogspot.com/2009/05/connectivism-dead.html#!/2009/05/connectivism-dead.html
Wheeler, S. (2009a). E-Learning 3.0, Learning with e’s. Retrieved March 21, 2012 from http://steve-
wheeler.blogspot.com/2009/04/learning-30.html#!/2009/04/learning-30.html
Wheeler, S. (2011). E-Learning 3.0: Learning through the eXtended. Smart Web. Steve Wheeler. University of Plymouth.
Keynote Speech for National IT Training Confernece, Dublin, Ireland. Retrieved March 12, 2012 from
http://ittrainingconference.files.wordpress.com/2011/04/ittc_stevewheeler_smartweb.pdf
18
LEARNING IN OR WITH GAMES?

QUALITY CRITERIA FOR DIGITAL LEARNING GAMES
FROM THE PERSPECTIVES OF LEARNING, EMOTION,
AND MOTIVATION THEORY
Jan Hense1 and Heinz Mandl2

Ludwig-Maximilians-Universität München, Dept. Psychology
Leopoldstr. 13, D-80802 München, Germany;
1
+49 89 21803257 / 2+49 89 21805145
ABSTRACT
This conceptual paper aims to clarify the theoretical underpinnings of game based learning (GBL) and learning with
digital learning games (DLGs). To do so, it analyses learning of game related skills and contents, which occurs constantly
during playing conventional entertainment games, from three perspectives: learning theory, emotion theory, and
motivation theory. It is assumed that by an analysis of the processes leading to implicit learning in conventional digital
games, the underlying principles can be made explicit and subsequently used for designing DLGs effective for curricular
learning. Theoretical approaches which are used in the analysis include behaviorism, cognitivism, individual and social
constructivism for the perspective of learning theory. For the perspective of emotion theory, research on learning related
effects of positive and negative emotion is used, and for the perspective of motivation theory constructs and approaches
such as self-efficacy, locus of control, interest and self-determination theory are drawn upon. All of these theoretical
perspectives help to understand how players of conventional entertainment games acquire a wide range of skills and
contents while playing, and why they do so with immense motivational and emotional involvement. The results of the
theoretical analysis are subsequently used to deduct criteria and guidelines for designing and applying digital learning
games as powerful learning environments. Early experiences derived from the checklist’s application are reported.
KEYWORDS
Game based learning; digital learning games; theory; learning; emotion; motivation
1. INTRODUCTION
Game based learning (GBL) or learning with digital learning games (DLGs) has been one of the most
discussed and propagated forms of media based learning in recent years. Some programmatic authors (e.g.
Gee, 2007; Prensky, 2007) are extremely optimistic in regard to the potential benefits of GBL, and there is a
growing corpus of empirical research on educational uses of DLGs (e.g. Shelton & Wiley, 2007; Tobias &
Fletcher, 2011). However, little effort has been spent until now in systematically analyzing the theoretical
underpinnings of learning with digital games (cf. Moreno-Ger et al., 2008).
This theoretical paper aims at closing this theory gap in research on DLGs. This seems a particularly
important task, as at the moment there is little but experiential knowledge on what makes a DLG effective for
learning. Methodologically, therefore we will analyze learning in conventional digital games from the
theoretical perspectives of learning theory, emotion theory, and motivation theory. Undoubtedly, players of
conventional digital games are often acquiring a range of skills and contents while playing, and they do so
with immense motivational and emotional involvement. It is assumed that by an analysis of the processes
leading to these kinds of implicit learning, the underlying principles can be made explicit and subsequently
used for designing effective DLGs. Accordingly, we will subsequently deduce criteria and guidelines for the
design and application of effective DLGs from the previous theoretical analysis. We conclude with an
outlook on possible applications and further challenges for the theoretical foundation of learning with and in
digital learning games.
19
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. THE PROMISE OF DIGITAL LEARNING GAMES

After a period as the “new kid on the block”, digital learning games have been developing into the next “big
thing” in the area of media based education approaches. Similar to earlier trends such as e-learning, many
have set enormous expectations in this area. On one hand, these expectations relate to profitability aspects, as
the market for DLGs is believed to have an enormous potential for growth (see Picot/Zahedan/Ziemer 2008).
On the other hand, even greater are the expectations of some advocates of digital learning with computers in
regard to their potential for educational effects.
Authors such as Gee (2007) or Prensky (2007; 2011) are at the helm of this movement. Their simple, but
also persuasive argumentation is as follows: computer games that originally were only designed for
entertainment purposes most often invoke substantial learning processes in players, which vary depending on
the nature of the game. For example, action and racing games are expected to increase motor and perception
skills, while design and strategy games will increase forward-planning skills, and adventure games can foster
complex problem-solving skills. In addition, depending on the background story and scenario of the game,
users may acquire substantial content knowledge as well. This can occur through the challenging complex
tasks of a special force commando team in the context of a tactical “Ego-Shooter”, or players may develop
knowledge of history through historically-based trade or strategy games. According to the proponents of this
line of argumentation, all these learning processes occur without the learner feeling as if the process is
difficult, burdensome or uncomfortable. On the contrary, digital games are able to generate an enormous
amount of motivation which leads to intensive, sustained and emotional engagement with the game contents
and mechanisms. To some degree, this engagement can extend far beyond the reaches of the game, either
when users create online communities to exchange information about the game or when they develop their
own game content in the form of “mods” (modifications).
Advocates of games such as Gee or Prensky also argue that the undeniable potential of digital games to
promote unsystematic and implicit learning processes can also be intentionally and directly used to facilitate
the acquisition of curricular subject matter. They often refer to showcase model projects such as, to name one
example, the program “Revolution” (www.educationarcade.org/node/357), which is based on a modification
of the 3D role play “Neverwinter Nights”. Set in the context of the American Revolutionary War, players of
this game are able to experience social situations firsthand in order to develop historical knowledge about this
time period (see Forman 2004). In this visually and technically well developed MMORPG (Massively
Multiplayer Online Role Playing Game) learners are able to take on a variety of roles such as farmer, artisan
or slave and move around in an authentic Williamsburg setting and interact with other human players as well
as computer-controlled NPCs (Non Playing Characters). In the context of game episodes (chapters), a story
thread is generated that enables users to better understand the path to revolution.
As impressive as milestone projects such as “Revolution” and others may seem, the question remains as
to whether the principle can truly be applied on a broader scale, such as the proponents of DLGs claim to be
the case. Setting aside the question of the resources needed to develop such complex learning games, the
main problem concerns didactic quality. For without a doubt, it is not the games themselves that are effective
for learning per se. This can easily demonstrated by drawing on negative examples of expensively designed
DLGs which not necessarily provide an effective learning environment (cf. O’Neil et al. 2005).
3. LEARNING IN CONVENTIONAL ENTERTAINMENT GAMES

In order to answer questions regarding the quality of DLGs, we first need to have a better understanding of
the learning processes that take place while playing the games (cf. Garris, Ahlers & Driskell, 2002). It makes
sense to first analyze games that are intended not for learning but for entertainment purposes as the idea is to
take the mechanisms that are effective for learning from conventional digital entertainment games and
transfer them to the development of digital learning games. In addition, when analyzing computer games
from the perspective of theory, it is important not only to look at aspects pertaining to teaching and learning
theory, but also to consider at the motivational and emotional perspectives that play an important role while
playing these games (see Bartlett et al., 2009).
20
3.1 A Learning and Instruction Theory Perspective

The first theoretical perspective used in this analysis is the perspective of learning and instruction sciences.
Here, different theoretical approaches can be used to analyse the mechanisms which foster learning in games.
The most important are behaviorism, cognitivism, individual and social constructivism (Hense & Mandl
2009; Woolfolk, 2004). In our context, these do not preclude one another. Instead they should be regarded as
complementary, since the learning mechanisms proposed by the different theoretical approaches can be
relevant for different learning goals and outcomes. Furthermore they may be activated to varying degrees in
different game types and genres.
Starting with the behaviorist perspective, many games teach new skills and contents via operant
conditioning with its main principles of positive reinforcement and punishment. Reinforcement in games is
often realized by successfully mastering a sequence of tasks or levels, by collecting some kind of tokens or
symbolic currency, or by beating a high score. Punishment, on the other hand, can consist in losing a virtual
life, failing a level, losing a position in a race or ranking, or by being defeated by either a human or
computer-controlled opponent. These behaviorist principles are most dominant in action, racing and sport
games which need a lot of motor and perception skills with little cognitive processing. Here the players are
continually receiving immediate feedback about the success or failure of their actions. Accordingly,
behaviorist learning mechanisms can be expected to be most effective in terms of practicing and repeating
routines, primarily in the areas of perception and motor skills, but they can potentially also be useful for the
acquisition of factual knowledge.
From a cognitive perspective, as represented for example by the instructional design approach (e.g.
Reigeluth, 1983), there are many digital games that can mainly be viewed as problem-solving activities and
accordingly can train learners’ problem solving skills in different content domains. This generally occurs
when the players use the information that is embedded either within the game context or the game scenario to
more or less solve complex cognitive problems. Games that operate on this principle contain a strong
narrative component and players often have to decide between various potential solutions or alternative
paths. Adventure and role playing games are classical applications for these principles. In addition to helping
players build problem-solving skills, these kinds of games can also be used to foster knowledge acquisition
and increase comprehension. This is accomplished by providing information within the narrative of the game,
which needs to be applied to the solution of a given problem.
From a individual-constructivist perspective (e.g. Brown, Collins & Duguid, 1989; Resnick, 1989), games
may be regarded as providing a rich, authentic and immersing environment for self-directed, discovery-,
inquiry- or problem-based learning activities. The prerequisite for this are challenging tasks or problems that
players regard as authentic and relevant, either in relation to the reality of the game that they can to relate to,
or in relation to their own experiences. Based on such problems, the game then allows the player to analyze
the situation and to test out a variety of solutions, as well as gain experience with and reflect on a specific
subject area or phenomenon. Examples of this are strategy and design games, since they are more or less
based on simulations or aspects from the real-world that serve as a context for the specific activities.
From the perspective of social constructivism (e.g. Bielaczyc & Collins, 1999), finally, the focus shifts to
the social and cooperative aspects of computer games. Learning in the context of computer games can here
be interpreted as the joint construction of socially shared knowledge, as this has been traditionally examined
through research on learning communities or on collective information processing. The processes that can be
examined most thoroughly occur in the context of MMORPGs. This is where the players come together as
teams with clearly defined roles in order to master tasks when the solution requires a high degree of common
planning and coordinated effort. The players communicate and cooperate with each other not only in the
context of the game, but also often in community elements such as online forums, chats or instant messaging
which allows players to coordinate and exchange ideas.
3.2 An Emotion Theory Perspective

The influence of emotions on the learning process have often been neglected to date in educational research
(see Astleitner, 2000). Especially in the context of learning in computer games, it is important that emotions
be taken into consideration as well. Even if the research to date has been relatively sparse, it can be said with
a degree of certainty that positive emotions such as joy or satisfaction generally have a positive influence on
21
ISBN: 978-989-8533-12-8 © 2012 IADIS
effective learning (Pekrun 1992). With respect to negative emotions, it is important to distinguish between
deactivating negative emotions such as boredom or hopelessness and activating negative emotions such as
fear or anger. While it can be assumed that deactivating emotions generally do not support learning
processes, the influence of activating negative emotions is more complex. If these are present in the right
amount, they can have an activating effect, but if they are excessive, they can have a blocking effect (see
Rheinberg 1999). Even when there is the right amount of an activating negative emotion, it is wise to use
caution because the motivational effect of negative emotions such as fear or anger is extrinsic and may
actually detract from the actual subject matter and learning process.
Fun and joy are the two things that first come to mind when examining individual emotions more closely
in relation to computer games. If one tries to identify exactly what makes a player experience fun and joy,
you will hear many different answers (see Choi et al., 1999). Reasons may include aesthetics such as
graphics, animation, music and sound effects or aspects of the game’s narrative. In addition, games often
provide players with the opportunity to immerse into a virtual world or to take on an artificial identity and to
experience the joy of success and other social aspects of the game. It is also important that the joy of playing
the game is not diminished by too low or too high a difficulty level, through subjective unfairness, or due to
usability issues. In addition to fun and joy, there are also other positive emotions such as curiosity,
satisfaction, and pride that can also be beneficial to learning processes.
With respect to negative emotions, it goes without saying that computer games aim to minimize
deactivating emotions such as boredom or hopelessness. Activating negative emotions, on the other hand, are
often specifically promoted. A certain amount of frustration when the goals of a game cannot be achieved on
the first try is a pre-requisite to motivate players to try a second time. Fear can also play an important role in
certain game genres such as ego shooters, especially when it plays a part in horror scenarios. However, this
also highlights the ambiguities pertaining to negative emotions because there are certain mechanisms that
would not be suitable to be used for processes intended to promote learning. When considering the use of
computer games for learning purposes, the basic conclusion is that it makes sense to maximize positive
emotions and to generally avoid negative emotions.
When analyzing the design of successful computer games from the perspective of emotion psychology, it
becomes clear that these games generally succeed when they adhere to the principles discussed above.
Examples of techniques that can be used for this purpose are state of the art design, an adaptive level of
difficulty, target-group specific virtual worlds and plots, immersing narratives, and intuitive operation. It is
also important to note that failing to meet one of these aspects may not prevent a game from being successful.
This indicates that the different aspects that affect emotion psychology may compensate for one another to a
certain degree. Therefore the individual and varied preferences of the players play an important role and
should be given due consideration.
3.3 A Motivation Theory Perspective

The final important theoretical perspective for analyzing learning processes in computer games is motivation
theory. There are a number of approaches can be drawn on to understand why computer games often are so
attractive and motivating for players. The most relevant are constructs such as achievement motivation, social
motivation, self-efficacy, interest and flow (Urhane 2008). Of particular interest is the self-determination
theory of motivation (Deci & Ryan 1993), which integrates certain elements of some of the other approaches
mentioned. It concentrates on explaining intrinsic motivation which is especially effective for learning
because it is not fueled by external rewards, but is rather directed at the specific activity itself. In the context
of learning in computer games, it makes sense to examine this approach more closely.
Self-determination theory postulates that intrinsic motivation depends on fulfilling three basic
psychological needs: competence, autonomy, and relatedness. Competence relates to the construct of self-
efficacy and describes the experience when an individual is in a position to be in control and master a
situation. There is no doubt that this is one of the most important and most attractive characteristics of well
designed computer games (cf. Salen & Zimmerman, 2004) since they continuously enable players to
experience self-efficacy. It is also interesting to note that this often occurs through contexts that users often
do not have access to in real life, such as driving race cars in a racing game, governing a city in a design
simulation or fighting dragons in a 3-D role-play, a fact which refers to the role of interest in this context (see
below).
22
In the context of the self-determination theory, autonomy describes the ability to strive towards one’s own
goals, interests, and aptitudes free from outside influences. While some computer games have a linear
structure, most offer certain degrees of freedom in specific aspects. Examples for a high level of autonomy in
computer games can be found in the aforementioned MMORPGs or in other games adhering to the “open
world” concept. Their main appeal is that they provide a simulated reality and allow players to develop their
character and its behavior in the direction of their choosing. In these cases, there is often no concrete goal or
end to the game. Of course there are limitations to this autonomy through the rules of the simulation and its
limitations. The game’s designers’ task therefore is to offer enough degrees of freedom and incentives to
stimulate players’ exploration.
The third important pre-requisite for motivating behavior postulated by the self-determination theory is
relatedness. This can be defined as the feeling of belonging to a social community, whether it be with like-
minded individuals, peers or colleagues. In this regard, the social elements that are part of modern multi-
player games have enormous potential. Even outside of the game itself, this can be observed in the many
online communities that are formed around popular games. It is also interesting to note that feelings of
relatedness can also develop with virtual characters. This could be with virtual family members in
simulations such as the popular “Sims” series, or computer-controlled “Party” members in adventure or
conflict-oriented games that use the help of film-like interim scenes to breathe life into the individual
characters.
Two other important motivational constructs beyond self-determination theory should be mentioned as
particularly important in regard to games, namely interest and flow. Interest can be defined as the special
relation between a person and a specific content domain or area of knowledge (Krapp, 2005). In regard to
games, the motivational potential of interest is relevant, as it highlights the role of game genre and narrative.
Both are important criteria for a game’s success among different groups of players, and it is important to note
that players of entertainment games are usually free to follow their specific interest in choosing a game.
A final construct to be mentioned here is flow (Csikszentmihalyi, 1975). Flow denotes an “optimal state”
of motivated action, in which a person is fully immersed in a challenging task or activity while being skilled
enough to master this task or activity. As cognition and affection both are entirely concentrated on the
activity, flow allows a maximum level of performance. To induce flow, a task or activity has to meet a
number of conditions: it has to have clear goals, the learner’s subjective skills have to match with the task’s
level of challenge, and immediate and informative feedback has to be provided. As good game design is
careful to meet these conditions, e.g. by successively and implicitly teaching players the skills needed in a
game, flow can be considered a potent element of players’ motivation.
4. QUALITY CRITERIA FOR DIGITAL LEARNING GAMES

What conclusions can be drawn from our analysis of educational, emotion and motivation theories of
learning in computer games? If one agrees with the argument that digital learning games can make use of the
mechanisms that are used in conventional entertainment games in order to support intended learning
processes (cf. Linehan, Kirman, Lawson & Chan, 2011), then it should be possible to use the results of our
analysis to derive theoretically well supported quality criteria for DLGs. On the basis of these considerations,
we have developed a list of quality criteria for DLGs (Figure 1).
In the recent past, we have used this list of criteria in a number of practice related projects, which were
either concerned with supporting the conceptual design phase of DLGs, with quality analyses of early
versions of DLGs, or with the formative evaluation of nearly finished games. Some important experiences
have come from these applications. The most important observation was that the full educational potential of
computer games, as indicated in our list of criteria, is often used only to a little degree. On the surface, it is
often immediately apparent that many DLGs cannot keep up technically with commercial games due their
smaller budgets. However, as already indicated, a simpler design may not necessarily prevent a (learning)
game from being successful, as can be seen in the growing market of casual games and mobile phone games.
Far more important than technological inferiority, however, would be inferiority relating to educational
aspects that can be identified using the criteria list. Three problems seem common to many DLGs.
23
ISBN: 978-989-8533-12-8 © 2012 IADIS
1. Clearly define the learning goals of the game without neglecting the playful elements
2. Make use of the full spectrum of learning principles used in digital games
a. Behaviorist principles
provide direct feedback (particularly reinforcement) on learners' actions
give opportunities for exercise and practice
b. Cognitivistic principles
embed complex problems within the game context
embed information needed to solve the problems within the game context and narrative
c. Constructivist principles
create realistic problems which are authentic and personally relevant to the players
offer different perspectives and contexts for a given content
create a social context for learning
provide instructional support
offer opportunities for learners' own construction processes
3. Evoke positive emotions
a. Guarantee that learners have fun, e.g.
provide an attractive game design
maximize usability
avoid frustration and disappointment
b. Provoke learners' curiosity, e.g.
offer different choices
offer opportunities for exploration
c. Allow for satisfaction and pride
provide positive feedback for learners' accomplishments
create opportunities for presentation of learners' accomplishments
don't let learners fail (too often)
4. Evoke and keep up motivation
a. Foster intrinsic motivation
make learning and playing intrinsically attractive
avoid too much focus on extrinsic rewards (score, awards etc.)
b. Allow for feelings of competence
set goals which are challenging yet realistic given the learners' ability
give learners complete control over their success (reduce influence of chance)
ensure frequent and constant opportunities for feeling competent
c. Provide autonomy
provide freedom choice, but avoid too muach uncertainty about possible negative consequences
provide freedom of action
d. Enable social relatedness
provide in-game cooperation with real and/or virtual partners
create game-related communities of learners
e. Meet learners’ interests
tailor game subject, narrative, and genre to learners’ interests
offer choices for the different interests of different learners
f. Enable flow
clearly state learners’ goals at each stage of the game
adapt difficulty level to learners’ ability and skills
provide constant, immediate and informative feedback
Figure 1. Quality criteria for the design, quality analysis, and evaluation of DLGs.
Note: for the sake of applicability the criteria here are presented in the form of recommendations
Firstly, it is sometimes the case that unsuitable learning mechanisms are used for the wrong learning goals
and contents. Behaviorist learning through reinforcement has its place, but more when it is important for
learners to practice and repeat facts rather than when learners must learn new information or when the goal is
to reach a more in-depth understanding of the subject matter. So it is important to provide for a close match
of learning goal and learning mechanism in each specific case.
Secondly, it is often the case that the wide-range of possible cognitive, emotional, and motivational
mechanisms to promote learning are not utilized and combined in meaningful ways. Instead, there is often a
one-sided focus on individual aspects such as attractive design, frequent incentives or a strong narrative
element. However, a good design doesn’t compensate for a less attractive game or learning mechanisms.
24
Frequent incentives lose their motivating power when they are too easy to achieve. And a strong narrative
element is only captivating when the players have enough opportunities to interact within the virtual world.
So care has to be taken in cautiously balancing the spectrum of possible learning mechanisms.
Thirdly, and herein seems to lie the biggest challenge, it is always important that game play and learning
are synthesized in a meaningful way. Our experience has shown that some products announced as DLGs are
in fact mere e-learning programs to which a number of game elements have been added. Although there is a
game-like aspect to these programs, the actual contents might still be transmitted through slide presentations
or spoken instructional passages, the difference being that these elements have been more or less cleverly
embedded within a game context.
5. CONCLUSION
Our theoretical analyses demonstrate that digital games in fact have a lot of inherent potential to foster
learning via a number of theoretically well established cognitive, emotional and motivational mechanisms. At
the same time it is evident that, given the state of the art of DLGs, many applications still fall short of making
use of the full range of mechanisms and often only realise the most basic promoting functions, such as
positive reinforcement. Accordingly, the results of the above theoretical analysis can be used to derive a
systemized list of criteria and guidelines for designing effective DLGs.
The educational significance of this paper is twofold. For the practice of designing and applying DLGs in
educational contexts, it gives guidance on what criteria need to be met to make them effective learning
environments. For further research, it provides a general framework which can be applied for the empirical
analysis of learning with DLGs.
Until today, the fact that computer games can provide influential learning environments had mostly been
considered in the context of research conducted on the effects of media. In the past, this research focused
primarily on the effect of depictions of violence and has brought forth evidence how these can have short-
and long-term effects on the experiences and behaviors of regular players (see Barlett et al., 2009). Despite
these negative aspects, we do not see any reason that the learning potential of computer games cannot be
used in a positive sense for productive learning processes.
The accompanying challenges can be seen in the challenges presented in this article that many DLGs
have been struggling with to this day. At the core of all of these difficulties is a basic issue relating to the
hypothesis that the advantages of entertainment computer games can be easily transferred to DLGs. The basic
problem lies in the fact that learning in computer games is something different to learning with computer
games. The hypothesis above can therefore only be fulfilled if it is possible to truly synthesize intended
learning processes with game processes.
However, we see positive opportunities for the effective use of DLGs when central principles of
educational psychology are considered, as we have summarized in our criteria list. It would be a mistake to
rely too heavily upon the learning effectiveness of the medium of the computer game alone. The risk is that
we would again take a promising approach with a lot of potential to effectively promote learning processes
and ruin it by deficits in the aspects relating to educational theory. This would lead to great disillusionment,
as has been the case with e-learning before.
REFERENCES
Astleitner, H., 2000. Designing emotionally sound instruction. The FEASP-approach. In Instructional science, Vol. 28,
pp. 169-198.
Barlett, C. P., Anderson, C. A. and Swing, E. L., 2009. Video Game Effects--Confirmed, Suspected, and Speculative: A
Review of the Evidence. In Simulation & Gaming, Vol. 40 No. 3, pp. 377-403.
Bielaczyc, K. and Collins, A. M., 1999. Learning communities in classrooms: A reconceptualization of educational
practice. In C. M. Reigeluth (Ed.), Instructional design theories and models. Volume II: A new paradigm of
instructional theory (pp. 269-292). Erlbaum, Mahwah, NJ, USA.
Brown, J.S., Collins, A. and Duguid, S., 1989. Situated cognition and the culture of learning. In Educational Researcher,
Vol. 18, No. 1, pp. 32-42.
25
ISBN: 978-989-8533-12-8 © 2012 IADIS
Choi, D., Kim, H. and Kim, J., 1999. Toward the construction of fun computer games: Differences in the views of
developers and players. In Personal Technologies, Vol. 3 No. 3, pp. 92-104.
Csikszentmihalyi, M., 1975. Beyond boredom and anxiety. Jossey-Bass, San Francisco, USA
Deci, E. L. and Ryan, R. M., 2000. The "what" and "why" of goal pursuits: Human needs and the self-determination of
behavior. In Psychological Inquiry, Vol. 11, pp. 227-268
Foreman, J., 2004. Video game studies and the emerging instructional revolution. In Innovate, Vol. 1 No. 1,
http://www.innovateonline.info/index.php?view=article&id=2 (12. 5. 2009).
Garris, R., Ahlers, R. and Driskell, J. E., 2002. Games, motivation, and learning. A research and practice model. In
Simulation & Gaming, Vol. 33, pp. 441-467.
Gee, J. P., 2007. Good video games + good learning. Collected essays on video games, learning and literacy. Peter Lang
Publishing, New York, USA.
Hense, J. and Mandl, H., 2009. Bildung im Zeitalter digitaler Medien - Zur wechselseitigen Verflechtung von Bildung
und Technologien [Education in the age of digital media. On the reciprocal interrelation of education and
technology]. In Henninger, M. and Mandl, H. (Ed.). Handbuch Medien- und Bildungsmanagement [Handbook of
media and education management](pp. 21–40). Beltz, Weinheim, Germany.
Krapp, A. (2005). Basic needs and the development of interest and intrinsic motivational orientations. In Learning and
Instruction, Vol. 15, pp. 381-395.
Linehan, C., Kirman, B., Lawson, S. and Chan, G. (2011). Practical, appropriate, empirically-validated guidelines for
designing educational games. Proceedings of the 2011 annual conference on Human factors in computing systems -
CHI ’11. New York, NY, USA, pp. 1979-1988.
Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L. and Fernández-Manjón, B., 2008. Educational game design
for online education. In Computers in Human Behavior, Vol. 24, pp. 2530-2540.
O’Neil, H.F., Wainess, R. and Baker, E.L., 2005. Classification of learning outcomes: evidence from the computer games
literature. In The Curriculum Journal, Vol. 16, pp. 455-474.
Pekrun, R., 1992). The impact of emotions on learning and achievement. Towards a theory of cognitive/motivational
mediators. In Applied Psychology: An International Review, Vol. 41, pp. 359-376.
Picot, A., Zahedani, S. and Ziemer, A. (Eg.), 2008. Spielend die Zukunft gewinnen. Wachstumsmarkt Elektronische Spiele
[Playfully winning the future. The growing market of electronic games]. Springer, Berlin, Germany.
Prensky, M., 2007. Digital Game-Based Learning. Paragon House Publishers, St. Paul, MA, USA.
Prensky, M., 2011. Comments on research comparing games to other instructional methods. In Tobias, S. and Fletcher,
J.D., (Eds.), Computer games and instruction (pp. 251-278). State University of New York, Albany, NY, USA.
Reigeluth, C. M., 1983. Instructional design: What is it and why is it? In Reigeluth. C. M. (Ed.), Instructional theories
and models: an overview of their current status (pp 3-36). Erlbaum, Hillsdale, NJ, USA.
Resnick, L. B. (Ed.)., 1989. Knowing, learning, and instruction: Essays in honor of Robert Glaser. Erlbaum, Hillsdale,
NJ, USA.
Rheinberg, F., 1999. Motivation und Emotion im Lernprozess. Aktuelle Befunde und Forschungsperspektiven
[Motivation and emotion in learning processes. Current research results and perspectives]. In Jerusalem, M. and
Pekrun, R. (eds.). Emotion, Motivation und Leistung [Emotion, motivation and performance] (pp. 189-204). Hogrefe,
Göttingen, Germany.
Salen, K. and Zimmerman, E., 2004. Rules of play: Game design fundamentals. MIT Press, Cambridge, MA, USA.
Shelton, B. E. and Wiley, D. (Eds.), 2007. The Educational Design and Use of Simulation Computer Games. Sense
Publishers, Rotterdam, The Netherlands.
Tobias, S. and Fletcher, J. D. (Eds.), 2011. Computer games and instruction. State University of New York, Albany, NY,
USA.
Urhahne, D., 2008. Sieben Arten der Lernmotivation. Ein Überblick über zentrale Forschungskonzepte [Seven kinds of
learning motivation. A review of current research perpectives]. In Psychologische Rundschau, Vol. 59, pp. 150-166.
Woolfolk, A., 2004.. Educational psychology. Pearson, Boston, MA, USA.
26
VIRTUAL SUMMER SCHOOL IN SECOND LIFE:

SUPPORTING CREATIVE COMMUNITIES
Mikhail Fominykh1, Ekaterina Prasolova-Førland1 and Monica Divitini2

1
Program for learning with ICT, Norwegian University of Science and Technology, Norway
2
Department of Computer and Information Science, Norwegian University of Science and Technology, Norway
ABSTRACT
Creativity can be applied to every domain of knowledge and must be seen as an important competence. Supporting social
creativity across different domains and disciplines in learning communities is an important part of collaborative process
in both university education and in the context of large-scale international projects. In this paper, we present the
experiences from the Virtual Summer School in Second Life as an attempt to support creative communities in a
systematic manner. The Virtual Summer School acted as a forum for the presentation of innovative approaches,
developments, and outcomes of research projects in the areas of technology-enhanced learning, serious games, and
collaborative technologies, facilitating the exchange of ideas between students, researchers, and practitioners. To the end,
we present lessons learned and implications for using this approach for supporting creative communities, outlining some
recommendations and directions for future work.
KEYWORDS
3D Virtual Worlds, creative communities, educational visualizations, social creativity, collaboration, Second Life
1. INTRODUCTION
Establishing and nurturing vibrant and creative learning communities is a complex process (Wenger et al.,
2002). Such communities are seen as highly important in developing and spreading new skills, insight, and
innovation (Johnson, 2010). The notion of a Community of Interest (CoI) incorporates the variety and dyna-
mism that are typical features of a modern workplace (Fischer et al., 2007). According to (Fischer, 2005,
Fischer et al., 2007), CoIs have potential to be more innovative and transforming than a single Community of
Practice if they can exploit “the symmetry of ignorance” for social creativity. Supporting social creativity
across different domains and disciplines in learning communities is an important part of collaborative process
in both university education and in the context of large-scale international projects.
We argue that three-dimensional Virtual Worlds (3D VWs) can benefit creating and supporting learning
communities. However, it requires a careful design that incorporates various activities and exploits ad-
vantages of the technology. 3D VW are often seen as a type of social media which are known for community
support (Jina et al., 2010) and they have some unique features in addition (Molka-Danielsen, 2011). They
support synchronous interaction, providing a sense of presence, which is important for the development of
online communities (Bronack et al., 2008). Many 3D VWs support user-generated content, allowing to leave
traces of activities, which may become part of the shared repertoire of the community through reification
(Wenger, 1998). Wide opportunities for interaction and simulating environments make 3D VWs suitable for
conducting a range of virtual events, including meetings, performances, and role playing (Sant, 2009).
The above features of 3D VWs extend the possibilities of using boundary objects (Star, 1989) and shared
artifacts as catalysts of collaboration (Wenger, 1998, Thompson, 2005). Boundary objects are externaliza-
tions that have meaning across the boundaries of the individual knowledge systems or sub-communities and
are necessary for overcoming distances in social creativity (Bruner, 1996, Papert and Harel, 1991a). Exam-
ples of such objects include ‘monuments’ (symbols strengthening identity within the community); ‘instru-
ments’ (an infrastructure supporting interactive communication), and ‘points of focus’ around which the col-
laboration is structured (Thompson, 2005). In addition, online communities can benefit from such VW envi-
ronments being dedicated community spaces (Wenger et al., 2002).
27
ISBN: 978-989-8533-12-8 © 2012 IADIS
In this paper, we explore alternative approaches to technology-enhanced learning, community building,

and creativity support. We have chosen the Virtual Summer School as an innovative education form exploit-
ing the strengths of 3D VW and combining 3D visualizations, role-plays, and virtual events. The Second
International Summer School on Collaborative Technologies, Serious Games, and Educational Visualizations
was held in the Virtual Campus of the Norwegian University of Science and Technology (NTNU) in Second
Life. The school was conducted in conjunction with the Cooperation Technology course at NTNU and orga-
nized by two research projects supported by the European Union (EU) – TARGET
(http://www.reachyourtarget.org/) and CoCreat (http://www.cocreat.eu/).
2. STUDY SETTINGS
In order to evaluate the effect of the Summer School on learning communities involved, we have conducted
an exploratory case study. Educational activities of the study were systematically designed using a theoretical
framework of collaborative creative process (Schneiderman, 2002), as presented below (Table 1).
Table 1. Creativity phases and Summer School activities
Creativity phase Course activities
– Collect (searching for material and – brainstorming the topic to be visualized
visualizing it) – describing the design in group blogs
– participating in virtual events
– Relate (consulting with peers)
– exploring other constructions
– Create (trying out solutions, creating – collaborative construction
associations, composing artifacts) – accessing building resources
– role-play presentations
– Donate (disseminating results)
– preserving constructions in the Virtual Gallery
2.1 Collaborative Educational Visualizations and Role-Plays

Collaborative educational visualizations and role-plays were conducted as part of the Summer School with 37
students working in 10 groups, 3–4 students in each. The students were required to build an educational
module representing a major curriculum topic and present it at a joint session by role-playing (Fig. 1).
Figure 1. Student visualization project Awareness Lab

We used pre- and post-questionnaires to identify the previous experience of the participants, their
expectations of the forthcoming activities, and how the activities conducted matched their expectations. Each
group was required to keep a blog for sharing and discussing proposals, reflecting and documenting the
progress, and for the final discussion. In addition, each student was required to keep an individual blog for
weekly reflection. The final presentations were attended, apart from the students, by representatives from EU
projects and the general public. The resultant constructions have also been evaluated by students from the
28
College of Education (COE), the University of Hawaii at Manoa (UHM). After the role-play session, each
group saved its construction in a repository called Virtual Gallery and evaluated the work of two other
groups.
2.2 Virtual Events

Two international events were conducted as part of the Summer School. The first was organized as a seminar
on EU projects, which included five presentations on relevant topics and a question-and-answer session. The
objective of this event was to demonstrate to the students how international cooperation can be established
and supported using modern technologies and to disseminate the results from TARGET, CoCreat, and other
EU projects, exploring the possibilities for cooperation. The seminar took place in a formal lecture setting,
with an amphitheater for the public, slide show, and interactive posters (Fig. 2). The event involved about 35
participants – presenters and the audience from several countries.
Figure 2. Virtual seminar at NTNU

The second event was organized as a virtual tour to the virtual campus of COE UHM and augmented with
a feedback session with an invited expert. The students visited the major highlights of the COE UHM virtual
campus. They were informed that the goals for the COE virtual campus are creating places for experimental
teaching and research, socializing and collaboration, outreach, culture, and place for entrepreneurship. The
visit was followed up by the return visit of the Hawaiian students. The goal of this exchange has been raising
awareness of each’ other research projects and seeding creative communities based on the joint interests.
2.3 Method and Data Collection

Our approach to using educational visualizations in 3D VW has been developed in several previous studies
(Fominykh and Prasolova-Førland, 2012). It is based on constructionism – an educational philosophy which
implies that learning is more effective through building of personally meaningful artifacts than consuming
information alone (Papert and Harel, 1991b). Constructionism is related to social constructivism which
proposes that learners co-construct their understanding together with their peers (Vygotsky, 1978). In
addition, we applied role-playing, which implies an active behavior in accordance with a specific role
(Craciun, 2010).
The data were collected from the direct observation of students’ activities, pre- and post-questionnaires,
virtual artifacts (chat log and 3D constructions), and user feedback in the form of blogs. For data analysis, we
use the constant comparative method (Glaser, 1965) that was originally developed for the use in grounded
theory methodology and is now applied more widely as a method of analysis in qualitative research.
29
ISBN: 978-989-8533-12-8 © 2012 IADIS
3. SUMMARY OF THE STUDY RESULTS
3.1 Collect Phase
3.1.1 Brainstorming the Topic to be visualized
For performing the visualization task of the Virtual Summer School, the students had the option of using both
Second Life and other tools, both synchronous and asynchronous modes.
Six groups explicitly stated that the process of their project work was creative. In particular, four groups
(including some of already mentioned) noted that they had a creative and productive idea generation process:
– Generally, we are of the opinion that our construction process was somewhat more creative than in real
life.
Three of the groups noted that their creativity was not affected by the technology as they were
brainstorming the constructions before starting to work in Second Life and designing on paper:
– In the beginning, we spent time brainstorming about our project, at this point we ignored any technical
limitations and decided that we would adapt our idea to these limitations when we started to build.
3.1.2 Describing Construction Design in Group Blogs
The students were required to describe the design of the constructions in their group blogs to allow the ideas
found during the brainstorming to crystalize. Reflecting on this task, they acknowledged its usefulness:
– Exploring and visualizing the topic textually through blogging had the advantage of allowing a more
detailed description of the topic and about the functionality of the application.
Blogging technology was found useful at this stage of the project work by many groups. The students
mentioned advantages of the technology 17 times, but the disadvantages only five times. Blogging was found
easy accessible and simple. At the same time, it has low interactivity and weak support for synchronous
activities, which however, was found to be positive by some of the groups:
– Another upside is that your mental work will not be disrupted. That might be the number one advantage
of avoiding instant communication. Disruptive communication may ruin your creative work when you focus
on intensive thinking.
3.2 Relate Phase

3.2.1 Participating in Virtual Events
After the first virtual event, the students were asked to provide feedbacks to the seminar in their individual
blogs, identifying both positive and negative aspects. Among the positive aspects, the following themes were
mentioned most frequently (with the number of students discussing them).
• Geographical independence of the virtual meetings, allowing the attendance of participants from
different EU projects and countries (15)
• The novelty and excitement when facing the technology and learning approaches “different from the
normal kind of lectures” (5)
• The comfort of use both for the lecturer and the audience, including low threshold for asking
questions and the flexibility of giving a talk from own office (8)
– The main advantage is that you can have lectures with both speakers and audience from all over the
world. […] Also, comments and discussions with people from around the world might be completely different
than what would result from an audience with just Norwegians.
Among the negative aspects, the following items were mentioned most frequently (with the number of
students discussing them).
• Technical problems, especially with the sound, diminishing the overall educational experience (15)
• Attention distractions both inside (“unusual surroundings”) and outside the virtual environment (e.g.
accessing social tools) and therefore difficulties with concentrating on the content (6)
– May be harder to keep focus during the presentation. Easier for the mind to slip when you’re at the
computer.
30
The analysis of the feedback from the second virtual event showed the different types of learning that
occurred during the virtual tour. We identified eleven major themes. Those related to creativity and
community support are (with the number of students discussing them) – campus atmosphere (11), campus
infrastructure (10), Hawaiian culture (11), sense of place and immersion (18), and places for informal
learning (11). It was evident that the majority of the students felt an immersive Hawaiian sense of place.
However, some students were not convinced by the immersive qualities of the environment:
– I did not feel ‘transported’ to Hawaii as the whole concept of a 3d-simulation does not appeal very
strongly to me, and I usually draw a very clear distinction between real life and a virtual imitation.
In the general feedback to both events, the students discussed the possibilities of 3D VWs for
international collaboration and discussion, communication, promotion, corporate training, and emergency
simulations:
– There might be some merit in using 3D virtual environments in creating communities across bounda-
ries. […] we want to mention the potential of events; one-time happenings where one is able to gather
around a common interest at a specified point in time and experience it together with other attendees.
However, the community building was understood as a long-term process that requires time:
– We could not really develop a bigger community based on our virtual events, because there was only
very little time to communicate informally with other participants, but nevertheless they are vital for develop-
ing a community.
3.2.2 Exploring Other Constructions
We explored in what way the students were inspired by other constructions available in the virtual campus,
including the constructions resulting from the First Virtual Summer School in 2010. The students expressed
very different opinions from stressing the importance of studying previous students’ constructions to
mentioning a minor effect of this kind of studying for inexperienced users. Five groups stated that they were
inspired by the available resources and examples of constructions:
– […] the student constructions can stimulate the community development by providing new ideas and
inspire other people to create their own constructions.
The students discussed how resources and examples of similar projects available in the Virtual Campus
affected their creativity. Only one student group stated that their creativity was positively affected by the
resources and other constructions in the Virtual Campus. The other groups were to different degrees certain
that their creativity was not affected:
– We looked at the earlier projects to get a feeling of what is possible of achieving in the given time for
the project. Of course, our building was a bit inspired of the style of building […].
3.3 Create Phase
3.3.1 Collaborative Construction

The students applied different metaphors and design approaches that can be sorted into three main categories.
They are ‘scenes for their role-plays’ (purposes were too unclear without the presentations), ‘facilities’
(workplaces, which visitors could use, games, where they could play, or tools, where a single user could
learn), and ‘museums’ (exhibition and guided tour instead of the role-play).
Half of the groups stated that 3D VW positively affects creativity and supports generation of new ideas:
– New ideas were often generated by “playing around” with objects without a concrete plan of what we
wanted to achieve but by combining elements (prims) which we liked into a greater construct.
At the same time the other groups argued that the technology, being unknown, hinders creativity:
– It affected our creativity in that manner that neither of us had any experience […]. So when we were
supposed to start building, we did not know what was possible, and how to do the things that were possible.
3.3.2 Accessing Building Resources
The building resources available in the Virtual Campus were used to a limited degree. Most of the groups did
not see them contributing to the community support. However, three groups explicitly mentioned these
resources ease the constructing process:
31
ISBN: 978-989-8533-12-8 © 2012 IADIS
– We discovered elements from other projects and generally around in second life that we wanted to
incorporate into our [project]. Other things gave us inspiration to try to make ourselves or improve […].
– The amount of previous constructions was small, but it still showed what could be done, and what to
aim for. Especially the latter might be inhibiting to creativity, as it might not be especially motivating to
surpass the previously created work […]. The already available scripts and textures made building cheap,
although it might lock participant into a narrow thought process […].
3.4 Donate Phase

3.4.1 Role-play Presentations
During the final phase of the Summer School, the students were presenting their constructions to other
participants. In the discussions, all the groups noted advantages of role-playing as a learning activity. The
most popular of them include efficiency and safety comparing to the real-life training, possibility to have a
good contact with the audience, and offering experience together with information:
– 3D role-plays can be useful and sometimes necessary for imitations of real-life situations that can be
dangerous, or that can happen (but still useful) with some lesser probability.
The students identified two most serious challenges for such type of activity: not enough realistic
experience and the amount of effort required to make a play. Half of the groups discussed these challenges:
– […] even though we are presenting something based on a role-play we are still in a virtual
environment. We think that it is not the same having a role-play in virtual environment or in real life.
Role-playing activity was also found to be an important part of the visualizations. In some cases, they
clarified the purpose of static constructions. In some other cases, role-playing became the central part of the
projects, while 3D constructions were serving as a stage.
3.4.2 Preserving Constructions in the Virtual Gallery
The students acknowledged the possibilities of 3D VWs for international collaboration, virtual visits, and
knowledge sharing as it was done in the Summer School events. Virtual Campus of NTNU and generally 3D
VWs were talked about as suitable for supporting communities in the long term.
First advantages for community, preservation of constructions, community repository, dissemination
Sharing 3D constructions received a positive feedback. Most of the groups stressed the importance of
studying previous students’ constructions to have inspiration. Some of the groups stated also that they get
additional motivation from exhibiting their construction for other people:
– Sharing and exhibiting constructions in the Virtual Gallery is good because it can help newcomers
introduce what 3D VWs […] are capable of, what is possible to do, what types of collaboration are possible.
However, a number of strong limitations were identified, such as low accessibility, technical problems,
and that experience is not realistic enough:
– The “general public” uses small computers, mobiles and other platforms that don’t have the power to
run 3D VWs […]. That’s more barriers added to the task.
4. DISCUSSION
The objective of Virtual Summer School was to explore learning environments by inviting participants into
practices where knowledge and insight is emergent from the diversity of the contributions. The virtual format
of the Summer School demonstrated the possibilities of modern educational technologies for working and
learning. It was a deliberate choice to organize the Summer School and the corresponding environment in
accordance with the four phases of creative collaborative process by Schneiderman. In the following, we
briefly discuss how the activities in the different phases contributed to seeding and nurturing creative com-
munities as well as how the existing Summer School facilities supported these activities.
• Collect phase: Brainstorming the topic to be visualized and discussing the design in group blogs
contributed to establishing an initial domain, engaging issues, insights, and practices for learning
communities. A set of resources in the Summer Schools such as existing student construction,
tutorials, and joint feedbacks sessions in Second Life as well as feedbacks to the blogs provided
initial motivation and facilitation for collaboration and brainstorming in blogs and other arenas.
32
• Relate phase: Participating in virtual events and exploring other constructions contributed to
establishing new connections and multi-membership in learning communities involved. These
processes were supported in the Virtual Summer School by providing boundary objects to enable
dialog and collaboration between learners from diverse backgrounds and disciplines (such as
exhibition booths and slides from different projects) and by supporting a flexible infrastructure,
enabling both formal and informal meeting and workplaces for members of different creative
communities.
• Create phase: Collaborative construction of 3D visualizations contributed to unleashing and
supporting social creativity in the participating communities during the Create phase, establishing a
joint practice and trying out different solutions. This process was supported and motivated by the
possibility of accessing building resources in the Summer School, both student constructions from
earlier generations and various building tools and facilities.
• Donate phase: Presenting the 3D constructions with the role-plays contributed to disseminating the
results from the participants and projects involved and enriching the reflective dialog in the
communities with innovative expression forms. In addition, the visualized results are available 24/7
in Second Life as a part of the Virtual Gallery, thus constituting a shared repository of community
knowledge. These activities have been supported in the Summer School by providing seminars on
role-playing in a workplace context as well as storage and retrieval facilities for 3D content.
Based on our experiences, we can outline the following implications for organizing learning processes
and supporting learning communities in the context of a virtual summer school. These lessons learned can
serve as guidelines for educators/project consortia in similar situations.
• 3D visualizations are important for community building and dissemination of educational content,
supporting exchange of ideas in a virtual workplace as well as enhancing creativity across
boundaries of different CoIs. Therefore, there is a need to explore alternative and innovative ways of
visualizing, storing, and managing community knowledge.
• 3D visualizations provide alternative possibilities for teaching and presenting innovative concepts
and research results in an easy-to-understand way. These possibilities should be further explored.
• Virtual events are an integral part of the educational process and, therefore, of the Summer School
organization and planning. We have explored different types of events and corresponding modes of
learning. In order to facilitate such events and different learning modes, it is necessary to provide
both social and educational spaces for community building and collaborative creative activities.
• Role-playing in 3D VWs constitutes a powerful disseminating tool and an integral part of the
collaborative creative process. Role-plays can also serve as workplace training for students (as
identified by their feedback). Therefore, a further exploration of the potentials of role-playing and
serious games for supporting learning at the workplace is recommended.
• All the mentioned elements, i.e. 3D visualizations, associated role-plays, and virtual events, are
interconnected, supplementary to each other, and necessary for creative communities support. For
example, without the role-plays, the results of the creative process, i.e. the 3D visualizations, were
not fully obvious. The 3D visualizations served as boundary objects and were, therefore, necessary
to create joint understanding between different CoIs. The virtual events, in accordance with
recommendations from Wenger, created opportunities for generation and exchange of new ideas as
well as new ties between communities (Wenger et al., 2002).
5. CONCLUSIONS AND FUTURE WORK

In this paper, we have presented the experience from the Virtual Summer School in Second Life as an
attempt to provide a systematized support for creative communities in a multi-cultural, cross-disciplinary
context. In this way, a virtual summer school could be thought of as a framework or a technique that provides
support for community building, collaborative creativity, and idea dissemination. Based on the results of our
study, we presented some implications for using this approach that might be reused in the future. In addition,
we identified some challenges, both related to the fluid and diverse nature of creative communities and the
technology, in particular Second Life. Although the latter was chosen for its general popularity and
33
ISBN: 978-989-8533-12-8 © 2012 IADIS
accessibility, the results will be relevant for other social VWs as long as they support collaborative co-
construction.
Our future work will focus on addressing the identified challenges. In addition, we will explore further
the possibilities of 3D virtual worlds for supporting creative communities, both in terms of the virtual envi-
ronment design and organizational forms for educational and social events.
ACKNOWLEDGEMENT
The work is conducted in collaboration with TARGET and CoCreat – projects supported by the European
Commission. This publication reflects the views of the author only, and the Commission cannot be held re-
sponsible for any use which may be made of the information contained therein. The authors wish to thank
participants from the EU projects as well as all the students from NTNU and COE UHM.
REFERENCES
Bronack, S., Sanders, R., Cheney, A., Riedl, R., Tashner, J. and Matzen, N. (2008), Presence Pedagogy: Teaching and
Learning in a 3D Virtual Immersive World. International Journal of Teaching and Learning in Higher Education, 20,
59–69.
Bruner, J. S. (1996), The Culture of Education, Cambridge, MA, Harvard University Press.
Craciun, D. (2010), Role – playing as a Creative Method in Science Education. Journal of Science and Arts, 1, 175–182.
Fischer, G. (2005), Distances and Diversity: Sources for Social Creativity. 5th Conference on Creativity & Cognition,
April 12–15, London. ACM, 128–136.
Fischer, G., Rohde, M. and Wulf, V. (2007), Community-Based Learning: The Core Competency of Residential,
Research Based Universities. International Journal for Computer-Supported Collaborative Learning, 2, 9–40.
Fominykh, M. and Prasolova-Førland, E. (2012), Collaborative Work on 3D Content in Virtual Environments: a
Methodology. International Journal of Interactive Technology and Smart Education (ITSE), 9, 33–45.
Glaser, B. G. (1965), The Constant Comparative Method of Qualitative Analysis. Social Problems, 12, 436–445.
Jina, L., Wenb, Z. and Goughc, N. (2010), Social virtual worlds for technology-enhanced learning on an augmented
learning platform. Learning, Media and Technology, 35, 39–153.
Johnson, S. (2010), Where Good Ideas Come From: The Natural History of Innovation, New York, USA, Riverhead
Books.
Molka-Danielsen, J. (2011), Exploring the Role of Virtual Worlds in the Evolution of a Co-Creation Design Culture. In:
Salmela, H. and Sell, A., eds. 2nd Scandinavian Conference on Information Systems (SCIS), August 16–19, Turku,
Finnland. Springer, 3–15.
Papert, S. and Harel, I. (eds.) 1991a. Constructionism: research reports and essays 1985 - 1990 by the Epistemology and
Learning Research Group, Norwood, NJ: Ablex Publishing Corporation.
Papert, S. and Harel, I. (1991b), Situating Constructionism. In: Papert, S. and Harel, I. (eds.) Constructionism. Westport,
CT, USA: Ablex Publishing Corporation.
Sant, T. (2009), Performance in Second Life: some possibilities for learning and teaching. In: Molka-Danielsen, J. and
Deutschmann, M. (eds.) Learning and Teaching in the Virtual World of Second Life. Trondheim, Norway: Tapir
Academic Press.
Schneiderman, B. (2002), Creativity Support Tools – Establishing a framework of activities for creative work.
Communications of the ACM, 45, 116–120.
Star, S. L. (1989), The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem
Solving. In: Gasser, L. and Huhns, M. N. (eds.) Distributed Artificial Intelligence. San Mateo, CA, USA: Morgan
Kaufmann Publishers Inc.
Thompson, M. (2005), Structural and Epistemic Parameters in Communities of Practice. Organization Science, 16, 151–
164.
Vygotsky, L. S. (1978), Mind in society: the development of higher psychological processes, Cambridge, MA, USA,
Harvard University Press.
Wenger, E. (1998), Communities of Practice: Learning, Meaning, and Identity, New York, USA / Cambridge, UK,
Cambridge University Press.
Wenger, E., McDermott, R. and Snyder, W. (2002), Cultivating Communities of Practice: A Guide to Managing
Knowledge, Boston, MA, USA, Harvard Business School Press.
34
FIVE COMPONENTS TO CONSIDER FOR BYOT/BYOD
Amy S. Ackerman and Melissa L. Krupp

The Richard Stockton College of New Jersey, Galloway, NJ, USA
ABSTRACT
Although school budgets have plummeted due to federal and state funding reductions, adopting Bring Your Own
Technology (BYOT), may address monetary tightening while simultaneously infusing 21st century learning.
Implementing BYOT may provide real, rigorous, and relevant learning for the students while posing higher-order
thinking questions from the teacher. This paper expands Williard’s Pyramid (Ullman, 2011) by proposing five BYOT
components with descriptions, examples, and recommendations within each. The five components are security,
stakeholders, policies, professional development, and financial planning. The paper closes with future predictions as well
as words of caution regarding BYOT initiatives.
KEYWORDS
Mobile learning, BYOT, BYOD, Educational Technology
1. INTRODUCTION
Throughout the world, the influence of technology has forever changed the familiar framework of schools.
We were once constructed with the Industrial Revolution in mind, but now, through the implementation of
21st century skills, we are embracing a new paradigm that no one has ever built before. Through fast-paced
technology integration, we can no longer educate the way we did, but rather provide a new form of education
that promotes authentic learning. (Jacobs, 2010). To provide students with optimal learning, one must offer a
school structure where real, rigor, and relevant learning occurs at all times. One way in which to achieve this
goal is to break the barriers between students/teachers and his/her learning. The initiative, Bring Your Own
Technology/Device (BYOT or BYOD), provides the platform for which students become active participants
in learning both in and outside of the classroom setting by enabling employees and students to use their
personal devices and connect with school/employers network. Such devices people bring include, but are not
limited to the following: smartphones, tablets, E-readers such as Nook and Kindle, laptops, iPods, and many
others. This paper addresses the problem and framework, five components to consider with BYOT, summary
and conclusion.
2. PROBLEM AND FRAMEWORK
2.1 Problem
As financial budgets become tight and restrictive, school districts and employers may not afford a
contemporary device for every learner and/or employee. As a result, a gap exists between what is taught and
what needs to be taught in the classroom today. Are we preparing our students to be productive citizens for
society as it is or as it ought to be? Or are we preparing them for the past? Students seem more motivated and
interested to learn from their own devices rather than a teacher lecturing on various topics, yet we often
prohibit the use of personal devices as described in technology policies and student handbooks worldwide.
Through the use of Bring Your Own Technology (BYOT), teachers and students can change the focus of the
classroom and become more student-centered. The role of each member of the classroom may also change to
reach the student’s maximum learning potential by creating a customized education for each type of learner.
35
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.2 Definition of Terms

Bring Your Own Device (BYOD) (Norris & Soloway, 2011).
Bring Your Own Technology (BYOT) (Stanley, 2012).
Bring Your Own (BYO) (R.Cave, personal communication, April 26, 2012).
Bring Your Own Computer (BYOC) (BYOT.ME, n.d.)
Flipped Classroom – switch of instruction by giving a lecture the night before via video and using class time
to help students with problems/questions or ask higher order thinking skills (Hart, 2012).
Mobile (Internet) Devices – handheld devices having internet capabilities such as iPads, netbooks, Kindle
Fires (Stanley, 2012)
Tablet – mobile computer operated through a touch screen (PCmag.com, n.d.)
2.3 Previous Work

Eric Williard, chief technology officer of Community Unit School District in Illinois, has developed
“Williard’s Pyramid” (Ullman, 2011) explaining how one can implement BYOT effectively. These
components (listed in order from most to least important) are: funding and leadership, tech planning and
support, standards and infrastructure, hardware and software, and actualize. These five components coupled
with professional development can be helpful when developing one’s own program (Ullman, 2011).
2.4 Purpose
The purpose of this paper is to enhance Williard’s pyramid by providing groundwork for effective
implementation of BYOT and project how BYOT will transform the medium of how students learn. One
needs to consider many components prior to successful implementation. Some of these include security,
involvement of all stakeholders, school policy, professional development, and a financial plan for
sustainability. With any missing components, integration of BYOT may be unsuccessful; however, creating a
seamless connection among all five components will not only strengthen the likelihood of a successful
implementation, but also help the students reach beyond imaginable heights. Desired pedagogy for academic
success may be realized by providing a learning environment where no child is left behind. This type of
customization may enhance student productivity, engagement, equitable participation, and a wide variety of
instructional resources while providing ongoing student accountability.
3. FIVE COMPONENTS TO CONSIDER WITH BYOT

As an expansion of Williard’s Pyramid (Ulllman, 2011), five components are presented here with
descriptions, examples, and recommendations within each. The five components are security, stakeholders,
policies, professional development, and financial planning.
3.1 Security (Component One)

Common concerns held by most districts are how can a school monitor one-on-one devices while not
jeopardizing school security? How can schools provide necessary firewalls to secure confidential data?
Security is often considered to be the most important focal point when developing a BYOT plan to develop
necessary safety measures to protect all parties: students, faculty, administrators, and databases filled with
private information.
Whether in school or at work, people are using their personal mobile devices regardless of having a
formal policy of BYOT (McCaney, 2012); therefore, security is at-risk and is an essential part of this process.
Security issues include infrastructure, bandwidth, wireless networks, access points, and many others. Both
administrators and members of the technology department need to converse and discuss options regarding
parameters and privileges for all members to ensure maximum security while maintaining a world-class
education. In addition, options such as setting up a separate wireless network for students are a viable
36
alternative to maintain security on important information while not compromising teachers or administrators
accessibility.
3.1.1 Security Examples
New Canaan High School in Connecticut (Ullman, 2011) has curbed a security problem into a working
solution by asking students to register their Apple devices through a MAC address
(https://sites.google.com/a/ncps-k12.org/macaddress) to monitor each individual device carefully. Due to
budget cuts, Alvarado Independent School (Ullman, 2011) in Texas, has turned to BYOT and has found a
possible solution to verify the device and user by providing a screen that flashes and asks every user to agree
to connect to the network similar to using one at a hotel (Ulman, 2011). At this time, all devices will be
updated with antivirus tools and become secure for the network. Although security is at the forefront of any
new technology integration, parallel efforts often include ill-willed attempts to compromise the systems.
3.1.2 Security Recommendations
To unravel the best security, we must provide students the opportunity to safely and securely use devices and
provide a means for which to store content. Some suggestions for alternate means of security might include
an additional security code for e-mail, using iCloud or web-based programs such as Dropbox/Evernote,
google docs, and/or other free web 2.0 tools. Providing these alternatives for student storage will not impact
the overall security of the school district, but on the contrary, will increase collaboration and communication
amongst users.
3.2 Stakeholders need to be vested (Component Two)

Creating a partnership among all stakeholders may enable “buy-in” and promote success on any initiative. If
all members see the value of integration, the success may be astounding and all parties involved will benefit
exponentially. There are several components to effective partnership that extend beyond the walls of the
classroom. Stakeholders such as superintendents, principals, supervisors, curriculum developers, parents,
technology departments, and members of the community, must be well-informed and see the importance of
such integration because it may affect everyone beyond the classroom environment.
No longer will education be defined by student/teacher, but rather a collaborative and cooperative effort
by all. The classroom school day will no longer be seven to two, but rather 24/7/365. Through shared
partnership, all stakeholders may benefit and see the astounding learning we will all gain from such
implementation. Partnering with technology provides the premise for which maximum learning takes place
and all stakeholders contribute to the success of such a partnership (Prensky, 2010). Stakeholder buy-in
enables observation of how BYOT will transform the paradigm of education where learning will be
continuous, and the school will not be the sole proprietor of information transfer.
3.2.1 Partnering Examples of stakeholders
Partnering is a key component to any integration, but also yields success in the classroom environment.
Creating a partnership between students and teachers is essential for maximum learning to occur on both
platforms. Both participants shall contribute to the classroom differently. Teachers will develop and design a
real and rigorous learning environment that provokes the student’s passions while also providing the ability
to integrate technology to enhance learning and create products that represents transference of knowledge. A
strong partnership between the student and teacher can foster great discovery and provide the structure for
lifelong learning. Bring Your Own Technology (BYOT) allows this type of learning because it eliminates the
disconnect felt between what students want to learn and what they are expected to learn. In addition, it
provides the student with the opportunity to build one’s knowledge and skill set while providing growth in
and outside the classroom setting.
The purpose of implementing BYOT at Forsyth (Georgia) and Williamson (Tennessee) County Schools is
to provide student access to technology to impart student accountability, collaboration, and immediate
feedback (Stanley, 2012). This initiative provides the student with necessary tools for research and affords
students to reach their individual potential (Giordano, 2011). Students will be given opportunities to see the
connections between their lives and learning and will no longer be grouped by age, but rather on ability and
personal interests.
37
ISBN: 978-989-8533-12-8 © 2012 IADIS
3.2.2 Partnering with Stakeholders Recommendations

Having a shared, common goal is essential to promote and implement BYOT in one’s school district. To
ensure optimal success, each member’s contributions may directly affect the group as a whole. Working
together will not only provide all to excel, but most importantly, enable the student to see the connection
between school and real life. Students will no longer ask, “why do I need to know this,” but rather say, “I
need to know this because it will affect me today, tomorrow, and always.” Partnerships provide pathways for
positive communication to effectively carry out an objective while providing astute knowledge and
observations that might not be available if working solo. In addition, global collaboration provides growth
and enhances divert thinking to create innovative solutions to existing problems and unforeseen quandaries in
the future.
3.3 Policies (Component Three)

Permitting the use of personal electronic devices in a school district also influences Acceptable Use Policy
(AUP) and student handbooks (Harris, 2012). Prior to implementing these changes, one must inform
students, parents, and staff of required usage protocols (Harris, 2012; Sucre, 2012). Since students will
experience increased autonomy, they must be taught correct procedures of effective usage of devices to
augment learning and eliminate improper usage. The school district is responsible for providing a safe,
learning environment for all students, therefore, protecting against inappropriateness on the internet as
defined by the Children’s Internet Protection Act also referred to as CIPA (Quillen, 2011). While integrating
BYOT, the school district must provide a safe, appropriate, and secure network to ensure safety for all pupils.
3.3.1 Policy Example of Liability
Policy is very important regarding BYOT and must include financial liability of personal devices (Harris,
2012). Since these devices are based on the “student paid modules,” (devices that the students pay for, not the
school district) who will pay if it gets broken, lost, and/or stolen at school? Who will repair the device if
malfunctioning – carrier, insurance, and/or school? Will the school confiscate the device if the AUP policy is
broken?
What alternative means will the district provide if the student (family) cannot afford such a device to
ensure equitable learning? What happens if a student accesses inappropriate material from a 3G or 4G
network at school – who is at fault? (Quillen, 2011). When writing policy it is important to address all of
these concerns to avoid future problems and/or litigation. According to a BYOT Use Contract (p. 176),
Fayette County Schools (Georgia) has adopted a policy regarding no school liability of personal devices and
a breach of policy if accessing from a 3G or 4G network.
3.3.2 Policy Recommendations
Since BYOT is a new concept, research is limited regarding effective implementation and/or policy;
therefore, many districts are piloting programs to iron out any problems. Many suggest that prior to a full
implementation of such initiative, a pilot program be developed with the oldest aged students on a “trial and
error” basis. This small scale experimentation may supply the school district with valuable information to
amend existing policy and/or procedure prior to full engagement by all.
3.4 Professional Development (Component Four)

Ideally the classroom no longer takes the form of teacher-centered, but rather puts the attention in the hands
of the student. Strong emphasis is now being placed on various types of learning such as: student-centered,
problem-based, project-based, case-based, inquiry-based, active learning, constructivism, and learn by doing
(Prensky, 2010). A shift in focus has taken place from teacher to student, yet the teacher’s role is even more
vital now than ever. The educator must provide a strong foundation for the students to acquire knowledge and
skills, but the procedure in which one will learn may drastically change. Since information is easily
accessible, students no longer need a teacher to pose questions that are available in google, but rather present
higher-order thinking questions that promote critical thinking, analysis, collaboration, transfer, application,
and many others. As a result, providing professional development for all educators will not only augment
38
their success, but may provide a greater breadth of knowledge in a rich, technology-based environment where
learning takes place.
3.4.1 Professional Development Examples
Professional development allows all professionals to develop, apply, and hone necessary skills to increase
learning outcomes. Since technology is becoming more prevalent in schools today, teachers will expand their
knowledge and skill sets to augment student achievement in the classroom. If school districts provide
additional educational support in preparing teachers with effective technology integration, our students will
be best prepared for the classroom, but most importantly, for the future workplace. Having a skilled teacher
implement technology will advance creativity, collaboration, and partnership amongst all (Puente, 2012). A
way in which to boost teacher effectiveness in the 21st century classroom is to provide professional
development in the form of workshops, conferences, webinars, graduate school offerings, and/or peer-to-peer
sharing of new research and/or technological advancements through the form of informal learning
environments, blogs, and wikis, etc. Williamson County Schools (Tennessee) will provide extensive summer
professional development for teachers to have effective strategies for BYOT implementation. (BYOT
Summer Professional Development, 2012). These strategies include collaboration among peers regarding
topics such as classroom management, personal response and communication systems, Bloom’s Taxonomy
(Churches, 2011), and useful web 2.0 tools.
3.4.2 Professional Development Recommendation
The purpose of professional development is to provide personal growth by impacting one’s knowledge,
expertise, viewpoint, and awareness. Offering appropriate and relevant professional development for teachers
may contribute to effective BYOT to implement new methodology. Integration of flipped (Hart, 2012)
classrooms, student response systems (Puente, 2012), and global connection may make learning an integral
part of one’s everyday life. Teachers will be able to engage students so school will no longer be seven hours
of Carnegie units, but rather an opportunity to provide a deep, profound, and tailored learning experience for
the student. Furthering one’s education provides a great module to students while demonstrating the
importance of lifelong learning.
3.5 Financial Plan for Sustainability (Component Five)

Financial sustainability provides the framework for schools (students) to achieve today and plan for
tomorrow. Allocating funds is essential because it enables the district to provide necessary means for follow
through of projects, plans, and integration of technology. Having sustainability in one’s district, provides not
only strong health, but also the necessary resources to flourish (Jacobs, 2010). To ensure sustainability, one
must allocate financial resources towards this concept to warrant proper bandwidth, infrastructure, personnel,
and new technology that has yet to be discovered. We must be well-prepared for the possible challenges that
come along with BYOT, but also in its ability to revolutionize the classroom as we know it. To guarantee
success, school districts and business administrators must shift funds from textbooks and other supplies and
move toward supporting technology initiatives for 21st century learning (Puente, 2012).
3.5.1 Financial Plan Example
With any new technology initiative, such as Flipped Classroom (Hart, 2012), BYOT (Quillen, 2011), and
others in the imminent future, providing sustainability is a vital component for long-term achievement. After
implementation of one-to-one devices, school districts must still monitor, maintain, and support the necessary
infrastructure for perpetual achievement. To make BYOT/BYOD/BYOC effective, one must provide
financial backing to ensure long-term success. School districts must focus on funds to meet 21st century
learning initiatives.
School districts must prudently select certain technology software/hardware as budgets tighten further and
technology expands exponentially. Districts would be wise to invest additional time in researching
software/hardware that is not only trendy, but will produce higher learning outcomes. A tool may have “bells
and whistles,” yet may not augment the student’s learning. Director of K-12 Technology (West Windsor-
Plainsboro Regional School District in New Jersey), Rick Cave (R.Cave, personal communication, April 26,
2012), believes that BYO provides an opportunity for increased student outcomes by using one’s own device,
39
ISBN: 978-989-8533-12-8 © 2012 IADIS
internet, and valuable web 2.0 tools. Maintaining the proper infrastructure and security through financial
sustainability may enable students to safely access valuable materials and create novel products for the 21st
century.
3.5.2 Financial Plan Recommendation
Allocating funds for technology is no longer considered a luxury, but rather an essential component to the
overall budget of a school district. The main purpose of BYOT is to not only provide a decrease in overhead
costs (since purchasing devices has been dropped to a nominal fee), but to provide students with various
opportunities that otherwise could not be granted in a different setting. Adopting a financial framework for
sustainability of BYOT and other new technology initiatives is essential so students have a fair and equitable
education. Technology is growing rapidly and the potential costs to any district may be exorbitant so it is
important to choose wisely to maximize funds. BYOT provides the ideal scaffold of success while
minimizing overhead costs.
4. SUMMARY AND CONCLUSIONS

The purpose of this paper is to focus on implementation of BYOT in any school district by addressing five
viable components: security measures; stakeholder involvement; policy creation; professional development,
and financial planning for sustainability.
Bring Your Own Technology (BYOT) provides the foundation for changing society into a new form. The
classroom configuration will look different as mobile integration infiltrates every aspect of one’s life,
therefore, affecting the pedagogy of learning. Learners will have a diverse, interdisciplinary, customized
global education at the disposal of one’s pocket as they access everything through their mobile devices
24/7/365. This “pocket-based” learning may revolutionize education and demand greater expectations of
critical thinking and problem-solving. Education will no longer be static, but will provide the learner with a
real, rigorous and relevant education. This research depicts how BYOT will revolutionize education and
create a new prototype of learning both inside and outside of the school setting.
With this concept and the infiltration of constructivism, students will no longer have their heads on their
desks due to boredom, but rather be active participants by transferring and applying knowledge in a plethora
of ways. Some examples include: podcasts, vodcasts, videoconferencing, displaying online portfolios,
blogging, social networking, epals, and many others. This paper addressed the problem and framework, five
components to consider with BYOT, and summary and conclusion.
4.1 Suggestions of what to expect in the Future

Through the use of new and infinite technological advancements, school is changing rapidly; therefore, our
design and approach shall also change to meet the needs of the individual student. From the classroom setting
of desks and chairs to a more collaborative approach of café style, brick and mortar perspective may no
longer be prevalent because the needs have changed drastically. In addition, the teacher’s digital blueprint for
academic achievement, shall outline a new perspective of innovation, critical thinking, problem-solving,
transference, and application of knowledge. The intent of the new model is to transform passive learning to
dynamic discovery of knowledge and skills. We shall begin to unravel universal apps compatible for all
electronic devices, a classroom truly as wide as the world through global connection, semantic web, and an
equitable learning environment where all students have the opportunity to experience the same education
despite, race, sex, gender, age, etc. For individuals who cannot afford such devices/services such as the
internet, school districts might need to consider allocating a budget for supplying school internet from home
or the ability to rent/buy devices at a lower cost. With infiltration of online information and accessibility,
cheating and plagiarism may be increased concerns.
Education in the 21st century has the capability of transforming education into something unimaginable,
yet probable. Bring Your Own Device/Technology is the first step making this change necessary, but also
possible. BYOT provides schools and businesses the ability to decrease overall costs through green initiatives
and blends technology of the person with school/work. The future may hold new dynamics of learning and
collaboration and may obliterate schoolhouse practices. In the future, tools may be worn versus carried to
specific locations (Maes, 2009). BYOT may be responsible for the shift in educational pedagogy for we will
40
not amend the existing educational infrastructure, but create an innovative, never seen before, global house
where Bloom’s taxonomy will not only occur in textbooks, but be seamlessly integrated throughout all
learning processes.
Changing the model of education will infuse the importance of the digital age and place precedence on
the art and science of learning. Students will need to know how to learn through a wide variety of media, sort
through what is essential/valid and what is not, and demonstrate digital citizenship. The student’s device has
exponential possibilities of knowledge, yet the role of an educator is to provide the framework that the digital
world cannot produce. Profound facilitation, questioning, customization, and feedback will change the setting
of schools as we know them today while providing an incalculable opportunity for student learning both
nationally as well as globally. Having access to a myriad of resources enables all learners the opportunities
for growth and constant reflection aspiring for excellence. BYOT is not just another policy or guideline that
one needs to follow in education, but is the means by which education will forever be transformed.
4.2 Limitations of Paper

BYOT is a new initiative for technology integration and has a paucity of research regarding the effectiveness
on learning outcomes. BYOT is an innovative approach of joining one’s personal technology with school
technology; therefore, lacking strong statistical data. In addition to the absence of quantifiable results in
current BYOT pilots, websites/blogs are popping up with potentially biased treatments of BYOT. For
example, blogs may appear to be providing information to advance the practitioner, yet a closer examination
suggests a commercial intent to promote the sale of their products/publications, often based on contributions
to the site.
REFERENCES
Bring Your Own Computer (BYOC). (n.d.). BYOT.ME. Retrieved from http://byot.me
Churches, A. (2011). Educational origami – Bloom’s digital taxonomy. Retrieved from
http://edorigami.wikispaces.com/#Bloom's TaxonomyFayette County Schools. Protocol for the use of personal
technology
Fayette County Schools B.Y.O.T pilot program. Retrieved from
http://www.fcboe.org/files/technology_services/BYOT_use_contract_pilot.pdf
Giordano, M. (2011). Williamson students can bring own technology to school. Retrieved from
http://www.tennessean.com/article/20110829/WILLIAMSON01/111229024/08-29-2011-Williamson-students-can-
bring-own-technology-school
Harris, C. (2012). Going mobile. School Library Journal. 58 (1), 14.
Hart, M. (2012). The expanding school day. T H E Journal. 39 (3), 6.
Jacobs, H. H. (2010). Curriculum 21: Essential education for a changing world. Alexandria, VA: ASCD
Maes, P (2009). Demo sixth sense. Ted Talks. Retrieved from
http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html
McCaney, K. (2012). 9 keys to making BYOD work. Retrieved from http://gcn.com/articles/2012/05/10/9-best-
practices-for-byod.aspx
Norris, C., & Soloway, E. (2011).Tips for BYOD k12 programs. District Administration. 47 (7), 77.
Prensky, M. (2010). Teaching digital natives, partnering for real learning. Thousand Oaks, California: Corwin Press.
Puente, K. (2012). High school pupils bring their own devices. District Administration; 48 (2), 64.
Quillen, I. (2011). Crafting your BYOT policy. Digital Directions. 23.
Stanley, C. (2012). At one school district, the motto is BYOT - bring your own technology. Retrieved from
http://dailynightly.msnbc.msn.com/_news/2012/05/06/11567170-at-one-school-district-the-motto-is-byot-bring-your-
own-technology
Sucre, G. Three BYOD benefits (2012). Learning & Leading with Technology, 39 (5), 8.
Tablet (n.d.). Pc mag online. http://www.pcmag.com/encyclopedia_term/0,2542,t=tablet+computer&i=52520,00.asp
Ullman, E. (2011). The new one-to-one. Technology & Learning, 31 (7), 54-57.
Ullman, E. (2011). BYOD and security. Technology & Learning, 31 (8), 32-36.
Williamson County Schools. BYOT summer professional development offerings for teachers 2012. Retrieved from
http://www.wcs.edu/byot/Summer%20Offerings%20Teachers%202012.pdf
41
ISBN: 978-989-8533-12-8 © 2012 IADIS
HOW PRIOR KNOWLEDGE AND COLOUR CONTRAST

INTERFERE VISUAL SEARCH PROCESSES IN NOVICE
LEARNERS: AN EYE TRACKING STUDY
Duygu Sönmez1, Arif Altun2 and Sacide Güzin Mazman2

1
Hacettepe University, Department of Elementary Education Division of Science Education, Beytepe, Ankara/TURKEY
2
Hacettepe University, Department of Computer Education and Instructional Technologies, Beytepe, Ankara/TURKEY
ABSTRACT
This study investigates how prior content knowledge and prior exposure to microscope slides on the phases of mitosis
effect students’ visual search strategies and their ability to differentiate cells that are going through any phases of mitosis.
Two different sets of microscope slide views were used for this purpose; with high and low colour contrasts and students’
ability of recognition were investigated. Study group was consisted of forty undergraduates majoring in science teacher
education at a major university in Turkey. All participants were considered novice microscope users based on their
ability. Participants prior content knowledge was assessed with “Mitosis Inventory” and they were grouped based on their
level of prior content knowledge. High and low level content knowledge groups took part in the study. Based on the
results of the eye-tracking data; prior content knowledge was found to have an effect on participants’ visual search
strategies and their recognition time for the slide samples that were in high colour contrast. High prior knowledge group
was able to recognize and identify the phases of mitosis correctly in a shorter period of time in comparison to low level
prior knowledge group. However, no difference was found between groups for the low colour contrast slides.
KEYWORDS
Visual search, prior content knowledge, prior exposure, biology instruction, eye tracking, colour contrast
1. INTRODUCTION
In science education, use of visual materials to support verbal communication is a common practice. This
way, the success of information transfer increases especially for abstract concepts. Visual representations are
often used and appreciated in science education including diagrams, photographs and illustrations (Lee 2010)
and they have a significant use in mainstream K-16 science education. Especially in biology education,
microscopes and in astronomy education telescopes are vital instruments to provide visual information for
materials that are not visible to naked eye, such as detailed visual information of cells and chromosomes
provided by microscopes. Unfortunately, in most cases; students’ ability to interpret visual materials is not
questioned during teaching. Students’ literacy level is a determinant on interpretation of materials. Novice
learners are reported to struggle and be less successful in determining the relevancy of information during the
interpretation of visual materials (Patrick, Carter & Wiebe, 2005).
With the increasing use of visual representations in science education, visual literacy and visual search
strategies emerge as the vital skills in recognition and interpretation of visual materials at individual level.
Visual search tasks are commonly used both as a tool and a method to investigate how attention influences
visual information processing, namely how the visual system extracts and combines features (Zhuang and
Papathomas 2011). It has been stated that the factors that increase or decrease efficiency of visual search
could be categorized into image driven factors (i.e. saliency, single features, colour contrast, abrupt onset
luminance contrast) and knowledge based factors (i.e. familiarity effect, novelty, experience, automaticity,
pre-knowledge) (Chun and Jiang 1998). Furthermore, it has been stated that human visual attention and eye
movements are in a close correlation and analysing eye movements would provide objective measures of
common attentional processes (Hoffman and Subramaniam, 1995; Frey et al. 2011). Since eye movement
data can provide information on how users receive and process visual information, general trend in literature
42
has been to identify differences among novice and experts’ performance with field related visual materials
through eye metrics data.
Based on the data provided in literature, an experimental study was designed to explore the visual search
strategies of novice learners in relation to their content knowledge level and colour contrast variations in
virtual representations. More specifically, this study was designed to explore how undergraduate students’
visual search strategies differ based on their prior content knowledge and the microscope slides’ colour
contrast variations when they were presented the same content information. In this case, the content
comprised of microscope slides of onion root cells that are going through various stages of mitosis.
1.1 Visual Search

Visual search is defined as a paradigm in which a person looks for a target item among a set of distracter
items (Wolfe, 1994) “to determine, as rapidly as possible, whether a target item is present or absent in a
display” (Enns and Rensik, 1990, p. 323). This process involves two elements, a targeted item and distracters
(Leonards et al. 2002). In visual search literature, it has been suggested that there two kinds of visual search
types. These are “feature search” and “conjunction search” (Wolfe 1994; Old and Fockler 2004; Zhuang and
Papathomas, 2011).
In single feature search tasks, the target differs from distracters on the basis of a single feature dimension
(e.g., colour, shape, size) and reaction time and accuracy to search for the target remain generally unchanged
even though the number of distracters increases in a feature search task. By contrast, in a conjunction search
task, target differs from distracters on the basis of a conjunction of two or more feature dimensions (e.g.,
colour and shape) and reaction time increases while accuracy decreases by the increase in the number of
distracters (Old and Fockler, 2004; Zhuang and Papathomas, 2011). In their Feature Integration Theory,
Treisman and Gelade (1980) suggested that conjunction search occurs in two stages; the first stage of visual
search is a parallel mechanism where information about one single feature dimension such as colour, size, or
orientation can help in the search; each type of feature is represented by a separate feature map in the brain.
The second stage of the model is a serial mechanism where each item must be considered in turn to
determine whether it satisfies each required feature. When targets and distracters in visual search differ by an
easily distinguishable feature such as colour, size, or orientation the reaction times decreases. On the
contrary, for targets that “are not easily discriminable or differ from distracters by a conjunction of features
(e.g., a particular combination of colour and shape) display size has a large effect on search times and the
reaction time slope is steep” (Trick and Enns 1998, p.350).
Ngyren (1996) found that if the target stimulus is visually conspicuous then search performance could be
up to 83% faster. One of the dimensions of conspicuity is differences of background colour or luminance and
stimulus colour or luminance. Colour is an influential factor of graphical display and when used
appropriately can lead to faster search times; conversely, poor use of colour can lead to decreases in
performance (Ling and Schaik 2002, p. 224). Schaik and Ling (2001) reports that colour combinations with
higher level of background and foreground contrast lead to better performance in comparison to lower
contrast. In addition to colour contrast, increasing luminance contrast is also reported to be influential on
visual search process and it was found that the mean fixation duration increases clearly with decreasing
luminance contrast (Ojanpaa and Nasanen, 2003). Since Donk and Theeuwes (2001) have argued that
luminance changes are necessary for marking to occur, Olds and Fockler (2004) suggested to further study of
the question whether visual marking could occur with the present equiluminant stimuli. Background
complexity is also suggested to effect visual information processing process and fixation duration and length
of saccades (Crosby, Iding and Chin, 2001). Increase in the background complexity will also result in
increase in information processing time and fixation duration.
Apart from those image-design driven factors (background contrast, luminance, conspicuity etc.)
individual factors i.e. prior content knowledge, prior exposure, visual ability, have an effect on visual search
performance. For example, success in identification and recognition of microscope display is found to be
depending on integration of general knowledge, visual knowledge, high level recognition and reasoning
(Pani, Chariker and Fell 2005) and content knowledge is reported to be an important component during
visual processing. Moreover prior exposure to an object is suggested to improve the recognition during the
upcoming encounters. The effect of this prior exposure is called “priming” and stated to improve one’s
ability to identify an item or an object (Schacter and Buckner 1998), not requiring conscious effort or explicit
43
ISBN: 978-989-8533-12-8 © 2012 IADIS
knowledge (Kristjansson, Wang and Nakayama 2002). Existing research focuses on visual search with the
purpose of identifying specific patterns to support the learning process. Studies conducted with the
individuals from different professions shows differences in visual search performance generally based on
expertise level, prior exposure and visual display design characteristics. Various studies (i.e. Cook, Wiebe
and Carter, 2008) point out the differences in interpretation of graphics among students with high and low
level of content knowledge as well as among complex designed displays and well-designed displays. Results
of such studies show a difference favouring experts including identification of visual images or interpretation
of those (Krupinsky et al. 2006; Pani, Chariker and Fell, 2005) and also favouring well designed visual
displays’ compensatory effect (Canham and Hegarty, 2010). Experts in various fields learn to recognize
specific patterns and make connections to the prior knowledge (Pani, Chariker and Fell, 2005).
Also eye movement studies investigating performance among various expertise levels support this
difference, indicating that experts perform visual search with fewer saccadic movements with more accuracy
in comparison to less trained participants (Krupinski et al. 2006; Leung et al. 2007). In addition, literature
supports Henderson’s (2003) suggestion that prior knowledge has a significant role on eye metrics such as
fixation. Canham and Hegarty (2010) reports on a study focusing on eye fixations with visual displays which
shows that with declarative knowledge subjects focus on more to relevant information rather than irrelevant
information. Understanding students’ interpretation of microscope slides in relation to their level of content
knowledge and their expertise level, which is developed through repetition, holds a merit for the future use of
such visual materials in education. Moreover, understanding the effect of prior knowledge and prior exposure
with or without a well-designed environment is also important in terms of science education.
1.2 Research Questions

Shedding light on how students conduct visual search and interpret microscope slides in relation to their level
of pre-existing content knowledge can help in the use and development of visual aids to support learning
process in education. Therefore, this study targets novice individuals who are lack of repetition and expertise
in interpreting microscope slides and investigates their visual search processes with the following questions;
1. Do the eye movements during the visual search task in different background colour contrast differ in
terms of individual’s prior knowledge level?
a. Do the eye movements during the visual search task in high colour contrast images differ in terms of
individual’s prior knowledge level?
b. Do the eye movements during the visual search task in low colour contrast images differ in terms of
individual’s prior knowledge level?
2. Do the eye movements during the visual search task in different background colour contrast differ in
terms of having prior exposure to microscope view?
a. Do the eye movements during the visual search task in high colour contrast images differ in terms of
having prior exposure to microscope view?
b. Do the eye movements during the visual search task in low colour contrast not images differ in terms
of having prior exposure to microscope view?
3. Is the interaction effect between prior knowledge and prior exposure significant?
2. METHODOLOGY
2.1 Participants
A total of ninety-eight, 3rd and 4th year undergraduate students took part in the study. All participants were
pre-service elementary science teachers who completed at least two general biology courses and had the
basic skills of microscope use.
Prior content knowledge test on mitosis and its phases was administered to those 98 students to identify
their level of prior content knowledge. Their scores were ranged from 0 to 10 with a M=5.67 mean score and
sd=2.46. Participants were grouped based on their prior content knowledge levels on the phases of mitosis as
high, medium or low level groups. According to their prior content knowledge scores, 40 students were
appointed to either high or low level groups, 20 students to each group. High and low level groups were
44
formed in regard to prior test scores (+/-) 1 standard deviation. Further data collection was conducted with
the participation of students who were placed in either low or high level content knowledge groups. For the
purpose of this study, all participants were novices in terms of their experience of using microscopes.
Demographic characteristic of the study group is presented in Table 1.
Table 1. Demographic characteristic of the participants
Demographic Characteristics F Percentage (%)

Female 28 70%
Gender
Male 12 30%
3 28 70%
Grade
4 12 30%
Low 20 50%
Prior Knowledge
High 20 50%
Yes 20 50%
Prior exposure to microscope view
No 20 50%
2.2 Data Collection and Instruments

The prior knowledge was measured by a content test consisting of 12 multiple choice questions which was
developed by a field expert who is also one of the researchers. Two more field experts outside of the study
were asked to examine the questions for content validity. Questions were revised according to their
suggestions. The test was administered to ninety-eight pre-service elementary science teachers for item
analysis and reliability studies. The analyses yielded two questions out of 12 to be extracted from the test
since the item discrimination indices for these were below 0.3. The item discrimination indices for the
remaining questions ranged between 0.32 and 0.66. The reliability of the test was assessed with KR-20
reliability coefficient and it was found to be .717. Each correct answer was scored 1 point and wrong answer
was scored 0 point. The maximum score of the test was 10 and the minimum was 0.
2.3 Data Analysis

A 2x2 within subject design is used for the purpose of this study. Participants were grouped by their level of
prior content knowledge and their statement on either having a prior exposure to slides showing phases of
mitotic division with the consideration of priming effect. During this study participants were shown a series
of images consist of onion root cells. Students’ task was to identify any cell/s that is/are undergoing through a
phase of mitosis then click on the cell with the cursor and select the correct mitotic phase from multiple
choices provided on the right side of the screen. During this task; participants were expected to go through
the following steps. 1- conducting visual search 2- recognizing cell/s that are undergoing mitosis 3- recalling
pre-existing knowledge and identifying the mitosis phase.
Based on prior knowledge test scores, 40 students (20 students from low prior knowledge group and 20
students from high prior knowledge groups) were selected among ninety-eight students. Experimental
sessions were executed individually for each participant on an eye tracking computer accompanied by a
researcher. Data collection session lasted from 10 to 15 min. The eye tracker was recalibrated for each
participant before starting the data collection procedure and at the beginning an instruction was displayed
including an audio-visual example. The eye track data were stored separately for further analysis.
2.4 Experimental Setting

Seven different microscope images of cells were displayed on a computer screen for each question. The
microscopes slide images were grouped in two sets; high and low contrast. High contrast images were from
slides that are stained with four different stains and had maroon coloured genetic material on a light blue cell
background. Four of the seven questions were placed in this first set. Second set included three low contrast
slides with orange coloured genetic material on a yellow cell background. The first question set was obtained
45
ISBN: 978-989-8533-12-8 © 2012 IADIS
from the internet and the second question set was obtained from microscope slides prepared by a field expert.
A sample question from each of the question set was presented in Fig. 1.
Figure 1. Sample question from different level background colour contrasts

All of the cell images were transformed to an experiment set by Tobii software on eye tracker. Tobii T120
Eye tracker was used to record eye movement data by two binocular infra-red cameras integrated within the
panels of the monitor. The eye tracker was paired with 17” LCD monitor with 1280 X 768 resolutions, 120
Hz sampling rate and an accuracy of 0.5°. Among the eye movement metrics, fixation duration and time to
first fixation were obtained from eye tracker for this study. Fixation duration is the amount of the time that a
particular element of a design is viewed. Time to first fixation provides to measure how long it takes before a
user recognizes a specific target. Undergoing mitosis phases on the cells were drawn as area of interests.
Those area of interests were the areas which students required to find and recognize the right mitosis phases.
Participants’ eye metrics data were recorded during the process and eye movement data on those areas were
collected to analyse. Figure 2 shows the area of interest samples from each question sets.
Figure 2. Area of interest samples from different level background colour contrasts questions
The images of the cells were presented on the left side of the screen while multiple choices responses of
mitotic phases were placed on the right side as seen in figure 1. Participants were asked to find the cell that is
undergoing through any mitotic phase and identify the cell by clicking on it with the curser and then to select
the name of the mitotic phase among multiple choices presented on the right side of the screen. Participants
were informed that each question could contain more than one cell vision, so they are allowed to select more
than one cell and a phase consecutively. Each of the questions was displayed 20 seconds on the screen. If
participants did not click on the forward button, the next question was displayed automatically after 20
seconds.
3. RESULTS
Statistics regarding dependent variables (fixation duration and time to first fixation) of low level prior
knowledge-high level prior knowledge and having prior exposure-not having prior exposure in different
background colours are presented in Table 2.
46
Table 2. Eye movement statistics across prior knowledge and prior exposure groups
Low level prior knowledge High level prior knowledge

Prior exposure No Prior exposure Prior exposure No Prior exposure
(n=10) (n=10) (n=10) (n=10)
M 1.46 1.29 2.79 2.81
Fix.Dur. Sd .806 .764 1.94
1.27
Blue
M 7.04 7.52 4.66 3.98

T_FirstFix.
Sd 5.2 5.07 3.61 4.33
M .818 .946 1.28 .889
Fix.Dur.
Yellow
Sd .728 .813 .852 .697

M 4.93 3.52 3.84 5.18
T_FirstFix
Sd 3.24 2.87 2.31 5.22
Descriptive analyses indicate that for high contrast images, students with higher content knowledge (a)
fixate on the cells faster in comparison to students with lower content knowledge and (b) have a longer
fixation period of time on the images. For further analysis, a two-factor MANOVA was conducted to
examine eye movement differences in different colour contrasted images across prior knowledge and prior
exposure factors.
Before the analyses, normality and homogeneity assumptions of covariance matrices were tested.
Normality assumption was controlled with skewness and kurtosis value. Kurtosis value ranged from -.5 to 1.6
and skewness value ranged from .83 to 1.38. These results confirms the normality assumption is not violated
(-10<kurtosis<10, -3<skewness<3) (Kline 2003). Homogeneity of covariance matrices assumption is tested
with Box M’s test and it was found that (Box’s M = 54.049, F=1.428, p = .062). Since the value is not
significant at 0.05 level, it was accepted that covariance matrices of groups are homogenous.
MANOVA analysis was executed to determine whether there were significant differences in time to first
fixation and fixation duration across prior knowledge and prior exposure in blue background (high contrast)
and yellow background (low contrast) questions. The main effect of prior knowledge was found significant
(Wilk’s Λ = . 677), F(4, 33) = 3.932, p =.010, ηp2 = .323). However neither the main effect of prior exposure
to microscope view (Wilk’s Λ =. 989), F(4, 33) = 0.91, p = .985, ηp2 = .011) nor the interaction effect of prior
knowledge and prior exposure to microscope view (Wilk’s Λ = . 880), F(4, 33) = 1.129, p =.369, ηp2 = .120)
were found significant.
In addition, between subject effects of prior knowledge was also examined. It was found that high level
prior knowledge group was fixated on the area of interests (time to first fixation) significantly in a shorter
time than low level prior knowledge group in blue background questions (high contrast); F(1, 36)=4.14,
p=.049, ηp2 = .103. In addition, in the blue background (high contrast) questions, total fixation duration of
high level prior knowledge group was found to be higher than their low level counterparts F(1, 36)=12,183,
p=.001, ηp2 = .253. However, in yellow background (low contrast) questions, no statistically significant
difference between prior knowledge level groups was found neither in time to first fixation F(1, 36)=.064
p=.801, ηp2 = .002; nor in total fixation duration in the area of interests F(1, 36)=.714, p=.404, ηp2 = .019.
4. CONCLUSION
This study investigated the effect of prior knowledge and prior exposure in different colour contrasted
microscope images. The images used as questions in the study were designed in two different colour
contrasts, high contrast; maroon targets on blue background versus low contrast; orange targets on yellow
background, to examine whether the colour contrast facilitate the recognition process in visual search tasks.
The findings of this study revealed out that for high colour contrast image group prior knowledge has an
effect in visual search tasks both in time to first fixation which shows recognition and fixation duration on the
related areas. This finding is consistent with individual differences principle of Mayer (2001) in the
multimedia design, which states that prior knowledge has a compensating effect on producing own mental
models in multimedia learning (Mayer and Moreno 1998). In high colour contrast questions, high level prior
knowledge group recognized area of interest in a shorter time and fixated on correct areas longer than other
areas. This finding is also consistent with prior research which reports that combinations of colours that had
47
ISBN: 978-989-8533-12-8 © 2012 IADIS
higher levels of contrast between background and foreground and also higher prior knowledge generally led
to better performance than combinations of lower contrast and lower prior knowledge (Ling and Schaik
2002; Theeuwes, Reimann and Mortier 2006; Huang 2008; Zhuang and Papathomas 2011).
On the other hand, visual performance of high level prior knowledge group and low level prior
knowledge group was found to be similar on low contrast images, contrary to the high contrast ones.
Nevertheless, based on the findings of this study it is difficult to suggest that prior knowledge effect has
disappeared in low contrast background microscope images. It should be noted that time to first fixation
occurs in a shorter period of time, and the difference is greater for low level prior knowledge group with no
prior exposure between high and low contrast pictures (M=3.52, sd=2.87 vs. M=7.52, sd=5.07). Although
this was not a main research question for this study, it may require an explanation. Remember that time to
first fixation and fixation duration were the two eye tracking metrics used in this study. As a follow up to
explore this difference, data were revisited and the length of first fixation duration metric results were also
computed in order to explore whether these first fixations would be taken as an evidence of visual search or a
mere scanning process. The length of first fixation durations were shorter in low contrast pictures (M=0.32,
sd=0,1) than high contrast pictures (M=0.5, sd=0.3). Those studying the low contrast pictures had a quicker
fixation but less time spent whereas those studying the high contrast pictures had relatively fixated later but
stayed longer on the areas of interests. It can be speculated that their first fixation could just be interpreted as
an indication of mere eye scanning on the picture. Since visual search and scanning are different cognitive
processes, additional data are required to be able to predict if the prior knowledge affect disappeared or not.
Furthermore, as Henderson and Hollingworth (1998) once stated, fixation duration is an indicator of the
cognitive complexity. Based on the findings in this study, it can be speculated that low contrast images might
increase the cognitive complexity for participants. Another interpretation could be the presentation order of
the pictures. The participants were first presented the high contrast images followed by low contrast images
so these results might be due to the familiarity effect. Yet, further studies investigating the role of fixation
duration with counter balanced designs are required to be able to draw more solid conclusions.
Individual differences (i.e. visual ability, prior knowledge, visual attention, working memory, visual
perception) are important in understanding the visual materials and also in learning from multimedia
environments. However, effect of individual differences could be sensitive to the design of the learning
environment. Although people could benefit from their positive individual characteristics in well-designed
learning environments, ever so novel or familiar, as found by Barry and Lazarte (1995), increase in the
complexity of material would diminish the advantage of previous knowledge. Reversely, well designed
display designs found to be facilitating comprehension of students who has low prior knowledge (Canham
and Hegarty, 2010). Further research could explore the effects of other individual differences such as gender,
age, and distinctive socio-cognitive characteristics on visual processes in multimedia environments.
In conclusion, the role of visual representations cannot be ignored in science education. This study
investigated visual search process by the measurement of eye movements which is an objective measurement
of cognitive processes. The results revealed that for novices, who are lack of expertise in interpreting
microscope images, such as participants of this study, prior knowledge has an effect when high contrast
images were in question. Findings of this study can be useful for both teachers and instructional designers in
science education. Instructional designers should consider design principles together with individual
differences when developing visual displays. Teachers also should consider individuals’ abilities and
properties of visual materials when choosing instructional materials.
However, findings are inconclusive for the low colour contrast image group due the lack of qualitative
measures as verbal protocols or cognitive task analysis and further studies are recommended. In addition, this
study was limited to 40 students at the undergraduate level. Studies with larges samples would be needed to
confirm the findings across grades and cultural settings.
REFERENCES
Barry, S. & Lazarte, A. A. (1995). Embedded Clause Effects on Recall: Does High Prior Knowledge of Content Domain
Overcome Syntactic Complexity in Students of panish? The Modern Language Journal. 79 (4), 491-504.
Canham, M., & Hegarty, M. (2010). Effects of knowledge and display design on comprehension of complex graphics.
Learning and Instruction. 20 (2), 155-166.
48
Chuang, H.-H., & Liu, H.-C. (2011). Effects of Different Multimedia Presentations on Viewers’ Information-Processing
Activities Measured by Eye-Tracking Technology. Journal of Science Education and Technology, 1-11.
Cook, M., Wiebe, E. N. & Carter, G. (2008). The influence of prior knowledge on viewing and interpreting graphics with
macroscopic and molecular representations. Science Education, 92: 848–867. doi: 10.1002/sce.20262.
Crosby, M., Iding, M., & Cin, D. (2001). Visual search and background complexity: Does the forest hide the trees? In
M.Baier, P. J.Dmytrasiewcz, & J.Vassileva (Eds.), Proceedings of the 8th International Conference on User
Modeling, Sonthofen, Germany, July, 225–227. Berlin, Germany: Springer.
Chun, M. M. and Jiang, (1998). Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial
Attention. Cognitive Psychology. 36. 28-71.
Donk M. & Theeuwes J, (2001). Visual marking beside the mark: prioritizing selection by abrupt onsets. Perception &
Psychophysics. 63. 891- 900.
Enns, J.T. & Rensink, R.A. (1990). Influence of scene-based properties on visual search. Science. 247, 721–723.
Frey H., Wirz K. T., Willenbockel V. , Betz T., Schreiber C., Troscianko T. & Konig P. (2011). Beyond correlation: do
colour features influence attention in rainforest? Frontiers in Human Neuroscience. 5 (36).
Henderson, J. M. (2003). Human gaze control during real-world scene viewing. Trends in Cognitive Sciences, 7, 498-504.
Henderson, J. M., & Hollingworth, A. (1998). Eye movements during scene viewing: An overview. In G. Underwood
(Ed.), Eye Guidance in Reading and Scene Perception (pp. 269-293). Oxford: Elsevier
Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception &
Psychophysics. 57, 787–795.
Huang, K. C., (2008). "Effects of computer icons and figure/background area ratios and color combinations on visual
search performance on an LCD monitor." Displays, 29(3). 237-242.
Kristjánsson, Á., Wang, D. & Nakayama, K. (2002). The role of priming in conjunctive visual search. Cognition. 85(1).
Krupinski E, et al. (2006). Eye-movement study and human performance using telepathology virtual slides. Implications
for medical education and differences with experience. Human Pathology. 7. 1543-56.
Lee, V. R. (2010). Adaptations and Continuities in the Use and Design of Visual Representations in US Middle School
Science Textbooks. International Journal of Science Education. 32(8). 1099-1126.
Leonards, U. et al. (2002). Perceptual learning of highly demanding visual search tasks. Vision Research. 42(18).
Leung J.W, et.al. (2007). Performance parameters for screening and diagnostic mammography in a community practice:
are there differences between specialists and general radiologists? American Journal of Roentgenlogy.188(1).236–241.
Ling, J. & Schaik, P. Van. (2002). The eﬀect of text and background colour on visual search of Web pages, Displays. 23.
Mayer, R. (2001). Multimedia learning. Cambridge, UK: Cambridge University Press.
Mayer, R. E. & Moreno, R. (1998). A Cognitive Theory of Multimedia Learning: Implications for Design Principles .
Paper presented at the annual meeting of the ACM SIGCHI Conference on Human Factors in Computing Systems,
Los Angeles, CA.
Nygren,E. (1996). From paper to computer screen: human information-processing and user interface design,
Comprehensive Summaries of Uppsala Dissertations from the Faculty of Science and Technology, 188. Almqvist &
Wiksell, Stockholm.
Ojanpaa H. & Näsänen R. (2003). Effects of luminance and colour contrast on the search of information on display
devices. Displays. 24, 167-78.
Olds, E. S., & Fockler, K. A. (2004). Does previewing one stimulus feature help conjunction search? Perception, 33.
Pani, J. R., Chariker, J. H., & Fell, R. D. (2005). Visual cog-nition in microscopy. In B. G. Bara, L. Barsalou, & M.
Bucciarelli (Eds.), Proceedings of the XXVII Annual Conference of the Cognitive Science Society, 27, 1702-1707.
Patrick, M. D., Carter, G., & Wiebe, E. N. (2005). Visual representations of DNA replication: Middle grades students‘
perceptions and interpretations. Journal of Science Education and Technology, 14(3), 353–365.
Schacter, D.L. & Buckner RL. (1998). Priming and the brain: review. Neuron. 20. 185–95.
Schaik, P. Van. & Ling, J. (2001). The effects of frame layout and differential background contrast on visual search
performance in Web pages. Interacting with Computers. 13(5).513-525.
Theeuwes, J., Reimann, B. & Mortier, K. (2006). Visual search for featural singletons: No top-down modulation, only
bottom-up priming. Visual Cognition. 14. 466-489.
Treisman, A. M. & Gelade G., (1980). A feature-integration theory of attention. Cognitive Psychology. 12. 97- 136.
Trick, L.M. & Enns, J.T. (1998). Lifespan changes in attention: visual search. Cognitive Development, 13, 369–386.
Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin and Review, 1. 202-238.
Zhuang , X. & Papathomas, T. V. (2011). Cue relevance effects in conjunctive visual search: Cueing for location, colour,
and orientation. Journal of Vision. 11(7). 6, 1–13.
49
ISBN: 978-989-8533-12-8 © 2012 IADIS
THE IMPACT OF ICT AS ANOTHER ROUTE TO

OVERCOME LEARNING BARRIERS FOR STUDENTS
WITH SEN: A CASE STUDY IN AN EGYPTIAN CONTEXT
Al-Gawhary, Wedad and Kambouri, Maria

Institute of Education, University of London
ABSTRACT
The purpose of this case study was to measure the impact of using ICT in Individual Learning Programmes of students
with learning disabilities. Twenty five students and thirteen teachers took part in the research which was based on
classroom observations. The Kappa coefficient was employed as a measure to statistically quantify the students’
acquisition and progress in learning computer and literacy skills through raters’ observations. A high correlation between
Keyboard, PowerPoint, and literacy skills (writing, reading, and maths) was revealed suggesting the significance and
impact of keyboard training as an essential component towards achieving educational objectives. Moreover, thematic
analysis of the teachers’ interviews provided a further dimension for understanding factors influencing the ICT
integration process. These results further confirmed the positive impact of computer skills training on the students’
learning experience which also revealed a strong belief in the right of every child to all learning opportunities,
particularly ICT regardless of his/her abilities.
KEYWORDS
ICT, Special Education Needs (SEN), measurement, observation, acquisition, skills
1. INTRODUCTION
The formula for the effective integration of any type of Information and Communication Technology (ICT)
in education is not the technology in itself, but rather the perception of why, how and whether it can
invigorate one’s learning. Several researchers have recently presented new insights into the effect of ICT on
education but mainly focusing on mainstream environment. Unfavorably, the area of ICT and learning
difficulties is not yet extensively researched and necessitates more peer-reviewed body of literature in order
to build in-depth knowledge of the optimum means of using ICT effectively with Special Education Needs
(SEN)s (Abbot, 2007).
2. RESEARCH CONTEXT
The research was conducted in a center specialized in children with SEN education who were not
mainstreamed due to the severity of their needs and will be referred to throughout the research as ‘Center’.
The Center operates in Cairo, Egypt and caters services to Egyptian and international students, 2 years to 21
years old, with autism and other disorders that impact on their communication, physical, mental or social
development. Students are grouped according to both age and skill level in ten classes where instruction is
conducted in Arabic except for two English instruction classes. Instruction is conducted in one-to-one and
small group sessions, incorporating each student's Individualized Educational Plan (IEP) objectives through
activities that promote cognitive development, language, motor skills as well as academic skills. A computer
department was structured four years ago in order to integrate computer sessions into students’ IEPs where
computer skills are taught on a 1:1 basis at different levels to approximately 37 out of 60 students who
qualified both physically and mentally for this task. Computers are equipped with regular screens, keyboards
and mouses as well as external speakers.
50
3. METHODOLOGY
The nature of the impact of integrating ICTs with the IEPs of students with SEN was the main question that
this research aimed to answer. In particular, the following questions were processed into the research on
what type of ICT was employed, how to measure the ICT skills of students with SEN if achieved, and what
were the factors behind the development of this practice.
3.1 Participants
An initial selection of 37 students at the Center identified upon an interview with the senior computer teacher
who agreed to include all students currently enrolled for computer sessions. However, 25 students only were
observed and rated for performance because of four parent refusals, and eight absences during the
observation period. Due to the distinctive nature of the Center students’ disabilities and broad age range, it
was not possible to select a specific age range for the study and hence this sample was included as a unique
heterogeneous group of students. At the sample selection phase of the research, a plausible grouping of the
students was made according to their formal assessments and also, due to the severity of cases, ability took
more precedence than age in the students’ assignment to different classes. Therefore, Group A included
autism diagnosed students with variable severity: N=16 mean age 12;5 (range 4;5- 24;10) 13 boys and 3
girls; Group B included Down Syndrome diagnosed students: N=4, mean age 10;9 (range 6;8-14;7) 2 boys
and 2 girls; Group C included Cerebral Palsy diagnosed students with variable severity: N=3, mean age 12;5
(range 11;1-14;9) 2 boys and 1 girl; Group D included one 3rd chromosome deficiency diagnosed boy age
6;5, and finally Group E included one moderate developmental delay diagnosed boy age 21;8. Ten class
teachers, three computer teachers, a Center coordinator, and four parents were invited respondents for the
interviews.
3.2 Data Collection Methods

A pilot interview with the Center coordinator and senior computer teacher showed that computer skills were
taught with the objective of setting up the Center students to be computer users. Hence, due to the
unavailability of sufficient records containing information about students’ performance in computer sessions,
some were partially developed in response to the researchers’ request. Students’ background information,
formal assessment of students’ skills in relation to dealing with different types of ICT, reasons for approval
or rejection to join specialized computer sessions, task dates and specific measurement of performance
throughout the period of computer training, progress data, sessions’ observations, and more were examples of
required and requested data for establishing a clear history and also for comparative objectives over different
time periods. On the other hand, the available data representations were a comprehensive checklist of
sequential categorized computer skills used for an informal evaluation of each student’s current computer
skills before starting sessions in addition to a non-descriptive register of progress in yearly reports. A mixed
design approach combining the application of both qualitative and quantitative research methods (Johnson &
Onwuegbuze, 2004; Hanson et al., 2005) was adopted to explore the current practice of integrating computer
sessions in students’ IEPs and register progress whenever verified and thus demonstrate the impact of
training students with SEN to be computer users as an ICT intervention.
3.3 Kappa Measure of Agreement on Learned Skills

Selection of the quantitative pole in the mixed design aimed at obtaining a quantified measurement of the
practice investigated. One major preoccupation of quantitative method is measurement as classified by
Bryman (2001) which generates consistent results and provides basis for comparative measurement over
periods of time as well as among other researchers. To utilize this concept, the Fleiss (1981) Kappa
Coefficient was selected to measure the agreement of more than two raters on the occurrence of progress in
learning computer skills for the Center’s students with SEN. This measurement corroborated the occurrence
of progress as acknowledged by the teachers in the interviews and enhanced the internal validity of the
51
ISBN: 978-989-8533-12-8 © 2012 IADIS
findings. It also served as a baseline for prospective measurements over periods of time to capture
anticipated developments and/or limitations to this practice.
3.4 Procedure
Based on a comprehensive list of the achieved computer skills per student, a rater sheet was developed for
each student in the sample with the specific skills expected to be observed and rated by the judges. The rater
sheet divided the skills on the y-axis into various categories: Mouse Control, Keyboard manipulation, Word,
Paint, PowerPoint, Internet Explorer/email, and academic objectives. The academic objectives were selected
by computer teachers from the annual multi-departmental IEPs based on their applicability for drilling using
software programs. On the x-axis, each skill was evaluated upon various attributes: Independent without
support, independent with verbal support, independent with physical support, not achieved, compliance, and
motivation. A general performance scale of 1-2-3 was assigned for all attributes except for an ‘X’ for not
achieved where 1=30%, 2=50%, and 3=100% to prevent binary evaluation and add more depth into the
measurement analysis. Four raters were selected: the senior ICT teacher, the Center coordinator who is also
a professional SEN tutor, and the researchers. A meeting for all raters to explain the attributes and scale was
held before the five pilot observations. A cross validation meeting was held after the pilot observations and
raters commented on the limitation and ambiguity of the performance scale, which was agreed to be changed
to: 1= 0-30%, 2=30%-70%, and 3=70%-100%. The senior teacher also commented that his pre-knowledge
of the students’ performance interfered with his judgment and that he tended to be tougher, so, the objective
of rating was re-explained emphasizing the need for fairness and practical accuracy. It was possible to re-
observe the same students later, which yielded the same results for three raters but with variation towards
higher rating in the teacher’s sheet. In reflection, post-observation meetings proved to be valuable in
maintaining the same perspective for all raters which were held consistently until closure of observations.
Calculating the kappa coefficient (K) developed into a complex matrix due to the addition of
measurement layers of attributes and performance scales. For this reason, a complex excel worksheet was
designed to calculate the final coefficient taking into consideration the multiple attributes and several
measurement layers and also solve the kappa algorithm per skill. To further expand the dimension of
interpretation and reveal more informative results, a specific software application was designed for two
objectives: firstly, counter-calculate results as means of cross validation and secondly, explore further
relations between attributes. The application aimed at: first, the creation of a database for the evaluation
results of n raters to a number of students based on: a. students’ data (could be grouped into various
classes/categories)., b. raters/judges data, c. Skills classified into two levels (Main skill and subset of skills
related to each main skill)An evaluation scheme to represent the grading marked by the rater. Grading
entailed two levels: (Main level, A,B,C…) and Detailed level (A1, A2, A3…). Secondly the aim was to
analyse the database content by student and per skill for all students, calculating Kappa coefficients and
correlations between compliance and motivation during computer sessions.
The application was designed to retain specific features to ensure replication of evaluation at different
periods of time for any number of students and raters to monitor pedagogy outcomes and inform practice.
3.5 Semi Structured Interviews

Semi-structured interviews were held and taped for re-examination of data and reduction of internal validity
threats (Robson, 2002). Thirteen teachers, one coordinator, and four parents were interviewed where
teachers’ interviews consumed approximately forty minutes each and the parents an average of 25 minutes.
All interviews were held in Arabic since it is the native language for all respondents. Thematic analysis was
employed as a flexible analytic tool to decipher the flow of data in the semi-structured interviews Braun &
Clarke (2006). Themes were recurrently reviewed and categorization was refined as deemed crucial Two
independent readers were called for examining researcher’s classification of two interviews and classification
of themes upon which there was 80% agreement on the suggested main themes and 100% agreement on the
assignment of utterances to the different themes as reported in the results and discussion.
52
3.6 Kappa Coefficient Values

The four raters rated independently the performance of these students during 1:1 computer sessions held in
the Center. A total of 101 computer skills were observed among the 25 students, each according to his/her
IEP. Selection of skills was initially grouped by program type: mouse control, keyboard manipulation, using
CD drive, Paint Program, Word program, PowerPoint program, Internet/email and academic objectives.
However, the type of data found during the rating phase and the outcome of results signified the need for a
different categorization since not all students displayed achievement through common routes. Therefore, an
alternate categorization of three major skill divisions was derived as follows:
1. Basic computer skills: incorporating mouse control and keyboard manipulation, and using CD drives.
All 25 students were observed using basic skills since they are the fundamental keys for operating
computers.
2. Intermediate computer skills: incorporating Paint, Word, PowerPoint Programs, and Internet/email
which are considered interactive computer tools and require good basic skills.
3. Digital Literacy skills: where learning and/or consolidation of classroom academic objectives that
take place via computer programs (reading/writing/math).
In the raters’ sheet, five main variables comprising ‘Independent without support’, ‘Independent with
physical support’, ‘Independent with verbal support’, ‘motivation’, ‘compliance’, employed the same rating
scheme of 1/2/3 where 1= 0-30%, 2=30%-70%, and 3=70%-100% which constituted sub-variables for each
of the main variables and a sixth variable of ‘not achieved’ attribute which required a binary rating of ‘x’ or
‘no mark’. Raters used the same rating scheme for consistency. Tables 2 through 9 demonstrate the
calculated K coefficient for all observed skills grouped according to the three new categories.
4. RESULTS
According to the senior computer teacher, all skills included for rating and observation were entirely new to
these students before joining the program. In other words, the rated computer skills value before the
intervention could be set up to nil at the beginning of training. Achieving a K value for a skill represented the
students’ ability to perform the task based upon raters’ scores during observation, which in other words,
signified progress in learning that skill. However, the variant K values represented the degree of agreement
among raters regarding the level of mastery rather than the skill acquisition. For example, 1.00 indicated
‘almost perfect agreement’ by raters on one specific level of mastery versus 0.39 that indicated ‘fair
agreement’ by raters on one specific level of mastery and accordingly -0.50 indicated ‘poor agreement’ by
raters on one specific level of mastery rather than unsatisfactory knowledge.
4.1 Basic Computer Skills

Since basic skills represented a main threshold for operating a computer, all 25 students were observed and
raters agreed that these students acquired these skills and could master it at different levels. None of the
raters marked ‘not achieved’ for any of the basic skills but rather showed variable dependency rates where
some students required either verbal or physical support to accomplish the task. Moreover, independent
observation did not register a negative impact on the students’ general performance as the students seemed to
understand the task and act instantaneously upon instruction. However, some students displayed slower
reaction due to behavioural or disability issues which was compensated by the instructor’s verbal or physical
support. The K values for both mouse and keyboard skills conveyed 90% and 63% mastery of skills
respectively. Comparably, as stated in Olson et al.’s (1997) study, the researchers called attention to the
importance of mouse control training as a basic skill for using the software program prior to starting a
computer-based intervention which set the participants up for improved performance as computer users.
Additionally, a high correlation was revealed between keyboard skills and higher skills learning as will be
shown later in this section, that further corroborates the basic skills value suggested by Olson et al. (1997).
Interestingly, a teacher during her interview supported this notion mentioning: ‘...especially that their mouse
skills is always better than their hand skills in moving flash cards in a task while working on a table …’
(interview no. 04, lines 12 & 13).
53
ISBN: 978-989-8533-12-8 © 2012 IADIS
4.2 Intermediate Computer Skills

K values for Paint, CD drive, Word, and PowerPoint programs were calculated. The number of observed
students for each skill varied according to each student’s IEP.
Paint Program: There was perfect agreement for skills: Reduce and enlarge size of geometric figures,
Create a group of fruits' images and Save working file but poor for Choosing a colour from tray.
Using CD Drive: There was perfect agreement for skills: Open CD ROM and Insert disc into drive but not
for Close CD ROM.
Word Program: Almost Perfect to moderate agreement (1.0-0.49) for most skills (9) in this category and
fair agreement for 3 skills (.3)
PowerPoint Program: K was from 1 to 0.32 for 12 out of 17 skills in this category showing high overall
agreement.
The higher K agreement values on both Paint program and CD drive resulted from competent students’
performance and small number of students observed (n=4). Students’ performance on Word gained high
agreement among raters with poor agreement on mastery level of only 3 skills out of 12.The PowerPoint K
values showed total rater agreement on achievement of learning program functions with variant mastery
levels. Of those, 12 out of 17 skills gained high K values which suggested high mastery of skills. However,
it should be noted that the remaining 7 skills also obtained rater agreement for achievement but with
dependant performance on verbal support. None were marked ‘Not achieved’ by raters. Internet and email K
values also followed the same pattern of achieved performance as the preceding programs with variable
mastery levels and no ‘Not achieved’ markings. Here, 5 out of 8 internet/email skills gained raters’
agreement for independent performance yielding high K values while the rest of skills were acquired but
required verbal support.
Abbot (2007) stressed the value of graphic symbols in software programs and its impact on facilitating
the use of the various software functions. In this framework, the software programs employed in this
research especially Word and PowerPoint highly utilized graphic symbols. It was observed that students
trained on these programs mostly relied on these symbols, particularly non-readers, for choice/selection of
tasks that markedly supported them in using the programs to a far extent. Considerably, interviews with all
class teachers escalated the value of students’ learning to use basic software functions to its possible impact
on their future prospect. For example, teachers expressed their contentment of the students’ progress in
keyboard skills in conjunction with using the Word program for the objective of typing text. One computer
teacher described the level of two students (Child07 and Child14) as ‘…faster than me in some
tasks…’(interview no.01, line 90) and another class teachers upgraded this opinion to the level of actual
employment opportunity in a clerical job mainly responsible for typing text in a factory for Child11 ‘…and I
employed these computer skills in the factory…’(interview no.05, line 72). It is noteworthy that all class
teachers concurred with this opinion and perceived this stage as a possibility given the right conditions to
develop it which may be further researched in another context.
4.3 Literacy Skills

The observed students utilized their acquired computer skills as a tool for learning and drilling on their class
academic objectives. The raters’ agreements indirectly involved an evaluation of the students’ academic
performance regarding these specific tasks, nevertheless, none was marked ‘Not achieved’ which signified
total agreement on the acquisition of skills but with variant mastery levels as per raters’ agreement from
‘almost perfect’ to ‘poor’. There was almost perfect agreement for the following skills: Re-arrange words to
form correct sentence, Read passage and choose word opposites Size concept (small/big) choose correct
image upon verbal prompt, Same/Different concept choose correct images upon a verbal prompt ,Select
correct color from an array of 3 colors upon a verbal prompt and Categorize objects.; Moderate agreement
(.49) for skill: Choose accurately currency upon verbal prompt and poor agreement for all other literacy skills
observed. This category may be perceived as a further utilization of the previous acquired skills in the
preceding two categories of basic and intermediate skills. It entailed the drill and practice aspect of
technology discussed by Abbot (2007) as using technology to train or rehearse.
54
4.3.1 Correlations across Skills

At this point, the results pointed confidently at all students’ successful acquisition of basic computer skills
and the continued progress of some students throughout the following two phases of intermediate skills and
using computer skills as a learning tool. Further investigation of these results yielded interesting correlation
results. Various cross-relations among the above data components were examined and thus revealed two
major correlations between the K values of academic objectives and other programs and another correlation
between keyboard skills and PowerPoint skills.
Table 1. Correlation between agreement values of academic skills and other programs.
K Values Correlation Coefficient Correlation Classification
Academic Skills Dependent variable (reference skill for correlations)
PowerPoint 0.62 High
Keyboard 0.51 Average
Paint 0.32 Slight
Word 0.17 Low
Mouse 0.17 Low
As shown in table 1, correlations between each program skills and academic skills provided interesting
results. The two highest correlations with acquiring academic skills were PowerPoint and Keyboard skills
respectively. To further establish the relationship among the three highest correlation values presented in
table 1, another investigation was calculated to determine the magnitude of basic skills category training,
mouse control and keyboard manipulation, in relation to students’ performance in other programs. Table 2
presented the results of a correlation matrix between keyboard skills and other programs as shown below:
Table 2. Correlation between keyboard skills and other programs
K Values Correlation Correlation Classification
Coefficient
Keyboard Skills Dependent variable (reference skill for correlations)
PowerPoint 0.75 Very high
Academic skills 0.51 High
Paint 1 Almost perfect
Mouse 0.23 Slight
Word 0.15 Low
Table 2 shows the highest correlation relationship between a trio of keyboard skills, PowerPoint and
Academic skills suggesting a strong relationship in the acquisition of skills among these programs. The
preceding quantitative analysis and discussion of the students’ performance corroborated by the teachers’
judgments reasonably illustrated a positive impact on their learning as a result of integrating computer skills
training into their IEPs, which provided an answer to the main research question. As demonstrated in the
above tables, each of the 25 students was observed to master a set of skills according to his/her ability and
training duration. The demonstrated progress was measured on a base of zero computer knowledge of the
rated skills as formerly mentioned. In fact, it is important to note that other interesting correlations between
motivation and compliance of students during computer sessions emerged in the course of data analysis
which may be further researched and reported in another context.
5. CONCLUSIONS
The acquisition of computer skills was evidenced for the 25 participant students by the quantitative measure
of Kappa in a mixed design methodology. The results further unfolded a range of acquired skills at mastery
levels shown in the range of K values. Furthermore, data analysis revealed a possible categorization of skills
where the population of students were distributed according to their level of achievement in learning
computer skills. These categories began with a basic skills level, intermediate, and concluded with literacy
55
ISBN: 978-989-8533-12-8 © 2012 IADIS
skills learning category. Subsequently, correlations across learned skills were computed and revealed a high
correlation relationship between Keyboard, PowerPoint, and academic skills, suggesting the significance and
impact of keyboard training as an essential component for further learning of academic objectives. Results
further demonstrated the impact of computer skills training on learning literacy skills. Learning and/or
drilling of academic objectives such as basic reading, writing, and math was evidenced for students who were
able to advance through the basic and intermediate skills categories. This finding further confirmed the
significant impact of ICT as part of the students with SEN learning experience and a possible additional
route for overcoming fundamental learning barriers. Additional correlations between other variables such as
motivation and compliance were also captured during the observation and rating phases, which will be
reported in another study.
To corroborate the above findings, teachers and parents opinions were obtained to capture their insiders’
perspectives incorporating the researchers’ observations. They further confirmed the positive impact of
computer skills training on the students’ learning experience. Findings highlighted the need for more
research on the value of current and potential applications of ICT for students with SEN to position them
among the major beneficiaries of the technology revolution. Within this framework, it may be suggested to
replicate this study in the same context for comparative value of ICT impact issues and also in different
contexts using the same as well as other types of ICT. Moreover, it may be beneficial to suggest the need for
incorporating this report’s findings with pedagogy aspects and parental perspectives in future research.
REFERENCES
Abbot, C. (2007). Learning difficulties and digital Technologies. Futurelab Series (15).
Braun, V., Clarke, V., (2006). ‘Using thematic Analysis in Psychology’. Qualitative Research in Psychology, 3:77-101
Bryman, A. (2001). Social Research Methods. Oxford University Press
Fleiss, J. (1981). Statistical Methods for Rates and Proportions second edition. Wiley Series in Probability and
Mathematical Statistics. p. 212-236
Hanson, W.E., Creswell, J. W, Clark, V.L.P, Petska, K.S., Creswell, J. D., (2005). ‘Mixed Research Designs in
Counseling Psychology’. Journal of Counseling Psychology 52:2 pp.224-235
Johnson, R.B, Onwuvegbuzie, A.J., (2004). ‘Mixed Methods Research: A Research Paradigm Whose Time Has Come’.
Educational Researcher 33;14
Olson, R. K., Wise, B., Ring, J., and Johnson, M., (1997). ‘Computer-Based Remedial Training in Phoneme Awareness
and Phonological Decoding: Effects on the Posttraining Development of Word Recognition’. Scientific Studies of
Reading, 1:3,235-253
Robson, C. (2002). Real World Research. 2nd Ed. Blackwell Publishers
56
THEORIZING WHY IN E-LEARNING – A FRONTIER FOR

COGNITIVE ENGAGEMENT
Jon Mason
Queensland University of Technology
Faculty of Education, Kelvin Grove QLD, Australia
ABSTRACT
Asking why is an important foundation of inquiry and fundamental to the development of reasoning skills and learning.
Despite this, and despite the relentless and often disruptive nature of innovations in information and communications
technology (ICT), sophisticated tools that directly support this basic act of learning appear to be undeveloped, not yet
recognized, or in the very early stages of development. Why is this so? To this question, there is no single satisfactory
answer; instead, numerous plausible explanations and related questions arise. After learning something, however,
explaining why can be revealing of a person’s understanding (or lack of it). What then differentiates explanation from
information; and, explanatory from descriptive content? What ICT scaffolding might support inquiry instigated by why-
questioning? What is the role of reflective practice in inquiry-based learning? These and other questions have emerged
from this investigation and underscore that why-questions often propagate further questions and are a catalyst for
cognitive engagement and dialogue. This paper reports on a multi-disciplinary, theoretical investigation with a view to
informing the broad discourse on e-learning by identifying a specific frontier for design and development of e-learning
tools. Probing why reveals that versatile and ambiguous semantics present the core challenge – asking, learning, knowing,
understanding, and explaining why.
KEYWORDS
Scaffolding, inquiry, why-questioning, reflection, understanding, cognitive engagement, cognition in education
1. INTRODUCTION
In introducing any topic of investigation it is usually helpful to understand why it is presented and what its
key drivers are; a listener or reader often finds it helpful to understand the context of an investigation in order
to make some initial sense prior to embarking on giving it further attention. Such context can also be
described in terms of motivation, purpose, rationale, and/or justification for the work – or as “advance
organizers” (Ausubel, 1960). Perspectives that emerge from responding to questions help to explicate some
context – e.g., (i) why is this paper submitted to CELDA 2012? (ii) what is the central argument of this
paper? Providing perspective of this kind can serve as a trigger for cognitive engagement and doing so – in
the form of a well-constructed abstract – is an established academic convention. Roots of influence for this
practice stretch back to the time of Aristotle, when logos was elaborated as well-formed argument within
reason and, as such, one of three modes of persuasion – the others being ethos and pathos.
Thus, motivation for this paper emerges from consideration of future prospects for e-learning activities
that probe the why dimension, or inquiry that involves why – asking, learning, understanding, knowing,
reflecting upon, and explaining why. In order to develop an overarching narrative a number of interrelated
topics are discussed: the evolution of e-learning; the role of questioning while learning; descriptive versus
explanatory content; inquiry-based learning; scaffolding using information and communications technology
(ICT); and, future prospects for ICT tools that support and promote why-questioning.
As a consequence of extensive academic literature on the subject it is assumed that asking why is an
important foundation of inquiry and fundamental to the development of reasoning skills and learning. But, in
direct contrast, sophisticated ICT tools that directly support this basic act of learning appear to be either
undeveloped or, at best, in the very early stages of development. Why is this so? On the one hand, this paper
suggests there are a number of very good reasons; on the other, it is focused on the implications of
57
ISBN: 978-989-8533-12-8 © 2012 IADIS
developing better tools that support asking why and understanding why in the specific context of e-learning.
It summarises relevant research with a view to informing the design of ICT tools that might stimulate deep
learning and cognitive engagement. In supporting a clear rationale for the investigation the following real-
world scenarios describe common, contemporary situations in e-learning involving concepts of why:
1.1 Scenario – University Student

Sarah is a university student majoring in international relations and history. She has opted to do much of her
studies online because it provides her with the flexibility to take on some part-time work. The university has
invested considerable funds into preparing appropriate content and assessment tasks for subjects offered in
online mode; it has also implemented a standard single-platform policy and installed BlackBoard, a Learning
Management System that helps structure learning content and contain interactions between staff and students.
Sarah uses Google to search for additional resources for an essay on the conflict in the Middle East. While
she finds numerous resources it is challenging for her to understand the causes of this conflict or what the
appropriate actions might be for it to be resolved. The course resources seem well-structured but she is
required to investigate sources beyond the prescribed texts. If she searches Google with ‘why’ questions she
feels very dissatisfied with the quality of the results. Likewise, when searching the library catalogue she is
overwhelmed by the volume of resources and is not confident in making a judgement about why this conflict
seems so deeply problematic because she finds so many plausible, yet contradictory, and politicized
explanations. Even though she has access to a number of ‘social software’ applications that enable her to
interact and share resources easily with others who might be investigating the same topic she feels like there
is something lacking in the online tools available. She feels that she needs assistance in discerning fact from
political rhetoric and some other way of navigating and evaluating the large amount of content on this topic.
She wants to understand the key issues at the heart of the conflict.
1.2 Scenario – High School Teacher

Dave is an art teacher at a high school with two decades of experience. The school has a reputation for
adopting ICT into the curriculum wherever possible; however, art has been the last subject to embrace ICT.
This is partly due to the fact that Dave feels more comfortable using traditional media. The school is now
urging him to make the shift. In moving his content into an electronic mode Dave discovers that he has to
anticipate many of the questions that students typically ask when in the classroom (‘but why do we have to
study Matisse, sir?’; ‘why is some abstract art seen as having great merit while some doesn’t?’; ‘why are
there different versions of what constitute primary colours?’). Because of his experience he knows that the
students need good answers to such questions so that they can be motivated to learn. He thinks that he may
need to create a bank of such questions together with suitable answers but hesitates because he knows that
when students ask questions a longer conversation often proceeds. He is unsure of the best way to make such
information available so chooses just to make it explicit in the introductory text to each task described in the
online version of his course. But he remains sceptical that anticipating such questions in a ‘canned’ way will
be as motivating for the students as being able to respond in real-time. He would prefer to foreground student
questioning and make it stimulating and interactive, rather than content that students read.
1.3 Scenario – Instructional Designer

Thor is an instructional designer for a publishing company that specialises in de-mystifying science. The
publisher has already had commercial success in preparing online materials that mimic the successful
television series in Australia during the 1960s, ‘Why is it so?’ Thor has been asked to assist in developing
innovative pathways to scientific content that will stimulate students to think and ask ‘why’, to motivate their
curiosity and that leads them into understanding scientific inquiry. He is not quite sure how to proceed and is
suspicious of Q&A approaches because providing answers can often close down inquisitiveness; he knows
that powerful search engines like Google can deliver responses to search queries but will also limit the
student to searching, not questioning; he also aware that none of the natural language search engines he
knows of seem to do a very good job with responding to why questions. How is he to proceed?
58
1.4 Scenario – Teacher Librarian

Lisa is a teacher-librarian. She has access to a range of repositories of high quality, digital learning content.
Most of this content is described using Dublin Core metadata (i.e., information such as the author, title,
keywords, and abstract) and some of it is described by IEEE Learning Object Metadata (i.e., information
similar to Dublin Core metadata but also includes information about the educational level associated with the
content and duration of the resource). Some resources also have metadata that describes associated learning
objectives and competence level required in order to interact effectively with the resource. Lisa has found
that many of the teachers she supports also want to know why a particular resource might be more suitable
than another for a particular learning activity or goal. Lisa has found that rating systems and user-generated
tags and ‘folksonomies’ are sometimes helpful in this regard but is frustrated that not all the repositories
support such services. She wonders whether there might be a better approach.
2. E-LEARNING EVOLVES
2.1 Terminology and Scope

Theory and practice of any domain of human activity are constantly evolving and mutually informing. But
while both philosophers and practitioners have discussed matters associated with learning for thousands of
years, it is not yet two decades since the term ‘e-learning’ entered mainstream discourse. It is therefore
important to make explicit what is meant by this term as it has been appropriated by diverse communities of
practice since it first appeared around 1998-99 (Cross, 2004; CIPD, 2008; Garrison and Anderson, 2003).
e-Learning can signify both a theoretical discourse and a range of activities that take place in many
contexts – formal and informal, within educational institutions and workplace settings, or elsewhere ‘any
time any place’ as the saying goes. Adopters of the term include corporate training associations, professional
associations, academic Web enthusiasts, government policy makers, software vendors, standards
development organizations, and military organizations, just to name a few (Mason, 2005:320). There are
distinctions according to context. For example, Bates identifies key differences between post-secondary
education and corporate settings – the latter being more concerned with the broader context of knowledge
management, the former focused on learning and research (Bates, 2004: 275). In an attempt to broaden
philosophical perspective, Friesen puts the case for “re-thinking e-learning research” and argues for a
“reconceptualization of e-learning as an inter- and cross-disciplinary endeavor” (Friesen, 2009:20).
Conceptualizing in even broader terms, Cooper argues that its scope of activity is best understood as
‘emergent’ and therefore subject to analyses that highlight perspectives on “complexity” (Cooper, 2010).
Others prefer to use the related terminology ‘online learning’ to frame the challenges of “integrating
technology into classroom instruction” (Tomei and Morris, 2011). For the purposes of this paper, e-learning
is defined as: learning that is facilitated by engagement with ICT.
2.2 Innovation and Practice

With the above definition in mind, a diversity of ICT development and innovation over the last few decades
can meaningfully be described as e-learning systems, environments, or platforms. Examples of structured,
contained, or purpose-built platforms include computer-based training systems, learning management
systems, intelligent tutoring systems, e-portfolio systems, performance support systems, virtual worlds,
gaming environments, e-books, and other related applications and services. Anyone with a young child who
has access to an iPad will also know how engaging and educational a single app can be – whether it is
explicitly educational or not. Examples of unstructured and open environments that can function as e-
learning environments include use of mainstream search engines and social media. Benefits and deficiencies
can be identified with all of these developments, as is documented in the extensive and growing discourse on
e-learning – for example, the number of peer-reviewed journals worldwide dedicated to the subject is now in
excess of 50 titles and the majority of these titles have emerged in the last five years. If related topics such as
59
ISBN: 978-989-8533-12-8 © 2012 IADIS
Distance Education, e-Research, Technology in Society, Knowledge Creation, and Performance Support are
included then there are hundreds of relevant journals.
As e-learning develops into an established academic field it brings with it a discourse that refines its core
concepts and terms while ICT innovations and trends evolve. It is also likely that certain trends and biases
will be revealed along the way. For instance, evidence suggests that much of the first generation of practice
associated with e-learning has been very focused on the delivery and access to purpose-built learning
content, not so much with learning activities or the cognitive processes associated with learning (Dalziel,
2003; Alonso, et al., 2005; LETSI, 2008; ADL, 2009). This first generation of learning content has also been
constrained by metadata that is descriptive in function – in other words, metadata that describes the content in
terms of semantics that have roots in who, what, when, and where.
It is also the case that the educational potential of existing, emerging and future developments in ICT is
now commonly discussed in many diverse settings (daily newspapers, school curriculum support materials,
political party policy documents, workplace human resource departments, standards-setting bodies, academic
literature, and in higher education strategic planning). The ‘Digital Education Revolution’ policy of the
Australian Governments during 2007-2012 is a prominent example of a public policy response. Such public
policy has been commonplace since the invention of the World Wide Web, although prior to this, the
transformative potential of educational technology was recognised at various other historical moments (such
as with the inventions of radio, television, personal computers, interactive and game-based digital media).
There are therefore multiple perspectives that help explain the history and viable developmental paths of e-
learning into the future. The Australian School of the Air, which began in 1951 and continues today in
servicing the needs of remote communities in Australia, represents an example of an older communications
technology that is still used effectively for educational purposes. This is significant because it suggests that
the viability of a technology is not necessarily made redundant by new technologies.
2.3 Historical and Social Narratives

Broader historical perspective provides further context. Not only has evolution of the World Wide Web taken
place within a short period of time accompanied by rapid innovation, it has been transformative, representing
a global revolution in the production, distribution, and access to information and communications (Castells,
1996; Benkler, 2006; Gleick, 2011) and can be seen as having enormous impact upon teaching and learning.
A number of commentators have consequently introduced narratives on the evolution of the Web in terms
of its impact upon learning. Taylor (2001), for instance, began visioning “fifth generation distance education”
around 2001-2002 as an “intelligent flexible learning model” – it was student-centric in conception but
impacted significantly the organizational structures and readiness for institutions concerned. In 2005,
Siemens proposed a new learning theory called “connectivism”, motivated principally by the impact of the
proliferation of networked ICT applications and the limitations of dominant learning theories (behaviourism,
cognitivism, and constructivism) to explain and support the scope of interactions a learner. The
distinguishing characteristic of Siemens’ theory is the prominent role of networks in creating connections
between disparate learning sources and events (Siemens, 2005). Siemens’ work resonates with the extensive
sociological work of Castells (1996, 2001) in outlining the “rise of the network society” and in the work of
Benkler on the social production of intellectual capital (Benkler, 2006).
More recently, there has been popular usage of the terminology ‘Web 2.0’ typically to describe
networking capabilities that leverage social media providing individuals with enormous scope for publishing
content and social interaction. Adoption of such terminology has also led to characterisations of “Learning
2.0” being learning that is facilitated by Web 2.0 social media applications (Brown and Adler, 2008) and
related commentary about the “post-LMS era” (Mott, 2010). The utility of such characterisations is yet to be
determined; however, in terms of the evolution of e-learning, they can be somewhat misleading because they
mask, or do not always explicitly acknowledge, the capabilities that already existed in early phases of
development – such as in Computer Based Training (CBT), Computer Assisted Learning (CAL), Computer
Managed Learning (CML), Computer Mediated Communication (CMC), and Computer Supported
Collaborative Learning (CSCL). The important observation here is that there are numerous technologies that
have shaped what e-learning is today. Secondly, and most importantly for this investigation, none of the
innovations mentioned hitherto have explicitly explored how why-questioning during learning might be
explored, supported or scaffolded.
60
2.4 Into the Future

With the foundations of e-learning now well-established there is enormous scope for new developments that
may enrich learning experiences through supporting deeper inquiry and cognitive engagement via
environments that stimulate reflective practice and the development of understanding while learning online.
A number of likely future trajectories can be discerned from the current context – for example, the broad
uptake of social media provides stimulus for the use of diverse collaborative environments at scales
unprecedented. Other developments will emerge as a consequence of ubiquitous broadband connectivity,
innovations in natural language search technologies, access to open content, the proliferation of mobile
technologies, work integrated learning programs, and intelligent tutoring systems. Will IT develop further as
an “intelligent technology” or an “interruption technology” (Carr, 2010)? No doubt, unexpected innovations
will also impact the evolutionary story.
This paper, however, is concerned with one of the frontiers that beckon further development – ICT that
supports deep learning instigated by questioning, reflective practice, and promotes cognitive engagement.
3. COGNITIVE ENGAGEMENT
3.1 Ubiquitous Distraction?

There can be little doubt that the Internet has spawned a proliferation of ICT tools useful for learning. But the
story of the impact of such relentless innovation is not an intrinsically positive one. It is also accompanied by
a growing discourse arguing that extended use of the Internet can also have detrimental effects on cognition
and behaviour (Clark, 2002; Bannister and Remenyi, 2009; Carr, 2010; Aguirre, 2011; Chalupa, 2011).
Evidence shows there is definitely an impact upon cognitive load (Verhoeven, 2009; Kleinberg, 2011), a
topic that instructional designers have been concerned with for decades (Sweller, 1994). For example, for
reasons that being online can be very distracting with the effect of weakening cognitive focus, the term
“interruption technology” has been a catch-phrase in contemporary popular commentary on the Internet:
the single most mind-altering technology that has ever come into general use … when we go online,
we enter an environment that promotes cursory reading, hurried and distracted thinking, and
superficial learning … The Net’s cacophony of stimuli short-circuits both conscious and
unconscious thought, preventing our minds from thinking either deeply or creatively. (Carr, 2010)
Of course, similar commentary and research has existed for decades about extended exposure to
television and virtual gaming environments. Thus, the discourse is not all negative – for example, research
shows that while extended Internet use can cause some loss of short-term memory there is also a gain in that
“The Internet has become a primary form of external or transactive memory, where information is stored
collectively outside ourselves” (Sparrow, et al., 2011).
There is truth in both arguments – so in terms of the nature of cognitive engagement while learning
online, evidence that drives this debate will be important for researchers to track.
3.2 The Search Paradigm

The enormous market success of the Google search engine can be seen as paradigm-shaping in the way that
much learning online and scholarship is now initiated – via search. Its functionality has also delivered routine
information retrieval and discovery into the mainstream. Of course, not all searches using Google are
concerned with learning and most are better classified as information-seeking and Google’s effectiveness has
also impacted corporate workflows, the socialization of information (Brown and Duguid, 2000),
Government-based services, and the expectations of citizens of the developed world. As Google (the
company) has developed its own services, such as Gmail and Google Docs and Drive, the flagship search
engine can be seen as the core piece of technical architecture – search being the key operator on, and
organizing technology for, content. Again, however, Carr notes a downside:
Google … shapes our relationship with the content that it serves up so efficiently and in such profusion.
The intellectual technologies it has pioneered promote the speedy, superficial skimming of information
61
ISBN: 978-989-8533-12-8 © 2012 IADIS
and discourage any deep, prolonged engagement with a single argument, idea, or narrative. “Our goal,”
says Irene Au, “is to get users in and out really quickly. All our design decisions are based on that
strategy.” (Carr, 2010:156)
The immediate counterpoint to this argument is that innovations in ICT are far richer than the Google
suite of services. But, there is a further issue with the ‘Google paradigm’ relevant here: its search engine is
calibrated with a design bias that privileges the aboutness of content – in other words, it is focused on parsing
information as data. Its internal indexes are all built on data that is factual and measurable; and searches are
typically instigated by keywords and phrases, not questions constructed in natural language. Thus,
interactions with Google can be seen as being constrained by “factoid” information (Verberne, 2010), or
what Mason describes as the “primitives of information-retrieval” – facets of information that are readily
associated with questions of who, what, when, and where (Mason, 2008). While Google uses sophisticated
algorithms involving various weightings associated with “backlinks” this still functions as factoid
information. Even with value-added services to Google search, such as ManagedQ, results to queries are
organized into sets associated with people (who), things (what), and places (where). This underlying
constraint has the effect of ‘information begetting information’ and interrupts prolonged inquiry or direct
pathways into the discovery of content that is explanatory in nature (Mason, 2008; 2011a). This does not
mean that explanatory content is not retrieved, just that it is not easily or directly discovered. In particular,
queries that are conceived with ‘why’ in mind are not parsed well by Google because of the semantic
ambiguity and linguistic versatility of the term why (Evered, 2005; Verberne, 2010; Mason, 2008). This has
significant repercussions for the design of ICT systems aimed at supporting learning.
3.3 Dimensions of Why –Related Research

Why distinguishes itself from other ‘primitive’ questions (who, what, when, where, and how), in that it often
requires a plausible explanation or rationale as an adequate response – in other words, reasoning as well as
information (Verberne, 2010:10). Thus, why-questioning has the potential to initiate a shift from information
processing to engagement of other cognitive functions, such as inquiry, analysis, problem-solving, and
reflection. As Walton has noted, why is a key initiator of dialogue (Walton, 2004).
For researchers pursuing question-generation techniques in intelligent tutoring, why questions are seen to
belong to a “deep/complex” category of all possible question types (Graesser, et al., 2007). Evered (2005)
provides an analysis in which the function of responses to why-questioning is categorized according to three
classes of explanation: Causal (Why E? Because C (C = cause)); Teleological (Why E? In order to P (P =
Purpose)); and Gestaltic (Why E? For these reasons, R (R = Reasons)) (Evered, 2005:201). Thus, in
identifying opportunities for ICT-enabled scaffolding that might support inquiry and reflection, access to and
production of explanatory content, as distinct from descriptive content, is of prime concern here.
It is also interesting, however, that while why can be shown to have wide linguistic versatility (Mason,
2011a:93) it is not regarded as a “semantic prime” by linguists developing Natural Semantic Metalanguage
(research that is focused on identifying concepts with irreducible semantics), primarily because this
versatility is not free from ambiguity (Goddard and Wierzbicka, 2007).
Thus, in probing the linguistic dimensions of why, at least five key activities relevant to e-learning can be
identified – asking, learning, knowing, understanding, and explaining why. The literature on educational
psychology tells us that asking why is an important foundation of inquiry and fundamental to the
development of reasoning skills and learning (Dewey, 1966; Piaget, 1966; Schank and Cleary, 1995; Bruce
and Casey, 2012). Processes of learning, knowing, and understanding why build upon inquiry and all involve
reflective practice (Schön, 1987:72). After learning something, explaining why can reveal a person’s
understanding (or lack of it). Thus the motivating question for this investigation: what ICT scaffolding – as
application, services, or interventions – might support inquiry instigated by why-questioning?
3.4 Tools for Scaffolding and Reflective Practice

Investigations into ICT tools that explicitly aim to support why-questioning reveals a number of search
technologies based upon natural language processing and computational linguistics, although findings to date
demonstrate that much research is yet to be done (Ferrucci, et al., 2010; Verberne, 2010). Research is also
proceeding in the fields of information science (metadata schemas and question-answer techniques) and
62
question-generation for intelligent tutoring (Kunze, 2001; Mason, 2008; Rus and Graesser, 2008). Of
immediate relevance, however, is the application of wikis and e-portfolio systems to support reflective
practice that is consistent with the goals of inquiry-based learning. Evidence is mounting that both
approaches – one via the route of enlisting open, social engagement in content production (wikis); the other,
individually-controlled reflective journalism that is discretionally shared – develop reflective practice and
therefore prolonged cognitive engagement (Loo, 2012; Mason 2011b). A challenge, then, that is specific to
the focus of this investigation is how scaffolding interventions might leverage these platforms.
4. CONCLUSION
Investigations into why-questioning reveal there are significant repercussions for the design, development,
and utilization of ICT systems aimed at supporting learning. In particular, accommodating multiple
dimensions of why – asking, learning, knowing, understanding, and explaining – point to a frontier that will
focus on the pivotal role of explanatory content and prolonged cognitive engagement through reflective
practice.
REFERENCES
ADL, 2004. Sharable Content Object Reference Model (SCORM). Advanced Distributed Learning, US Department of
Defense. http://www.adlnet.gov/Technologies/scorm/default.aspx
Aguirre, A., 2011. The Enemy of Insight? In Brockman J, ed. Is the Internet Changing the Way You Think? The Net’s
Impact on Our Minds and Future. Harper Perennial, New York, pp.304-306.
Alonso, F., et al., 2005. An instructional model for web-based e-learning education with a blended learning process
approach, British Journal of Educational Technology, Vol 36 No. 2, pp. 217–235.
Ausubel, D. P. 1960. The use of advance organizers in the learning and retention of meaningful verbal material. Journal
of Educational Psychology, Vol 51 No. 5, pp. 267-272
Bannister, F. and Remenyi, D., 2009. Multitasking: the Uncertain Impact of Technology on Knowledge Workers and
Managers. The Electronic Journal Information Systems Evaluation. Vol 12 No. 1, pp. 1 – 12
Benkler, Y., 2006. The Wealth of Networks: How Social Production Transforms Markets. Yale University Press, USA.
Brown, J.S., and Duguid, P., 2000. The Social Life of Information. Harvard Business School Press, Boston, MA.
Brown, J.S., and Adler, R.P., 2008. Minds on Fire – Open Education, The Long Tail, and Learning 2.0. Educause
Review, Vol 43 No. 1, pp. 17-32.
Bruce, B.C., and Casey, L., 2012. The practice of inquiry: A pedagogical ‘sweet spot’ for digital literacy? Computers in
the Schools, Special Issue on Signature Pedagogies. Vol 29 No.1-2, pp. 191-206.
Carr, N. 2010. The Shallows – What the Internet is Doing to Our Brains. W W Norton & Company, New York.
Castells, M., 2001. The Internet Galaxy – Reflections on the Internet, Business, and Society. Oxford University Press,
Devon, UK.
Castells, M., 1996. The Rise of the Network Society – The Information Age: Economy, Society and Culture Vol. I.
Blackwell: Oxford UK
Chalupa, L. 2011. The Greatest Detractor to Serious Thinking Since Television. In Brockman J, ed. Is the Internet
Changing the Way You Think? The Net’s Impact on Our Minds and Future. Harper Perennial, New York, p.44.
CIPD, 2008. E-learning: progress and prospects. Chartered Institute of Personnel and Development, UK.
http://www.cipd.co.uk/subjects/lrnanddev/elearning/elearnprog.htm
Clark, D., 2002. Psychological myths in e-learning. Medical Teacher. Vol 24, No. 6, pp. 598-604.
Cooper, A.R., 2010. Key Challenges in the Design of Learning Technology Standards: Observations and Proposals.
International Journal of IT Standards and Standardization Research, Vol. 8, No. 2, pp. 20-29.
Cross, J., 2004. A History of eLearning, The Future of eLearning, Internet Time Group
http://www.scribd.com/doc/12913198/History-Future-of-eLearning-On-the-Horizon
Dalziel, J. (2003). From re-usable e-learning content to re-usable learning designs: Lessons from LAMS.
http://www.lamsinternational.com/CD/html/resources/whitepapers/Dalziel.LAMS.doc
Dewey, J., 1966. Democracy and education: an introduction to the philosophy of education. The Free Press, New York.
63
ISBN: 978-989-8533-12-8 © 2012 IADIS
Evered, R., 2005. A Typology of Explicative Models. In C. Lundberg & C. Young (Eds.) Foundations for Inquiry:
Choices and Tradeoffs in the Organizational Sciences. Stanford Business Books, Stanford, CA.
Ferrucci, D., et al., 2010. Building Watson: An overview of the DeepQA project. AI Magazine, Vol 31 No 3, pp. 59–79.
Friesen, N., 2009. Re-Thinking e-Learning Research: Foundations, Methods, and Practices. Counterpoints – Studies in
the Postmodern Theory of Education, Vol 333. Peter Lang Publishing, New York.
Garrison, D.R. and Anderson, T., 2003. E-learning in the 21st Century: A Framework for Research and Practice.
Routledge/Falmer, New York
Gleick, J., 2011. The Information – A History, A Theory, A Flood. Pantheon, New York.
Goddard, C., and Wierzbicka, A., 2007. Semantic primes and cultural scripts in language learning and intercultural
communication, in G. Palmer and F. Sharifian (Eds.), Applied Cultural Linguistics: Implications for second language
learning and intercultural communication. John Benjamins, Amsterdam, The Netherlands.
Graesser, A., Rus, V., and Cai, Z. (2007). Question Classification Schemes. University of Memphis, USA.
http://www.cs.memphis.edu/~vrus/questiongeneration/16-GraesserEtAl-QG08.pdf
Junqi Shi, et al, 2011. Internet Self-Efficacy, the Need for Cognition, and Sensation Seeking as Predictors of Problematic
Use of the Internet. Cyberpsychology, Behavior, and Social Networking. Vol 14, No 4, pp. 231-234
Kleinberg, J. 2011. The Human Texture of Information. In Brockman J, ed. Is the Internet Changing the Way You Think?
The Net’s Impact on Our Minds and Future. Harper Perennial, New York, pp. 83-85.
Kunze, J., 2001. A Metadata Kernel for Electronic Permanence. Proceedings of the International Conference on Dublin
Core and Metadata Applications DC-2001. Tokyo. http://www.nii.ac.jp/dc2001/proceedings/product/paper-27.pdf
LETSI, 2008. SCORM 2.0 White Papers. Learning Education and Training Systems Interoperability initiative.
http://wiki.letsi.org/display/nextscorm/SCORM+2.0+White+Papers
Loo, T., 2012. Wikipedia in the Classroom, http://niche-canada.org/node/10397
Mason, J., 2011a. Cognitive Engagement and Questioning Online. In A. Méndez (Ed.). Education in a technological
world: communicating current and emerging research and technological efforts. Formatex, Badajoz, Spain.
Mason, J., 2011b. Scaffolding for Integrated Reflection. Proceedings of the 19th International Conference on Computer
in Education. Asia-Pacific Society for Computers in Education. Chiang Mai, Thailand. pp. 587-594.
Mason, J., 2008. A model for sense-making: exploring why in the context of learning and knowing. Proceedings of the
16th International Conference on Computers in Education, Asia-Pacific Society for Computers in Education, Taipei,
Taiwan, pp. 545-549.
Mason, J., 2005. From e-learning to e-knowledge. In M. Rao (ed.) Knowledge Management Tools and Techniques,
Elsevier, London, pp. 320-328.
Mott, J., 2010. Envisioning the Post-LMS Era: The Open Learning Network. Educause Quarterly, Vol 22 No. 1.
Paul, J.L., and Marfo, K., 2001. Preparation of Educational Researchers in Philosophical Foundations of Inquiry. Review
of Educational Research. Vol 71, No 4, pp. 525-547.
Piaget, J., 1966. The Child’s Conception of Physical Causality. Routledge and Kegan Paul, London.
Rus, V., and Graesser, A. (Eds.), 2008. The Question Generation Shared Task & Evaluation Challenge Workshop Report.
University of Memphis. http://www.questiongeneration.org/
Schank, R. and Cleary, C., 1995. Engines for Education. Lawrence Erlbaum Associates, New Jersey, NY.
Schön, D., 1987. Educating the reflective practitioner. Jossey-Bass, San Francisco.
Siemens, G., 2005. Connectivism: A learning Theory for the Digital Age.
http://www.elearnspace.org/Articles/connectivism.htm
Sparrow, B., et al, 2011. Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips.
Science 1207745. DOI:10.1126/science.1207745
Sweller, J., 1994. Cognitive load theory, learning difficulty, and instructional design, Learning and Instruction, Vol 4,
Issue 4, pp. 295–312.
Taylor, J.C. 2001. Fifth Generation Distance Education. Australian Department of Education, Training and Youth
Affairs, Higher Education Division, Higher Education Series, Report No. 40, Canberra, Australia.
Tomei, L.A. and Morris, R., 2011. Top Technologies for Integrating Online Instruction. International Journal of Online
Pedagogy and Course Design, Vol 1, No. 1, pp. 12-28.
Verbene, S., 2010. In Search of the Why: Developing a system for answering why-questions. Doctoral Dissertation,
Radboud Universiteit, Nijmegen, The Netherlands. http://repository.ubn.ru.nl/handle/2066/76174
Verhoeven L., 2009. Cognitive load in interactive knowledge construction. Learning and Instruction. Elsevier, pp.1-7.
Walton, D. (2004). A new dialectical theory of explanation. Philosophical Explorations: An International Journal for the
Philosophy of Mind and Action. Vol 7 No.1, pp. 71–89. doi:10.1080/1386979032000186863
64
RECOGNITION VS REVERSE ENGINEERING IN

BOOLEAN CONCEPTS LEARNING
Gabriel Shafat and Ilya Levin

Tel Aviv University
Ramat Aviv, Tel Aviv 69978 Israel
ABSTRACT
This paper deals with two types of logical problems - recognition problems and reverse engineering problems, and with
the interrelations between these types of problems. The recognition problems are modeled in the form of a visual
representation of various objects in a common pattern, with a composition of represented objects in the pattern. Solving
the recognition problem may therefore be understood as recognizing a visually-represented Boolean concept, with further
formulation of the concept. The recognition problems can be perceived as a parallel process, so the recognition problems
are considered a parallel type. Alternatively, solving a reverse engineering problem means reconstructing a Boolean
function/concept implemented within a given “black box”. Since such a reconstruction is typically performed
sequentially, step by step, this type of problem can be considered a sequential type. We study the above two types of
problems for the same set of Boolean concepts and compare the corresponding solutions obtained by a group of students.
The paper presents results of experiments that study how the complexity of Boolean concepts affects the students’
success in solving parallel and the sequential type problems respectively.
KEYWORDS
Boolean concepts, Recognition and Reverse engineering problems.
1. INTRODUCTION
Concepts are the atoms of thought and they are therefore at the nucleus of cognition science (Fodor, 1994).
People begin to acquire concepts from infancy and continue to acquire and plan new concepts throughout
their entire lives (Medin, Lynch, & Solomon, 2000; Medin & Smith, 1984). One way to create a new concept
is by utilizing existing concepts in different combinations. One of the problems in learning concepts is
determining the concept’s subjective difficulty. Logical thinking is the key to a wide variety of complex
problem solving and decision making processes and therefore Boolean concepts are essential. An important
aspect of concept learning theory is the ability to predict the level of difficulty in learning different types of
concepts. In this respect, Boolean concepts are a fundamental topic in the literature. Boolean concepts can be
defined by a Boolean expression composed of basic logic operations: negation, disjunction (“or”), and
conjunction (“and”). These types of Boolean concepts have been studied extensively by Shepard, Hovland,
and Jenkins-SHJ (1961), (Bourne 1966, 1974), (Nosofsky, Gluck Palmeri, McKinley, and Glauthier 1994).
These studies focused on Boolean concepts with three binary variables, where the concept receives “1” for 4
out of 8 possible combinations and “0” for the remaining 4 combinations. Since some of the 70 possible
Boolean concepts are congruent, they can be categorized as the same type into six subcategories. The six
subcategories with structural equivalence can be described in a geometrical representation using cubes
(Figure 1).
This notion of congruence seems to have been first introduced by Aiken and his colleagues (Aiken & the
Staff of the Computation Laboratory at Harvard University, 1951) and subsequently became prevalent in the
literature on the theory of switching circuits. It was introduced into psychology by Shepard, Hovland and
Jenkins (1961).
65
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 1. Shepard, Hovland, and Jenkins (SHJ) category types.
1.1 Cognitive Complexity of Boolean Concepts

Concept subcategories with structural equivalence belong to the same category and are defined as a Type.
The study results pertaining to the six types of concepts presented in Fig1 showed that the concepts
belonging to Type1 are the simplest to learn and the subgroup of concepts belonging to Type6 are the most
difficult, according to the following order: Type 1<Type2<(Type3,Type4,Type5)<Type6. The results of this
study are highly influential since SHJ proposed two informal hypotheses, the first being that the number of
literals in the minimal expression predicts the concept’s level of difficulty. The second hypothesis is that
ranking the difficulty among the concepts in each type depends on the number of binary variables in the
concept. The concept subcategory in Type1 has one variable, the subcategory concept in Type2 has only 2
variables, and concept subcategories in Types2 to 6 contain three variables.
Feldman (2000), based on the conclusions from the SHJ study, defined a quantitative relationship
between the level of difficulty of learning Boolean concepts and the concept’s Boolean complexity.
Assuming that D is the number of binary features and P is the number of combinations out of all the
combinations in which the Boolean concept receives “1” (SOP-Sum Of Products), D[P] indicates the family
of Boolean concepts with D variables and P combinations where the concept receives “1”. For example: the
concepts that were examined by SHJ were represented as 3[4], 3 binary variables and 4 combinations in
which the concept receives “1”. In his study, Feldman examined the 3[2], 3[3], 3[4], 4[2], 4[3] concept
family. Feldman also addressed the family of concepts where the number of combinations with “1” differs
from the number of combinations with “0”. For example, the concept 3[3] contains 3 combinations with “1”
and 5 combinations with “0”, unlike its mirror concept 3[5], which contains 3 combinations with “0” and 5
combinations with “1”. The complexity measure of a Boolean concept as defined by Feldman is the number
of literals in the most minimal expression that represents the concept’s complete SOP. According to
Feldman’s definition of the complexity of the concepts and use of heuristic minimizing technique to the
minimum literals, in SHJ’s classification model the complexity measures in each category are: Type1: 1,
Type2: 4, Type3: 6, Type4: 6, Type5: 6, Type6: 10. According to this complexity measure, it is possible to
predict that concepts from Types3, 4, 5 have the same complexity and are learned more easily than Type6 but
are more difficult to learn than Types1 and 2. These complexity measures predict difficulties in learning
Boolean concepts, as examined by SHJ.
Since there are several techniques for reducing an expression to the minimum, Vigo (2006) presented use
of the QM (Quine-McCluskey) technique to obtain a correct minimal description and showed that it is
possible to minimize the expressions more correctly than what Feldman presented in the heuristic technique.
According to Vigo, based on Feldman’s same definition as a measure of cognitive complexity in SHJ’s
classification model, the complexities of functions from each type are given in Table 1.
The definition of the Boolean concept’s complexity as a minimal number of literals in a minimal
expression creates several problems. The first: because the complexity is defined as the number of literals in
the minimal expression and the expressions can be minimized using several techniques, a uniform
complexity measure cannot be obtained. For example: according to Feldman’s heuristics, Types3, 4, 5 have
the same complexity. Contrary to Feldman, in a more correct minimal expression according to Vigo,
concepts from Types2 and 3 have the same complexity.
The second problem: studies show that the Boolean concept “xor” as an operator is learned and acquired
as a concept in human thought to the same degree or more easily than the Boolean concept “or” as an
operator (Evans, Newstead, & Byrne, 1993). By using the “xor” operator, some of the Boolean expressions
examined by Feldman and Vigo can be simplified significantly and therefore, the complexity decreases
according to the definition as a minimum of literals in the minimal expression. In light of the problems
presented, Feldman and Vigo developed alternative theories for Boolean complexity as a measure of
predicting the difficulty in learning Boolean concepts.
66
Table 1. The six concepts in SHJ and their descriptions according to Feldmans heuristic, correct minimal descriptions,
minimal descriptions using “xor” and Structural Complexity (SC)
The SHJ six Feldman’s correct minimal descriptions minimal descriptions SC

concepts heuristic using “xor”
Type13[4] a (1) a (1) a (1) 1.66
Type23[4] ab  a b (4) ab  a b (4) a  b (2) 2.00
Type33[4]  
a b c  a b c (6) a c  b c (4) a c  b c (4) 2.14
Type43[4] a  b c   a b c (6)  
c a  b  a b (5)  
c a  b  a b (5) 2.14
Type53[4] a  b c   a b c (6) a  b c   a b c (6) a   bc  (3) 2.34
Type63[4] a  b c+c b   a  b c  b c  (10) a  b c+c b   a  b c  b c  (10) a  b  c (3) 4.00
Aware of these difficulties, Feldman (2006) has recently introduced his spectral decomposition model. In
this updated model, the complexity of a concept is driven by its decomposition into a set of underlying
regularities. The basic idea is that learning from examples involves the extractions of patterns and
regularities. The formal model describes how a pattern (expressed in terms of a Boolean rule) may be
decomposed algebraically into a “spectrum'' of component patterns, each of which has a simpler or more
“atomic'' regularity. Regularities of a higher degree represent more idiosyncratic patterns while regularities of
a lower degree represent simpler patterns in the original decomposed pattern. There are two kinds of simple
concepts: those that consist of a single constant value of a variable, and those that consist of an implication
between the values of two variables. These two basic types of concepts can be algebraically combined to
represent more complex linear concepts. This step is based on an analysis of a concept’s ‘‘power spectrum’’.
Thus, any concept can be decomposed into a set of underlying rules, each of differing degrees of complexity,
depending on the number of variables that they instantiate. The weighted mean of the complexity of these
underlying rules then provides an overall index of the algebraic complexity of the concept. The model yields
a complexity index for any Boolean concept. This model is characterized by its ability to explain unique
phenomena of the learning process of concepts. The algebraic complexity makes it possible to rank each
concept combination according to its structure.
Vigo (2009), developed an alternative theory for calculating the complexity measure of a Boolean
concept, defined as structural complexity. The theory is based on a Boolean derivative. The Boolean
derivative was introduced by Reed (1954) in a discussion of error-correcting codes in electrical circuits. The
basic concept has been mainly relegated to this very specialized domain of applied Boolean algebra.
The question that the theory is supposed to address is: What is it about the internal structure of Boolean
concepts from any category that makes them harder to learn than concepts from a different category? For the
purpose of quantitative definition of the structural complexity, the degree of categorical invariance must be
calculated. In order to quantify the structural complexity, several calculations that will be presented below
must be conducted. For a Boolean function with n variables derived from the Boolean part similar to the
partial derivative of a mathematical function defined as follows:
F  x1 , x2 ,.. xi ,.. xn 
 xi

 F  x1 , x2 ,.. xi ,.. xn   F x1 , x2 ,..... xi ,... xn 
Let F be a Boolean expression that defines a Boolean category. The logical manifold or Lm of F in
respect to x is defined as:
Lm 
F 0  Fp 
F0
67
ISBN: 978-989-8533-12-8 © 2012 IADIS
were:
 F  x    F  x  
F 0  m in te rm s su m o f  F  x  , F p i  m a x te rm s su m o f    
  x i    x i 
The degree of structural complexity (SC) of a Boolean category defined by the Boolean expression F and
belonging to the family D[P] is indirectly proportional to its degree of categorical invariance and directly
proportional to its cardinality SOP(F):
1
 1

 2
n

 Lm i 
2
S C  F0   1
  i 1 
 
Vigo’s (2009) account of the invariance of concepts, as he acknowledges, does not specify how
individuals learn concepts. He suggests only that cognitive processes could detect invariances by comparing
a set of instances to the set yielded by the partial derivative of each variable. For SHJ’s six categories, the
complexity measures described above appear in Table1.
1.2 Recognition Problems

The recognition problems are modeled in the form of visual representation of various objects in a common
pattern, with composition of thus represented objects in the pattern. Solving the recognition problem may
thus be understood as recognizing a visually-represented Boolean concept, with further formulation of the
concept. The recognition problems can be perceived as a parallel process, so the recognition problems are
considered of a parallel type.
One of the important roles of human consciousness is to reveal patterns and find data sequences. Not all
the patterns leave the same impression on people as a basis for identification and perhaps subsequently,
identical patterns are not equally observed by different people.
1.3 Reverse Engineering (RE)

The process of finding and reconstructing operating mechanisms in a given functional system of a digital
electronic apparatus is defined as Reverse Engineering (RE) (Chikofsky & Cross, 1990). RE is applied in a
wide variety of fields: competition in manufacturing new products, from electronic components to cars,
among competing companies without infringing upon the competing company’s copyrights, replacing system
components with refurbished or spare parts (Ingle, 1994), solving problems and defects in a system (Eilam,
2005). RE can be referred to as a certain type of problem solving.
A reverse engineering problem means reconstructing a Boolean function implemented within a given “black
box”. Since such a reconstruction is typically performed sequentially, step-by-step, this type of problem can
be considered a sequential type.
When the values in the variables in a Boolean function express declarations or claims in specific
combinations, the result of the expression is not a Boolean concept but “true” or “false”. For example:
“Switch one and switch two are on or switch three is off”. People are capable of making such claims by
examining the combinations. Sequentially examining a series of combinations makes it possible to reveal all
the possible combinations that give a “true” or “false” value. A mental model for the possibilities that give a
“true” value (a mental model demonstrates the “true” principle) has been built based on a discovery of the
combinations. The mental model is translated into a representation of the system using literals, by
minimizing the variables that do not affect the “true” result (Johnson-Laird, 2006).
This paper deals with two main questions. The first: What effect does the cognitive complexity of a
Boolean concept have on the success of solving recognition and reverse engineering type problems? What is
the ratio between the cognitive complexity of a Boolean concept and the complexity of solving recognition
and reverse engineering type problems?
Reverse engineering problems were represented as a black box that can be used to control the lighting of
a bulb using independent switches. Recognition problems were given as a pattern containing geometric
shapes.
68
We study the above two types of problems for the same set of 9 Boolean concepts and compare the
corresponding solutions obtained by a group of students.
2. EXPERIMENT
The experiment was conducted in two stages for 9 concepts, where each concept was described by means of
a Boolean expression in Table 2.
During the first stage, RE problems were examined using a black box that could be used to control the
lighting of a bulb using three independent switches for seven concepts with three variables (concepts 1-7,
Table2) and using four independent switches for two concepts with four variables (concepts 8 and 9, Table2).
The participants were required to try the different switch combinations that light the bulb and describe the
combinations that light the bulb for each of the 9 concepts using a Boolean expression. A maximum of 5
minutes was allocated for each of the 9 tasks. The tasks were presented as a simulation on a computer
monitor and the time taken to complete each task was measured. Successful completion of the task was
measured based on correct solving during the allotted time.
During the second stage, recognition problems were examined using a questionnaire with nine patterns,
where each pattern represents one of the nine concepts examined, respectively, Figure 2 presents patterns for
concepts 1, 5 and 8. The participants were asked to describe each of the patterns with as few literals as
possible. A maximum of 45 minutes was allocated to completing the questionnaire for each of the nine
patterns. The two stages of the experiment were conducted two days apart.
Figure 2. Patterns for concepts 1, 5 and 8 respectively tested during the experiment.
Successful completion of the tasks solving both RE problems and the recognition questionnaire was
measured based on correct solving in the allotted time. All the solutions were examined compared to two
complexity measures: complexity according to a minimal representation of the expression (including using
the “xor” operator) and complexity according to structural complexity (see Table2)
An example for calculating structural complexity for concept 1 (PN-1 in Table2):
F10  a , b , c   b  a  c    1, 4, 5 
 F1  F1
a

 F1  a , b , c   F1 a , b , c     0, 4   F ap 
a
  1, 2, 3, 5, 6, 7 
 F1  F1
b

 F1  a , b , c   F1 a , b , c    1, 3, 4, 5, 6, 7   F bp 
b
   0, 2 
 F1  F1
c
 
 F1  a , b , c   F1 a , b , c    0,1   Fcp 
c
   2, 3, 4, 5, 6, 7 
 F10  Fap F10  Fap F10  Fap 
Lm  a , b , c    , , 
 F10 F10 F10 
 


 1, 4, 5    1, 2, 3, 5, 6, 7  ,  1, 4, 5     0, 2  ,  1, 4 , 5     2, 3, 4 , 5, 6, 7     2 , 0, 2 
 
 3 3 3  3 3
1 1
 n 1
  1

 2 2   2 2 2 
2 2

SC  F0    Lm i    1  3       0      1
  1.544
  i 1    3   3  
    
69
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.1 Method
The research population included thirty 1st year students studying for a Bachelor of Engineering degree at a
college. Twenty students studied in the Department of Electric and Electronic Engineering and 10 students
studied in the Software Engineering Department. All students studied the Logic Design course in the same
study group and with the same lecturer. All students completed the course successfully with the final exam
average grade of 75.
Table 2. The 9 concepts were tested during the experiment and their descriptions according to correct minimal
descriptions-MD, minimal descriptions using “xor”-MD (xor) and Structural Complexity (SC)
PN FCN MD MD (xor) SC Success (%) Success (%)
RE problems Rec. problems
1 Type13[3] b  a  c  (3) - 1.54 100 100
2 Type33[4] a cb c (4) - 2.14 90 70
3 Type33[4] a b  ab (4) a  b (2) 2.14 100 80
4 Type43[4] a  b  c   bc (5) - 2.14 100 90
5 Type53[4] a 
 b c  a b c (6) c   ab  (3) 2.34 60 40
6 Type63[4]   
a b c+c b  a b c  b c (10)  a  b  c (3)
4.00 40 30
7 Type63[4]   
a b c+c b  a b c  b c (10)  a  b  c (3) 4.00 60 30
8 Type44[5] a  b  c  d   b  d  c   cd (9) - 4.48 90 80
9 Type44[5]
  
a b  c  d  b d  c  c d (9)  - 4.48 90 70
An experimental environment using Lab View was developed on a computer monitor for “black box” RE
problems. The monitor displayed a black box with switches and a bulb. The state of the switches could be
changed by clicking on the appropriate key with the mouse. A change in the switch’s state resulted in a color
change from black to red and the written indication “off” (black) and “on” (red). According to the switch’s
state, the light bulb is either on or off. A lit bulb is green and has the word “on”, and a bulb that is off is with
the word “off”. As soon as the participants reached the conclusion that they knew the appropriate logical
function for the system of switches and the bulb, they were asked to write the states of the switches that light
the bulb using a Boolean expression, written using an equation generator, and to press “finish”. Time was
measured from the moment the first switch in the box is pressed until the task was completed. The course of
the experiment was filmed and a brief interview was conducted at the end of the task. The objective of the
interview was to examine the solution strategy employed by the participants to discover the functionality
regarding each of the nine tasks. The video clips were analyzed to compare the strategy that the students
stated during the interview and the solution strategy as observed in the video clips.
A two-part questionnaire was developed for recognition problems. The first part presented a pattern with
eight geometric shapes. Large and small triangles in red and blue, and large and small circles in red and blue.
The shape was defined using binary variable a, size was defined as binary variable b and color was defined
using binary variable c. Seven patterns were then presented, where each of the patterns matched the
examined Boolean concept (Table2), respectively. The second part presented a pattern with 16 geometric
shapes. Large and small triangles in red and blue and large and small circles in red and blue, with a nucleus
in the center of the shape and without a nucleus. The shape was defined using binary variable a, the size was
defined using binary variable b, the color was defined using binary variable c, and the nucleus was defined
using binary variable d. Time was measured from the commencement of solving the questionnaire to when
each participant submitted his questionnaire.
70
3. RESULTS AND CONCLUSION

By examining the results and conclusions, we will assess the effect the various complexity measures as
presented in Table 2 have on the success, or lack thereof, in solving the two types of problems – recognition
and RE problems, and the relationship between them. Additional aspects that we will examine are the effect
of the “xor” operator on complexity, the “true principle” in RE problems, meaning whether the tendency is to
focus on combinations that give “1” also for problems in which the number of combinations whose result is
“0” is significantly less than the combinations that give “1”.
Table 2 presents the success rates for solving RE problems and recognition problems for all the nine
concepts presented in the Table. The results show that participants are more successful in solving RE
problems than solving recognition problems. Not a single participant that did not succeed in solving RE
problems managed to solve recognition problems for the same concept. However, not all the participants that
managed to solve RE problems were also successful in solving the recognition problems. For example, for
problem 3, 27 participants managed to solve the RE problem, compared to 24 of them that also successfully
solved the recognition problem. Difficulty in comprehending and learning a Boolean concept cannot be
predicted based on Boolean complexity measures of the concept alone, but rather it also depends on the type
of problem representing the concept. It can be concluded from the results that for problems where the
information is absorbed concurrently, recognition problems are more difficult to learn than problems where
the information is sequentially obtained, in this case RE problems. Therefore, the difficulty not only depends
on the concept’s complexity but also on the complexity of the manner in which the problem is presented.
The greater the complexity measure, the lower the success rates, except for PN4, PN8 and PN9, which we
will elaborate on later. It cannot be unequivocally concluded that the greater the complexity measure the
lower the success rate for solving the problem. For problems 1, 2-3, 5, 6-7, the success rates for solving both
types of problems indeed decreases as the complexity measure increases, also according to the MD and SC
measures. For problem 1, the complexity is the lowest and the success rate the highest. For problems 2 and 3,
the complexity is slightly higher than problem 1 and the success rates are consequently smaller. The situation
is the same for problems 6, 5, and 7, respectively. The inverse relationship between the complexity and
success rate also manifests in the average time required to solve the problems. As the complexity increases,
the average time it takes to solve the problem increases accordingly.
Complexity with minimal literals when the “xor” operator is involved is a measure that is the least
predictive, relative to MD and SC, of the subjective difficulty of successfully solving the problem. The
majority of participants did not recognize the “xor” operator in both types of problems. Participants that
grasped the “xor” concept as an operator to the same degree as the “or” and “and” concepts were more
successful in solving the problem and their average time was substantially lower. Eight out the thirty
participants consistently used “xor” in problems 6, 3, and 7. All of the eight participants arrived at the correct
solution for both types of problems at an average time of 30 seconds, which is much lower than the average
time it took the total number of participants. Out of twenty two participants that did not use “xor” but that did
use the other operators, sixteen participants succeeded in solving problem 3 as a recognition problem. For
problem 6, four out of twenty two participants managed to solve the RE problem and nine out of twenty two
managed to solve the recognition problem. For problem 7, ten out of twenty two managed to solve the RE
problem and one managed to solve the recognition problem. It can be concluded from the results that if the
“xor” operator is acquired as a concept to the same degree as the operators “or” and “and”, the concepts’
level of complexity decreases, the success rates increase, and the difficulty in solving the problem decreases.
However, not everyone acquires the three concepts – “xor”, “or” and “and” to the same degree. For most, the
“xor” concept is more difficult to grasp than the other two concepts.
Although the level of complexity of the concepts in problems 4, 8, 9 are higher relative to problems 1, 2,
3, the success rates for solving both types of problems is significantly higher and the average time needed to
solve them is much lower. It can be hypothesized that the reason for this is that the concepts in problems 4, 8,
9 fulfill qualities of symmetric functions. Apparently, the MD and SC complexity measures are not
sufficiently reliable in predicting the level of difficulty in solving the problems for symmetrical functions.
Based on the hypothesis, it is interesting to examine the effect of qualities of Boolean functions such as
symmetry, linearity, etc. on the complexity of Boolean concepts.
Two RE problem solving strategies were observed. The first, adopted by twenty seven out of the thirty
participants, was to attribute a logical value to one of the variables and conduct fewer checks of the
71
ISBN: 978-989-8533-12-8 © 2012 IADIS
combinations to a check times 2 consistently, building a mental model for the combinations in which the
transitions among them, a change in one of the variables does not influence the state of the lit bulb. For
problems 8 and 9, only ten out of twenty two participants reached the solution for the states in which the bulb
is not lit and managed to solve the problem, since the number of combinations in which the bulb is not lit is
4, versus 12 states in which the bulb is lit. They took the same approach with recognition problems in solving
problems 8 and 9, and succeeded. For the remaining participants, the “true principle” guided them in solving
all the states, including the ones where it is more effective to examine the states in which the bulb is not lit,
which were significantly lower than the number for states in which the bulb is lit. Three students tended to
use the strategy of covering all the possible states. They managed to reach the correct solution only for RE
problems for concepts 1, 3, 4, and did not succeed in reaching the required solution for the other states.
REFERENCES
Aiken, H. H., & the Staff of the Computation Laboratory at Harvard University, 1951. Synthesis of electronic computing
and control circuits. Cambridge: Harvard University Press.
Boole, G., 1854. An investigation of the laws of thought on which are founded the mathematical theories of logic and
probabilities. New York: Dover
Bourne, L. E., 1966. Human conceptual behavior. Boston: Allyn and Bacon.
Bourne, L. E., 1974. An inference model for conceptual rule learning. In R. Solso (Ed.), Theories in cognitive
psychology. Washington, Erlbaum, pp. 231–256.
Chikofsky, E.J., & J.H. Cross II, 1990. Reverse Engineering and Design Recovery, A Taxonomy in IEEE Software. IEEE
Computer Society, January, 13.17.
Evans, J. S. P. T., Newstead, S. E., & Byrne, R. M. J., 1993. Human reasoning, The psychology of deduction. Hillsdale,
NJ: Erlbaum.
Eilam, E., 2005. Reversing: Secrets of reverse engineering, New York: Wiley.
Feldman, J., 2000. Minimization of Boolean complexity in human concept learning. Nature, 407, 630–633.
Feldman, J., 2006. An algebra of human concept learning, Journal of Mathematical Psychology, 50, 339_368.
Fodor, J., 1994. Concepts, A potboiler. Cognition, 50, 95–113.
Ingle, K.A., 1994. Reverse engineering. New York: McGraw-Hill.
Johnson-Laird, P. N., 2006. How we reason. New York: Oxford University Press.
Medin, D. L., & Smith, E. E., 1984 Concepts and concept formation. In M. R. Rosensweig & L. W. Porter (Eds.), Annual
review of psychology , Vol. 35, pp. 113–118.
Medin, D. L., Lynch, E. B., & Solomon, K. O., 2000. Are there kinds of concepts? Annual Review of Psychology, 51,
121–147.
Nosofsky, R. M., Gluck, M. A., Palmeri, T. J., McKinley, S. C., & Glauthier, P. G., 1994. Comparing models of rule-
based classification learning, A replication and extension of Shepard, Hovland, and Jenkins, 1961. Memory and
Cognition, 22(3), 352–369.
Paterson, M. S. (Ed.)., 1992. Boolean function complexity. Cambridge, Cambridge University Press.
Reed, I. S., 1954. A class of multiple error-correcting codes and the decoding scheme. IRE Trans, on Information
Theory, Vol. IT-4, 38_49.
Shepard, R., Hovland, C. L., & Jenkins, H. M., 1961. Learning and memorization of classifications. Psychological
Monographs: General and Applied, 75(13), 1–42.
Vigo, R., 2006. A note on the complexity of Boolean concepts. Journal of Mathematical Psychology, 50(5), 501_510.
Vigo, R., 2009. Categorical invariance and structural complexity in human concept learning. Journal of Mathematical
Psychology, 53, 203–221.
Wegener, I., 1987. The complexity of Boolean functions. Chichester: Wiley.
72
IN SEARCH OF AUTHENTIC LEARNING IN

KAZAKHSTAN
Francisco Chia Cua1 and Stephen A. Reames2

1
KIMEP University, Almaty, Kazakhstan
2
A’Sharqiyah University, Ibra, Oman
ABSTRACT
The foreign faculty in Kazakhstan faced the challenge of designing a total authentic learning curricula for undergraduate
students used to a traditional approach to learning. Students were asked to submit reflective learning journeys at the end
of the spring 2011 semester. This study uses their stories as a springboard to explore their perspective of authentic
learning. The case study was confined to data gathered from one class, which revealed that those students who remained
in the course wanted to be challenged by authentic learning. To them, authentic learning was both a force driving their
evolution from passive to active learners and a means for solving the real-world problems with a community of co-
learners. Authentic learning is an agent of change that is worth contemplating. Its design can apply to the multiple life-
spaces of a single student and other disciplines.
KEYWORDS
Authentic learning, cognitive learning, empowerment, teaching
1. INTRODUCTION
Kazakhstan (KZ) is the 9th largest country in the world, with about sixteen million people and ninety higher
education institutions. Newsweek (2010) ranked KZ 14th worldwide in education, with a rating of 91.35%
and a literacy rate of 99.5%. In Almaty, the KIMEP University has been ranked number one.
Kazakhstani students are generally ambitious. For example, many MBA students would overload
themselves concurrently in study and in work and would resort to complaining as a survival mechanism to
cope with the unrealistic pressure (Misch & Galantine, 2009).
A preference for rote learning (Lillis, 2009) appeared during the inquiry. It was further validated by first-
hand experiences with students during the fall semester of 2010. Nevertheless, many students would want to
be challenged by a demanding learning experience (McInnis, 2001). Therefore, authentic learning was
introduced. However, it posed a huge challenge to faculty.
1.1 How did the Study Come About?

McInnis and Newsweek provided reasons to the belief that an authentic learning curriculum could be
workable in an institution that claims to be world-class. As a consequence, the researcher incorporated
authentic learning in all his classes in the spring semester of 2011. In this semester, he facilitated a Financial
Statement Analysis (FSA) class (and two classes of Financial Accounting 2). The FSA was double-coded as
ACC4208/FIN4214 for both senior and graduating Accounting- and Finance-students. Two sections ran
concurrently from 17:30 to 19:30 every Tuesday and Thursday.
In the first section, a Kazakhstani instructor utilized a traditional teaching approach. His course consisted
of five short exams, two projects, and a forty-percent final exam covering the first twelve chapters of Penman
(2010). In the second section, the researcher approached the course non-traditionally.
The second section covered Chapters 13 and 14 of Stickney et al. (2007). All chapters of Penman and
other chapters of Stickey are resources to thoroughly understand these two chapters in focus. The syllabus
required each student to engage inquiry and to prepare evidence-based portfolio, which made up forty percent
of the summative assessment. The learning portfolio included a project of student’s own choosing, a
73
ISBN: 978-989-8533-12-8 © 2012 IADIS
reflective learning journal, and a mind map. The project required an executive summary with spreadsheets
and other pieces of supporting evidence.
Students are empowered to work on the project alone or in small team. Moreover, they would transform
themselves to be reflective storytellers and to see their higher education as a construction of personal and
social stories.
This second section went through several transition states. During the first two weeks, sessions blended
traditional lectures (to introduce theories) with cooperative learning methods (to inspire participations).
While exploring the strengths and interests of the FSA students, the researcher learned of a student who
worked at one of the Big Four accounting firms. The student was allotted an entire session to share his
knowledge about the valuation of firms in the real world. How students reacted, interacted, and responded to
the presentation and to the idea of teaching themselves weighed against tradition.
The time seemed ripe to advance the learning to the next level. Bonus projects were add. Similar to the
40% learning portfolio, students were empowered to select their own topic(s) and to organize themselves as a
community of learners.
Each student was distributed a yellow card. Three questions on the front side formalized their
commitments (1) to conduct independent active in-depth learning engagements; (2) to submit weekly
reflective journals; and (3) to undertake evidence-based and inquiry-based project in lieu of a written final
exam. If students answered No to any of the three questions, they were advised to transfer to the first section.
The yellow card would also be stamped for attendance (the front side) and student participation (the back
side), in order to encourage them to show up for class and to take part in the engagements.
The important feature of the authentic learning curriculum was the no examination policy. The other sixty
percent consisted of (1) participation in the two focus-group discussions; (2) weekly reflective journals; (3)
eight assignments (some of which could be done in groups); and (4) an essay. Students were also asked to
read the facilitator’s teaching philosophy and to write their own (5) learning philosophy. Rubrics were
designed to rate the assessments. The two focus-group discussions took place two weeks into the semester
and two weeks prior to its end. Students reflected on their learning points. They also had the opportunity to
participate in (6) an in-depth interview with an officer in the Research and PhD department. All these, except
for the bonus projects, were designed before the start of the spring semester.
Assessing team work was a challenge. To avoid free loaders in the group, the team needed to detail the
distribution of the jobs. Grade would be a one step lower if the distribution was equally divided among the
members. In real life, it is unlikely that a work could be shared evenly.
2. AUTHENTIC LEARNING: ITS ESSENTIAL ATTRIBUTES

“The main point of authentic learning is to let students encounter and master situations that
resemble real life” (Cronin, 1993, p. 79).
Authentic learning originated two decades ago as a simple idea of apprenticeship (Resnick, 1987). Such
situated learning (Collins, 1988; Collins et al., 1989) connects theoretical learning to practical learning.
The learning engagemens in classroom setting presuppose more authenticity (Cronin, 1993, p. 79) or
more situation likely to be encountered in real life. The essence of authentic learning in curriculum design
explores and exploits all available opportunities. The mere use of authentic learning tools in a traditional
learning curriculum does not make it an authentic learning curriculum.
One essential feature of authentic learning is segmentation. The traditional learning is deeply engrained in
many students. Authentic learning is radical. It disrupts the comfort of many students. The research of Brown
and Venkatesh (2003) in innovation shows that sixteen percent are pro innovation and eighty four percent are
against it. However, forty percent of those people who resist could be persuaded to reduce their negative bias
and to accept the innovation in the end. Based of this insight, students who accept authentic learning belong
to a special segment.
Second is the conception as agent of change. Authentic learning connects the “in-here” knowledge
(thoughts, beliefs, feelings, and dreams) with the “out-there” world (the physical and material domains). The
connectedness presupposes a heightened “in-here” awareness of one’s own learning engagements when
making sense of the “out-there” reality. It is useful to look at authentic learning from a deterministic or
74
instrumental perspective. In the first perspective, the learning is a force that imposes change and becomes
part of a web of influences that is always in motion and constantly influences the community of learners. In
the second perspective, the learning comes with perceived utilities or benefits that are a means to an end,
which is the change. Any impediments could be mitigated by heightening the construction of meaning, the
cognitive presence or the social presence during the four cognitive stages (Garrison et al., 2010; Garrison &
Arbaugh, 2007). Wolf (2001) refers the change as a new alchemy of mind.
The third feature concerns the knowledge construction stages (or the knowledge development stages).
Constructing and experiencing knowledge passes four cognitive stage (Garrison et al., 2001). First of the
learning experience is the Agenda-setting stage. It is marked by the perception of the relative newness and
values (Rogers, [1962] 2003) of the new curriculum. Second, the Exploration stage facilitates the
examination of problems and encourages a deep approach to learning. The ideas produced in this stage are
constructed into personal meaning during the Integration stage. Finally in the fourth stage, the Resolution
stage links the ideas and meanings and evidence and reasoning in a chain. The need for heightened cognitive
and for heightened social presence comes the need for heighten thinking (Lesh et al., 1999; Rule, 2006), such
as asking questions, conducting studies, drawing conclusions, thinking about thinking (metacognition),
revising theories, and communicating results and discourse (Akyol and Garrison, 2011; Callison and Lamb,
2004; Flavell, 1979; Renzulli et al., 2004; Vygotsky, 1978).
Fourth, a conception of authentic learning is as constructed knowledge. According to constructionism
(Papert, [1980] 1993), knowledge is constructed through interactions of the community of learners, the
creation of social artefacts (e.g., projects), and the building of new knowledge on old (prior) knowledge
through above four cognitive stages. Papert ([1980] 1993) uses the metaphor of communal property (Barron,
2004; Brandsford et al., 2006).
Fifth is empowerment, allowing students to make the choice of what, when, where, how, and with whom
the learning occurs (Barron, 2004; Kirby & Goodpaster, 2006). This puts the student’s chosen focus in an
individual personal frame (Kellouth, 2003; Rule, 2006) so that the learning activities remain student centered
(Callison & Lamb, 2004; Maina, 2004, Renzulli et al., 2004). Empowerment also presupposes the diversity
of learning pathways (Brandsford et al., 2006), with no single method for carrying out tasks (Bergeron &
Rudenga, 1996). However, empowerment accepts the non-empowered dimensions, such as codified
knowledge (concepts, facts, and procedures), learning strategies, and the learning environment. The latter set
influences observation, coaching, and practice (Lave, 1988).
3. RESEARCH METHODOLOGY
The domains of this case study could be imagined as the three layers in an inverted triangle. At the widest
level on top is the local setting of the higher education institution in Almaty, Kazakhstan. In the middle of the
triangle is the complex phenomenon: the authentic learning of the FSA in the spring 2011 semester. The
lowest level of the triangle is the abstract case. The fourteen learning journeys have highlighted four complex
issues of authentic learning (1) as an agent of change, (2) as a process to construct knowledge, (3) as
constructed knowledge, and (4) as empowerment. These four issues constitute the abstract case.
To explore the case and to thoroughly understand its issues, the research uses a tiered approach (Clark &
Linn, 2003). The approach avoids a pitfall of designing one huge multiple-tiered study to address every
single intertwining issue. Nevertheless, it assures that the complex phenomenon meets the rigor, yields good
data, and maximizes learning (Carver, 2006; Stake, 2005).
Four important theoretical assumptions are essential to analyzing and interpreting the findings. First,
authentic learning presupposes the cognitive readiness among individual students to form a community of
learners, in small groups or in a whole group (the whole class). Second, because not all students want to
undertake the authentic learning, the radical learning approach underscores the segmentation and the
percentage of acceptance. Third, authentic learning cannot happen apart from the contexts. A curriculum
developed for a certain student segment is not automatically applicable other students, other courses, and
other higher institutions unless it is modified or reflected to fit the new contexts. A second implication of this
assumption regards facilitators not as the source of expert knowledge but rather the source of
interdisciplinary knowledge. Fourth, authentic learning assumes that students learn better if the learning is
relevant to their needs.
75
ISBN: 978-989-8533-12-8 © 2012 IADIS
Fourteen students included their learning journeys as a part of their learning portfolio. The journeys
delved into their experiences, beliefs, emotions and thoughts and provide evidence that was otherwise not
measurable nor explicit. The case study utilized these journeys as the primary data source. However, five
journeys were highlighted in this paper. The other (1) nine journeys were used to triangulate the analysis.
Other pieces of evidence used in triangulation included (Yin, 2009): (2) seven fifteen-minute in-depth open-
ended interviews conducted by a colleague at the end of the semester (9th to 11th of May 2011); (3) submitted
assignments; (4) the students’ learning philosophy; (5) the two focus-group discussions; (6) the record books
with their GPAs and English proficiency scores; and observations.
The units of analysis, as indicated in the problem, corollary question, and the four issues in the structured
top-down diagram (see Figure 1), were as follows: (1) the cognitive presence in the Community of Inquiry
(Anderson et al., 2001; Garrison et al., 2001; Rouke et al., 2001); (2) the perceived attributes (Rogers, [1962]
2003) triggering interest and commitment to authentic learning; (3) the stages in constructing knowledge; and
(4) other necessary structures and procedures. The chosen units of analysis align with the three domains and
call for slightly different modes of analysis and inferential perspectives in the Discussions section. A possible
weakness is the inclusion of both structures and processes, but it is warranted by the exploratory nature of the
study. The strength is the researcher’s personal curiosity about the case. This intrinsic interest has made the
inquiry memorable, challenging, and fun.
The foundation of the modes of analysis was the continuous contemplation of the evidence, which
subsequently led to the emergence of insights (the theory). This general inductive approach balanced rigor,
flexibility and iteration in all stages, from the reading of the raw data to the construction of the chain of
reasoning. The analysis of qualitative data became simpler (Thomas, 2006). Issues were meshed to their
common ground (Guba & Lincoln, 2005). During the thinking process, raw textual structural and procedural
descriptions in the learning journeys were highlighted and/or summarized to underscore the experiences of
the teaching and learning processes. The interpretations and findings were linked to the research objective.
The authentic learning of the FSA course in the spring semester of 2010 was an imperfect, real, and
complex phenomenon. The interpretation of the learning journeys was the key to unlock its truth (Denzin &
Lincoln, 2005). Although these journeys convey many shades of truths (absolute truths, empirically adequate
truths, approximate truths, and even partial truths), nevertheless the stories passed through validation,
triangulation, and argumentation in Figure 1.
4. RESULTS – THE CONCEPTS OF AUTHENTIC LEARNING
4.1 Authentic Learning as Segmentation

The curriculum design in the second section was radical in many ways. First, there was the no examination
policy. Second, each student had a yellow card. It was used to inspire participations and at the same time
monitor attendance. Third, the facilitator and the students played the role of co-teachers and co-learners in a
community of learners. Thus, the teaching and learning passed through transitional stages.
Whereas traditional learning is deeply engrained in many students, authentic learning disrupts the comfort
of many students. Thus from initial enrolment of sixty one students, twenty one students remained and, with
the facilitator, formed a community of learners. These students were equivalent to thirty four percent of the
population (21 divided by 61 equals 34.4262%). Finally, sixteen students passed the course. Fourteen
students submitted their learning journeys. Many included mind maps. About half got an A+ or an A.
The 34% outcome was consistent to the research of Brown and Venkatesh (2003), arguing that sixteen
percent of the one hundred percent population are initially positive in favor of the innovation (e.g., the
authentic learning) and forty percent of the eighty four percent (that is, 40% * 84% = 33.6%) against it could
be persuaded to accept the innovation in the end. In short, the people of the population who would accept an
innovation range from a minimum of 16% to a maximum of 49.6%. The twenty one students who accepted
authentic learning (the 34%) belonged to the mentioned segment.
76
4.2 Authentic Learning as an Agent of Change

Through authentic learning curriculum, Kazakhstani students evolved from passive learners to active
learners. Student #1 recalled her learning journey, which started with hard knowledge in the textbook and
ended with soft knowledge.
“Generally, the topics we were covering grew more complicated throughout the course, as they
became closer to reality. Indeed, in practice, things are more complicated and not as easily
approachable as described in the textbook. In reality, each company has its own unique features
and performance patterns that need certain adjustments. These adjustments involve more judgment
and soft knowledge.” - Student #1
Likewise, students who conducted actively with their independent in-depth learning engagements realized
a heightened cognitive presence. An evidence is the recognition of unique features and patterns (see previous
quote). Another evidence is the construction of knowledge in stages. According to Student #1, the first hard-
pure stage established the foundation. In this stage, the lectures were entirely delivered by the facilitator. The
second hard-applied stage included simple and straightforward application of theory in practice. The
knowledge was based on textbook examples. Although the lecturer maintained the leading role, the learning
process gradually involved interactions with other students in small groups.
“[The] third … soft-applied stage… was the longest and important stage associated with most
advanced real-life approach to the study. It was aimed to apply the previous experience and
knowledge to existing companies in a more sophisticated real-life environment. It used more
complicated examples… [The] fourth … soft-pure stage…had an essay worth 20% of the grade
that had to cover the practical issues and problems we face[d] while implementing the final project
and others. These problems had to be considered not merely in the frames of the particular project,
but rather in relation to the whole country’s financial system. It was significant to realize how
these problems affect the decision quality made by investors not just in the case of the final
project, but…[with] regard to all companies in Kazakhstan….[Our] understanding of financial
analysis expanded to the scope of the environment and society we lived in.” - Student #1
Student #3 likewise experienced heightened cognitive presence as well as social presence. He recognized
the sensitive factors of capital structure of the firm, realized the privilege of being empowered to choose and
make decision, and the effective learning.
“[W]e decided to take a particular Kazakhstan company, and calculate all of the ratios for it… We
also decided to make our presentation more interactive… [and] to encourage students to be more
active… I understood more about the risk of the company and how it depends on the company’s
capital structure… (italics emphasized)” - Student #3
Heightened social presence accompanied the heightened cognitive presence. Students recognized the
motivation to solve problems as a community of learners.
“The process of working on the bonus projects was amazing… [W]e all were working in the
groups, communicating, observing and looking for the solution together. We all received
support… from each other. … It was a terrific motivational tool for making everyone engaged in
the process, discussions and conversations. The chance to give the lectures to other classmates
makes you responsible for the information that you give to others, as you should understand
everything concerning the topic and should be prepared to explain it in more detail. [The
discussion, a] winning strategy in [learning, making] the class more volatile in reaction to the
learning activities and new topics and are consistent with the studying procedure.” - Student #4
A key feature of authentic learning is the uncertainty. Student #4 anticipated the uncertainty (e.g., “more
volatile in reaction”). Likewise, Student #5 experienced the same feeling.
“The whole course for me was like [a] jump. When I got those yellow cards and all supplementary
materials, it seemed to be clear how, what and when to do the activities… However, soon you are
totally disoriented and don’t know what to do. There is so much uncertainty and you don’t know
how to start… [Teaching and learning] ease the educational process, as there is no need to research
every question; some of them were already done by other students. At some point, I realized that
learning became much easier. Finally, you see the result of an excellent job and realize that it was
not as difficult as it seemed to be at some point.” - Student #5
77
ISBN: 978-989-8533-12-8 © 2012 IADIS
4.3 Authentic Learning as Knowledge Development Stages

The four stages of learning, as conveyed by Student #1, actually correspond to the four types of knowledge,
namely: the hard-pure knowledge learned from the lecturers; the hard-applied knowledge learned from
cooperative learning and problem solving; the soft-applied knowledge, requiring intensive use of
spreadsheets, advanced real-life approaches, and a high level of judgment; and finally the soft-pure
knowledge where students took everything they learned and wove them into their new knowledge. Student #1
understood that the knowledge covered both the classroom issues and the wider context of society.
“Presentations of our co-learners demonstrated, using the example of existing companies, how to
apply the theoretical materials introduced in Chapter 13. This involved an Excel spreadsheet
application... to enhance our learning experience and develop our practical skills. [The]
spreadsheet became the main part of every presentation … to explain [the] practical side of every
topic... Chapter 14 was introduced to widen our learning of broader, abstract and soft topics like
accounting quality and forecasting… Additional bonus projects were undertaken to enrich our
practical experience and go beyond textbook materials. These projects focused on assessing the
risks of existing start-up companies. The objectives of the project were to learn how to obtain
necessary information from online data directories and how to use historical information and
statistics… It is significant to mention that softer skills and professional judgment were used
during this task.” - Student #1
The construction of knowledge did not happen only through group projects. The reflection on the learning
process was imposed as a means to construct knowledge. However, not many students (like Student #2
below), realized the importance of the thinking about the learning process.
“I didn’t like reflections; they took so much time... But work on projects was like fresh air for me.
I liked them most, because we tried to be creative, innovative and tried to implement our
knowledge from all fields. I wish we had more [of] that kind of work… [I also] liked that all our
works were evaluated with comments. (italics emphasized)” - Student #2
Heightened cognitive and social presence also came from the focus-group discussions at the start and
toward the end of the semester, the assignments, and the summative learning journey. This heightened
presence, in turn, was essential to quality knowledge development stages.
4.4 Authentic Learning as Constructed Knowledge

The knowledge development stages underscore the knowledge constructed, which comes in the form of
communal artefacts (projects and situated works) and which is created when teaching and learning methods
connect the individual learners to the community of learners. Student #2 conceived the project as a breath of
fresh air, going beyond textbook materials, enriching practical experience, encompassing knowledge from
many fields, and empowering learners to be more creative and innovative. The emphasis is on collective and
non-isolated knowledge structures. For instance, Chapters 13 and 14 were read not in isolation from other
chapters that were also critical to the understanding of the valuation of the firm. The sense of connectedness
and integration is the heart of constructed knowledge.
4.5 Authentic Learning as Empowerment

Constructed knowledge in authentic learning puts greater emphasis on collaborative inquiry and on the
opportunities for students to make choices and create the epistemic artefact (e.g., the project). Students
realized that collaboration and empowerment to make the learning easier (refer to the reflections of Student
#3, Student #4, and Student #5).
The case study revealed that providing choices to students results in better learning because they see the
learning as context-rich, diversified, and relevant to their needs and motivations. The empowerment is not
limited to what is learned, but includes the environments (schools or classrooms), situations (formal or
informal), and other traditional or non-traditional (radical) alternatives. Yet there do remain unavoidable
mandates, such as, in this case, the focus on Chapter 13 and 14 and on weekly reflection on the learning
process.
78
5. DISCUSSION
The case study has validated the four truth statements corresponding to the four issues in Figure 1. Firstly,
authentic learning acts as an agent of change, helping students develop positive perspectives about the
realities affecting them. Unlike traditional learning, authentic learning demands heightened cognitive
presence, heightened social presence, and heightened (teaching and) learning methods. Such a triple
heightened presence in turns requires (1) a community of learners, (2) the credibility of the facilitator and
students, (3) the innovativeness of the facilitator, (4) the commitment of students to independent in-depth
learning, (5) positive attitudes and beliefs of co-teachers and co-learners, and (6) a positive and empowered
learning environment. All these elements make authentic learning curriculum design an art in development,
facilitation and engagement.
Figure 1. Argumentation
Back to the corollary question: What patterns did the students conceive in their learning? Student #5’s
description of authentic learning as a jump (a force) acknowledge its elements of instability and uncertainty.
Student #1 expressed authentic learning as a means (a journey) to develop one’s codified (hard)
understanding of financial analysis first into judgmental (soft) understanding and later into something even
broader and more abstract.
The researcher found that many students wanted to be challenged by authentic learning. That is, they
were able to see the value of trying to solve the real-world problems with a community of co-learners. Thus,
authentic learning is an agent of change worth contemplating.
The learning journeys examined reveal the readiness of students to commit to authentic learning (the first
assumption in Table 1) and assume that the same commitment was not present in the students who withdrew
from the course. But two students did not submit their learning journeys and could be presumed to have
lower readiness, although they nevertheless passed the course. The fourth segment could be students who had
the readiness but chose to withdraw due to peer pressure. The implied challenge is to find ways to reduce
79
ISBN: 978-989-8533-12-8 © 2012 IADIS
negative bias so that authentic learning appears effective and meaningful.

A second assumption is that authentic learning cannot happen apart from the contexts and must be
modified to fit the individuals, the small groups, the whole class, the course, and the institute of higher
education. For instance, an Introductory Accounting for freshmen would require a different design from the
FSA course.
A third assumption presumes that students learn better if the learning is relevant to their needs or
interests. This takes for granted that students know what they want. Yet there is always the possibility that
they might choose something that would not maximize their learning. The curriculum designer’s concern
becomes how to minimize this risk.
6. CONCLUSION
From the segment of students who remained in the section, the evidence shows that students recognized the
alchemy in the course (Issue #1 in Figure 1). The authentic learning as an agent of change requires
heightened cognitive presence and heightened social presence to be able for the co-teachers and co-learners
to create positive perspectives from the micro level of the learners to the macro level of society.
Second, students evolved from passive learners to active learners through several stages. Aside from the
heightened cognitive presence and heightened social presence, a third pre-requisite in the knowledge
development stages is the heightened teaching and learning methods, such as true inquiry, thinking, thinking
about thinking (meta-cognition), and discourse (Issue #2 in Figure 1).
Third, the construction of knowledge connected the in-here judgment with the out-there performance
patterns of the firm. At the same time, the learning engagements were means to an end, namely, the
summative assessments. Students appeared to recognize both the deterministic and instrumental perspectives.
Thus, the constructed knowledge is the outcome of the authentic learning process. It highlights the learning
environment and the teaching and learning process and experience.
Fourth, an essence of authentic learning is empowerment, which comes with providing students with
adequate opportunities for them to make choices and to design their learning pathways, challenges, and
experiences.
The co-teachers and co-learners in a community of learners need to have heightened awareness of these
elements. The study affirms that only teacher who recognizes authentic learning would design and implement
such a non-traditional curriculum, while only students with a positive biased toward authentic learning would
remain and actively engage in the course. Furthermore, the study confirms that many senior Kazakhstani
students have the readiness and the desire to be challenged by authentic learning. They see themselves
becoming active learners in a nurturing authentic learning environment, and later on, experts in their chosen
fields.
7. DIRECTIONS FOR FUTURE RESEARCH

Authentic learning can potentially cross-pollinate learning opportunities across disciplines (e.g., accounting,
marketing and entrepreneurship) or across settings (e.g., learners belonging to multi-cultures). It also needs a
different balance of informal and formal learning opportunities as people move across multiple life-spaces. It
would be useful to look at how, when, and where learning occurs for people who straddle multiple life-
spaces.
Authentic learning holds students responsible for conducting independent in-depth engagements and also
holds the facilitators responsible for creating and nurturing a harmonious learning environment. These two
assumptions, however, do not apply to students who refused to commit themselves. The future research could
explore ways to capture the reasons for withdrawing from the course. Lastly, this exploratory study is of
instrumental interest to positivist research as an agenda for future research.
80
REFERENCES
Akyol, Z., & Garrison, D. R. (2011). Assessing metacognition in an online community of inquiry. The Internet and
Higher Education, 14(3), 183-190.
Anderson, T., Rourke, L., & Garrison, D. R. (2001). Assessing teaching presence in computer conferencing context. 5(2),
1-17.
Barron, B. (2004). Learning ecologies for technological fluency: Gender and experience differences. Journal of
Educational Computing Research, 31(1), 1-36.
Bergeron, B. S., & Rudenga, E. A. (1996). Seeking authenticity: What is "real" about thematic literacy instruction? The
Reading Teacher, 49(7), 544-551.
Brandsford, J. D., Barron, B., Pea, R. D., Meltzoff, A., Kuhl, P., Bell, P., et al. (2006). Foundations and opportunities for
an interdisciplinary science of learning. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences
(pp. 19-34). New York: Cambridge University Press.
Brown, S. A., & Venkatesh, V. (2003). Bringing non-adopters along: The challenge facing the PC industry.
Communications of the ACM, 46(4), 76-80.
Callison, D., & Lamb, A. (2004). Key words in instruction: Authentic learning. School Library Media Activities Monthly,
21(4), 34-39.
Carver, S. M. (2006). Assessing for deep understanding. In R. K. Sawyer (Ed.), The Cambridge Handbook of the
Learning Sciences (pp. 205-224). New York: Cambridge University Press.
Clark, D., & Linn, M. C. (2003). Designing for knowledge integration: The impact of instruction time. The Journal of the
Learning Sciences, 12(4), 451-493.
Collins, A. (1988). Cognitive Apprenticeship and Instructional Technology (Technical report no. 6899). Cambridge, MA:
BBN Labs, Inc.
Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and
mathematics. In L. B. Resnick (Ed.), Knowing, Learning, and Instruction: Essays in honor of Robert Glaser (pp. 453-
493). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Cronin, J. F. (1993). Four misconceptions about authentic learning. Educational Leadership, 50(7), 78-80.
Denzin, N. K., & Lincoln, Y. S. (2005). Paradigms and perspectives in contention. In N. K. Denzin & Y. S. Lincoln
(Eds.), Handbook of Qualitative Research (3rd ed., pp. 183-190). Thousand Oaks, CA: Sage Publications.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-development inquiry. American
Psychologist, 34(10), 906-911.
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking and computer conferencing: A model and tool to
assess cognitive presence. American Journal of Distance Education, 15(1), 7-23.
Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future
directions. The Internet and Higher Education, 10(3), 157-172.
Garrison, D. R., Cleveland-Innes, M., & Fung, T. S. (2010). Exploring causal relationships among teaching, cognitive
and social presence: Student perceptions of the community of inquiry framework. The Internet and Higher
Education, 13(1-2), 31-36.
Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic controversies, contradictions, and emerging confluences. In N. K.
Denzin & Y. S. Lincoln (Eds.), Handbook of Qualitative Research (3rd ed., pp. 191-215). New York: Sage
Publications.
Kellouth, R. D. (2003). A Resource Guide for Teaching (4th ed.). Upper Saddle, NJ: Pearson Education Inc.
Kirby, G. R., & Goodpaster, J. R. (2006). Thinking (4th ed.). Upper Saddle River, NJ: Prentice-Hall.
Lave, J. (1988). The Culture of Acquisition and the Practice of Understanding (Report No. IRL88-0007). Palo Alto, CA:
Institute for Research on Learning.
Lesh, R. A., Hoover, M., Hole, B., Kelly, A. E., & Post, T. (1999). Principles for developing thought-revealing activities
for students and teachers. In R. A. Lesh & A. E. Kelly (Eds.), Handbook of Research Design in Mathematics and
Science Education. Mahwah, NJ: Lawrence Erlbaum Associates.
Lillis, J. (2009). Kazakhstan: Economic crisis crimps Astana's grand plans for higher education. Retrieved 14 June 2010,
from http://www.eurasianet.org/print/58794
Maina, F. W. (2004). Authentic learning: Perspectives from contemporary educators. Journal of Authentic Learning,
1(1), 1-8.
McInnis, C. (2001). Signs of Disengagement? The Changing Undergraduate Experience in Australian Universities,
Inaugural Professorial Lecturer at Centre for the Study of Higher Education, Faculty of Education, University of
Melbourne.
81
ISBN: 978-989-8533-12-8 © 2012 IADIS
Newsweek. (2010). Interactive infographic of the world's best countries. Retrieved 22 September 2011, from
http://www.thedailybeast.com/newsweek/2010/08/15/interactive-infographic-of-the-worlds-best-countries.html
Papert, S. ([1980] 1993). Mindstorm: Children, Computers, and Powerful Ideas (2nd ed.). New York: The Perseus Books
Group.
Penman, S. H. (2010). Financial Statement Analysis and Security Valuation (4th ed.). Sydney, Australia: McGraw-
Hill/Irwin.
Renzulli, J. S., Gentry, M., & Reis, S. M. (2004). A time and a place for authentic learning. Educational Leadership,
62(1), 73-77.
Resnick, L. B. (1987). Learning in school and out. Educational Researcher, 16(9), 13-20.
Rogers, E. M. ([1962] 2003). Diffusion of Innovations (5th ed.). New York: Free Press/Simon & Schuster, Inc.
Rourke, L., Andersen, T., Garrison, D. R., & Archer, W. (2001). Assessing social presence in asynchronous, text-based
computer conferencing. Journal of Distance Education, 14(3), 51-70.
Rule, A. C. (2006). Editorial: The components of authentic learning. Journal of Authentic Learning, 3(1), 1019.
Stake, R. E. (2005). Qualitative case studies. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage Handbook of Qualitative
Research (3rd ed., pp. 443-466). Thousand Oaks, CA: Sage Publications.
Stickney, C. P., Brown, P., & Wahlen, J. M. (2007). Financial Reporting, Financial Statement Analysis, and Valuation: A
Strategic Perspective (6th ed.). Australia: Thomson South-Western.
Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of
Evaluation, 27(2), 237-246.
Vygotsky, L. S., Cole, M., John-Steiner, V., & Scribner, S. (1978). Mind in Society: The Development of Higher
Psychological Processes. Cambridge, MA: Harvard University Press.
Wolf, F. A. (1991). Mind into Matter: A New Alchemy of Science and Spirit. Needham, MA: Moment Point Press.
Yin, R. K. (2009). Case Study Research: Design and Methods (4th ed.). Thousand Oaks, California: Sage Inc.
82
STIMULATING LEARNING VIA TUTORING AND

COLLABORATIVE ENTREPRENEURSHIP GAMING
António Manuel Cerqueira Alves1, Anabela Maria de Sousa Pereira2, Hélder Castanheira3,
Inês Direito2 and A.Manuel de Oliveira Duarte1
1
Institute of Telecommunications and University of Aveiro, Department of Electronics, Telecommunications and
Informatics, 3880-193 Aveiro, P
2
Department of Education, University of Aveiro, 3880-193 Aveiro, P
3
Social Action Services, University of Aveiro, 3880-193 Aveiro, P
ABSTRACT
This paper presents results from a multidisciplinary program targeted at engineering education and at the development of
entrepreneurial mind in telecommunications engineering students.
The basic concept is rooted in a capstone-like project with the following characteristics: (i) Creation of student awareness
about real world engineering activities, involving engineering professionals and enterprises in selected classroom
activities; (ii) Simulation of a business environment in capstone project classes; (iii) Market gaming around a set of
business cases where students are organized in teams playing different professional roles (role playing); (iv) Linking the
outcome of market gaming and associated business cases with syllabus topics and with practical issues resorting to
engineering decisions that have to prepared based on technology choices, network design, market simulation and
economic-financial analysis.
The results showed an improvement in average marks, the development of teamwork skills, the consolidation of previous
knowledge and a better understanding of the telecommunications business markets.
KEYWORDS
Tutoring, problem based learning, entrepreneurship.
1. INTRODUCTION
The profound mutations that took place over the last decades in terms of enabling technologies, emerging
business models and organizational structures challenges continuously the telecommunications sector and
demands, more than ever, a competent and flexible workforce. The sector’s companies look for graduated
engineers equipped with the competences required by the new professional environment. However, a well
succeeded recruitment is hard to achieve, as recent graduated engineers are considered not sufficiently
prepared with knowledge and skills to face the uncertainties of the market. They frequently lack an integrated
vision of the telecommunications sector, sound scientific background and ability to cope with technological
and organizational changes. Some fundamental soft skills such as problem-solving, teamwork and
communication have also been referred by companies’ as increasingly important.
Higher Education institutions have responsibilities in preparing future engineers to work competently and
deal with unpredictable challenges. Unfortunately, engineering curricula are still characterized by a strong
emphasis on science and technology disciplines instead of more practical approaches. Science and
technology are essential requirements to prepare students with the analytical skills that an engineer must
have. However this preparation on propaedeutic and specific subject matters of each engineering field
frequently is not accompanied by an effort to prepare students about equally important non-technical aspects
of their profession. These shortcomings are particularly felt in relation to soft skills such as planning,
organization and inter-personal communication. All this is further aggravated when they have to work within
a team. In addition, it is also frequent that during their courses, students develop very little awareness about
the outside world, namely about the markets where soon they will be looking for a job or fighting to keep it.
83
ISBN: 978-989-8533-12-8 © 2012 IADIS
For many engineering graduates, when starting a career, the unsuitability of companies’ demands and
educational programs results in serious behavioral mismatches and very limited knowledge about the activity
sectors and businesses where they become involved These circumstances can represent an important
handicap in their careers and the resulting limitations can significantly impair their capability to play the roles
that enterprises expect from them. In addition, these weaknesses also do not favor the emergence of an
entrepreneurial spirit [1] among young engineers, restricting their ability to contribute to economical and
social growth. Ultimately, all this can jeopardize their employability.
This situation creates new responsibilities on the part of Higher Education institutions. Curricula should
match industry needs, not only in terms of contents but also in terms of pedagogical approach. Learning
processes should be focused on the learner, captivating his interest and promoting active and autonomous
competence development.
2. EXPLORATORY STUDY
Over the last 8 years a study has been conducted encompassing approximately 250 students of engineering
courses (higher education) and approximately 500 students of foundation courses (post-secondary education)
[2]., in other to gain a better understanding about the matching between industry needs and curricula, and
also to prepare future actions. This study addressed the following aspects:
• Student’s representations with respect to the specific subjects of study of their courses.
• Representations of enterprises that received either young graduates or trainees from engineering and
foundation courses.
Among the findings of this study where the following aspects:
• Engineering and technology students receive tools for solving problems that they have never faced
before and for which they do not have an adequate appreciation.
• Because of their limited real world experience, engineering and technology students have difficulty
in understanding the practical applications of their studies.
Another frequent feeling among large amounts of engineering students is that they find that classes
were boring [3]. Previous research [2] found that this is mainly due to the following causes:
• First, because of their limited real world experience, students have difficulty in understanding the
practical applications of their studies;
• Second, as a direct consequence of the traditional universities’ teaching approach: students receive
tools for solving problems that they have never faced before and for which they do not have an
adequate appreciation.
In summary, many students don’t develop meaningful knowledge and competences and this is caused by
the adopted pedagogical approaches that don’t promote active learning.
3. ROLE-PLAYING: PROMOTING ACTIVE AND MEANINGFUL

KNOWLEDGE
In order to provide answers to the problems identified in the study outlined above, a pedagogical initiative
was launched targeted at the promotion of active and meaningful knowledge creation in engineering students.
This initiative is currently taking place in the context of several courses in the area of Electrical Engineering
(with majors in Telecommunications and Information Systems) at post-secondary, B. Sc. and M. Sc. levels
(Bologna system).The basic concept behind it is rooted on a capstone-like project where groups of students
play the role of telecommunication companies competing against each other resorting to decisions that they
have to prepare based on sound engineering studies: technology choices, network design and dimensioning,
market simulation and economic-financial analysis.
With this role-playing approach students are engaged in authentic real-world problems and active
learning, having the opportunity of learning by doing, receiving feedback, continually refining their
understanding and building new knowledge [4]. This initiative is intrinsically dynamic, learner-centred and
more experiential than traditional ones. It represents an attempt to improve student’s classroom involvement,
84
bridging the gap between the engineering profession and the classroom, attempting to contribute towards
better success rates and improved employability. It also helps the development of professional identities.
The initiative followed two phases:
• First, definition of project ideas made with the contributions of practicing engineers from several
companies that are invited to present some of their real-work challenges in a series of seminars.
Students engaged in weekly discussion sessions with practicing engineers and experts (industrial
guest speakers) in order to exchange ideas and discuss career paths. The main objectives of these
sessions were: provision of the “big picture” about core characteristics of what telecommunication
engineers do; exposition to positive role models; encourage questions and understanding.
• Second, projects designed around a situation where teams play the role of competing companies in a
market place. Competition initiatives among teams playing the roles of competing companies in an
open market were delivered, in order to expose students to business dynamics [5].
This leads to an atmosphere of project-based active learning combined with an interactive entrepreneurial
atmosphere in the area of telecommunications engineering. The role-playing competitions followed three
steps:
• Faced with a specific challenge (as will be outlined ahead in the paper) each team tries to identify
possible solutions and must make its evaluation, both in technical and economical terms.
• Chosen solutions must be converted into a business case, with different teams playing the roles of
competing companies in a marketplace.
• A didactic market simulator is used to create condition similar to those found in real markets and to
convey experimental lessons transferable to the real world.
A description of this market simulator is provided next.
4. DIDACTIC MARKET SIMULATOR

Training simulators are designed for education purposes, providing significant hands-on experiences that
motivate and facilitate learning [6]. Additionally they can also offer experiences that resemble those of the
real world and, thus, can give students the opportunity to apply theory in an efficient, economic and
interactive fashion.
The work described in this paper was supported by the usage of a didactic market simulator that can be
used to make students familiar with the dynamics of the telecommunications sector. It can easily be
transposed to other economic sectors. Its structure, in its present state, is depicted in the following figure:
Figure 1. Didactic market simulator structure
85
ISBN: 978-989-8533-12-8 © 2012 IADIS
4.1 Purpose of the Didactic Market Simulator

The simulator is designed so that students will learn how telecommunications engineering decisions (e.g.,
network architecture, physical media, bandwidth, latency) associated with marketing, economic and financial
decisions (e.g., offered services, tariffs, competition among operators, etc) affect the overall network
performance and the ways markets react.
In a preliminary phase Excel was used as the basic platform. This enabled some fine tuning of the
mathematical model and also proved very useful for the determination of several parameters. At a later stage
the implementation was migrated to a web environment supported by database and appropriate query
languages.
The following figures illustrate some screen shots obtained with the market simulator in a web
environment:
Figure 2. Example of screen shot obtained with the market simulator (investment analysis in an access network with 3
operators)
4.2 Test and Validation

In order to test and validate the approach described in this paper a series of experiments was trialed over the
last two years. This was done in the context of a Capstone Project in the 3rd year of an MSc in Electronics
and Telecommunications Engineering (total duration: 5 years ; 3 years 1st cycle; 2 years 2nd cycle).
The basic objective of this Capstone Project is to face students with the challenge of projecting an access
network using up-to-date technologies (e.g., Fiber-to-the-home, LTE, WiMAX, etc) and evaluating the
different architectures (point-to-point, point-to-multipoint, etc), different engineering solutions (active,
passive, etc), roll-out strategies (market size estimates, time plan of investments, tariffs, etc). In this work
students are required to integrate knowledge and skills developed in other disciplines, probably over a period
of more several years.
To estimate (quantitatively) the impact of the approach described in this project on student learning and
understanding, during the last 3 weeks of a semester (over the last 2 academic years) , the class (45 students,
average) was given an assessment test (multiple-choice questions) on Access Networks (a subject not
specifically studied in the 9 preceding weeks, and which requires the integration of knowledge and skills
developed in other disciplines, over a period of approximately two years before the capstone project) and the
market simulator was introduced.
After this test the class had the opportunity to attend a seminar (1 hour) by an invited senior
telecommunications engineer responsible for the access network planning in a major telecom operator. Here
students had the opportunity to witness some of the challenges faced by a telecommunications engineer in
86
planning, designing and operating an access network under severe market competition conditions. At this
point the class was split in 9 groups of 5 students for a short period (1 hour) doing hands-on familiarization
with the market simulator. This was followed by a period of 2 more working sessions (4 hours over 2 weeks)
where the class was organized in sets of 3 groups. In each set each group played the role of a telecom
operator competing with the other.
Tablet lap-tops were made available for these sessions in order to facilitate interaction and discussion of
ideas inside groups and among groups.
In the first of these 2 sessions every group started with equal market share as the other groups. Following
a choice of engineering options related to the specific access network under consideration (architectures,
active or passive network elements, market size estimates, expected competition, time plan of investments,
tariffs, etc,) the simulator produced the market share situation for every competitor, corresponding to half of
the study period under consideration (as illustrated in Erro! A origem da referência não foi encontrada.).
During the period until the following session, in the week after, every group tried to devise possible strategies
to either recover from the bad position where the first run had left them or to keep the advantage that
eventually they had already obtained. The second run dictated the final results of the market game.
After this experience an assessment test on Access Networks as similar as possible to the original one (but
not equal…) was given again to all 45 students in order to measure eventual changes in student learning and
comprehension.
Figure 3 shows the aggregate results of these tests over the period of 2 academic years.
Figure 3. Impact of the capstone project approach on student outcomes
4.3 Assessment of Experience’s Impact on Student’s Learning

The results obtained, in spite of referring to just 2 runs of the experiment over the last 2 years (other will
follow in subsequent years), were very encouraging:
• The classes, as a whole (2009-9, 2009-10) showed an average improvement of 2,06 points (out of
possible 20), that is, approximately 10,3%.
• It was interesting to notice that the improvement was particularly significant in students with
average marks, where the vast majority of engineering students do stand more frequently.
The above results were complemented by a set of (informal) interviews with a sample of 10 students
(out of 45), in both academic years, in order to gain some feedback about how students felt with the
experiment. The outcome of these interviews was generally very positive, underlining particularly
the following aspects:
• The very positive effect of having a practicing engineer sharing with students some of it
professional experience in problems very similar to those that they were facing in the capstone
project (a typical case of “situated learning” [3]).
• Having the possibility to play with the didactic market simulator proved to be extremely useful to
integrate and consolidate previous learning, to help gaining a better understanding of businesses
dynamics and to improve teamwork.
87
ISBN: 978-989-8533-12-8 © 2012 IADIS
4.4 Assessment of Experience’s Impact on Employers

Given the fact that the experiment was done with students attending their last year of the 1st cycle of the
engineering degree (Bologna type) it was possible to track some of these students in their first employment.
This was done with a group of 5 students that graduated in 2008-9. Results and is currently being done with 5
additional students that graduated in 2009-10.
As part of this exercise several interviews were made with responsible personnel of the employing
companies following the first 3 months of employment of the graduates.
In spite of the limited statistical value that this limited number of enquires might have it is very
encouraging to notice that, in general they seem to point out to the following: as compared to their company
colleagues, test graduates exhibit better teamwork skills, show good ability to integrate and associate
knowledge from different fields and reveal good understanding of the telecommunications business markets.
5. CONCLUSIONS AND FUTURE DEVELOPMENTS

The results obtained, in spite of referring to just one single run of the experiment (other will follow in
subsequent years), were very encouraging:
• The classes as a whole (2009-9, 2009-10) showed an average improvement of 2,06 points (out of
possible 20), that is, approximately 10,3%.
• It was interesting to notice that the improvement was particularly significant in students with
average marks, where the vast majority of engineering students do stand more frequently.
• The above results were complemented by a set of (informal) interviews with a sample of 20 students
(out of 90) in order to gain some feedback about how students felt with the experiment. The
outcome of these interviews was generally very positive, stressing in particular the following
aspects:
• The very positive effect of having a practicing engineers sharing with students some of their
professional experience in problems very similar to those that they were facing in the capstone
project (a typical case of “situated learning” [5]).
• Having the possibility to play with a didactic market simulator closely linked to the engineering
variables present in a typical telecommunication project proved to be extremely useful to integrate
and consolidate previous learning, to help gaining a better understanding about businesses dynamics
and to improve teamwork skills.
• As compared to their company colleagues, test graduates exhibit better teamwork skills, show good
ability to integrate and associate knowledge from different fields and reveal good understanding of
the telecommunications business markets.
The implementation of role-playing activities proved to be a fruitful pedagogical technique with the
potential to transform theoretical concepts into experiential outcomes. In this way, educational role-plays
engage students in close to real-world learning, providing students opportunities for learning by doing,
refining their understanding and building new knowledge. Improved employability is also a potential
important result since graduates capability to play the roles that enterprises expect from them is significantly
benefited.
ACKNOWLEDGEMENTS
Some of the work reported in this paper has been supported by an “HP Innovations on Education Grant”.
This support is deeply appreciated.
Thanks are also due to the following entities:
• Institute of Telecommunications for supporting the participation in CELDA of one of the authors.
• Social Action Services, University of Aveiro for supporting the grant of António Alves.
• Fundação para a Ciência e Tecnologia for supporting the grant of Inês Direito.
88
REFERENCES
[1] A.J. Smith, L.A. Collins, P.D. Hannon, (2006) "Embedding new entrepreneurship programmes in UK higher
education institutions: Challenges and considerations", Education + Training, Vol. 48 Iss: 8/9, pp.555 - 567
[2] A.M.O. Duarte, and I. Direito,”Vocational Education and Engineering Enrollment: a case study”, International
Conference on Engineering Education & Research “ICEE/ICEER2009 KOREA”, Seoul, 23-28August 2009.
[3] J. R. Anderson, L. M. Reder, and H. A. Simon, “Situated learning and education”, in Educational Researcher, 25,
1996, pp. 5–11.
[4] J. Bransford, A. Brown, and R. Cocking (Eds.), “How People Learn: Brain, Mind, Experience, and School,
Committee on Developments in the Science of Learning”, Commission on Behavioral and Social Sciences and
Education, National Research Council. National Academy Press, Washington, 2000.
[5] J. Carpio, G. Quejo, R. Guirado, M. Valcárcel, P. Simón, A. Santamaría, N. Acero, M. García-Lorenzo, R. Chácon,
Q. Martín-Moreno, and M. D. Fernández-Pérez, “Educational Application of role-playing and simulation of
professional environments related to the power systems and the electricity market”, in proceedings of Meeting the
Growing Demand for Engineers and Their Educators 2010-2020 International Summit, IEEE, Munich, 2007, Vol. 50
Papers, pp. 1-10.
[6] N. Kartam, an K. Al-Reshaid, “Design and Implementation of Web-based Multimedia Techniques for Construction
Education”, in International Journal of Engineering Education, 18(6), 2002, pp. 682-696.
[7] D. Cooper, and R. Dougherty, Control Station: “An Interactive Simulator for Process Control Education”, in
International Journal of Engineering Education, 17(3), 2001, pp. 276-287.
89
ISBN: 978-989-8533-12-8 © 2012 IADIS
PROMOTING LEARNING SKILLS THROUGH

TEAMWORK ASSESSMENT AND SELF/PEER
EVALUATION IN HIGHER EDUCATION
Tomayess Issa
Curtin University – GPO Box U1987 Perth WA Australia 5845
ABSTRACT
In the education sector, teamwork assessment and self/peer evaluation are widely applied in higher education nationally
and internationally. This assessment is designed to encourage students to promote and improve their skills in teamwork,
communication (writing, interpersonal interaction and cultural awareness, and presenting), critical and creative thinking
(problem solving and decision-making), Information Technology literacy, and information literacy. Beside the above, this
assessment will improve work performance and productivity and increase self-confidence in a range of situations. These
skills are developed by formative and summative feedback, collaboration, and cooperation between students and the
lecturer. This study aims to discuss the development of teamwork assessment and self/peer evaluation in the BT2
undergraduate unit at an Australian university to promote students’ skills and to prepare them for the workplace in the
future, as most of these skills are indispensable not only from the academic perspective but for business as well. This
study provides empirical evidence from 267 students, based on quantitative and qualitative data derived from two
sources. The first is the anonymous informal feedback collected during the semester, while the second (formal) source of
students’ evaluations and perspectives towards the BT2 unit is ‘eVALUate’, the anonymous online system for gathering
and reporting students’ perceptions of their learning experiences at the university. The students’ remarks show their
satisfaction with this assessment, as it develops specific skills for the current study and for the future workplace, i.e. time
management; problem solving, decision-making, cultural awareness, presentation, communication and meeting a
deadline.
KEYWORDS
Teamwork assessment, self/peer evaluation, learning skills, higher education, Australia
1. INTRODUCTION
From the business perspective, it has been noted that Information Systems practitioners have been eagerly
exploring the need for effective skills, i.e. teamwork, critical thinking, writing, communication, and oral and
written presentation, among Information Systems graduates (Lanning et al. 2011; Van Deursen & Van Dijk
2009). Thus, university academics have started to integrate these skills in their curricula and units to assist
students in university life, their future workplace and life in general. To align with businesses’ needs and
desires, a teamwork assessment and self/peer evaluation was developed and integrated in the Business
Technology (BT2) unit to motivate and challenge students to explore and investigate the strengths and
weaknesses of the new technologies in the marketplace and present the outcomes as a report in an academic
style. Furthermore, this assessment will identify how these technologies will assist businesses to improve
their performance, increase productivity, and raise their awareness and variation in the workplace. The
teamwork assessment discussed innovative topics, namely ‘fiber to the home’ (FTTH); security, the Internet,
and the latest operating systems; mobile technologies and Internet telephony; 2015 new laptops; wireless
technology, i.e. WiMax (Worldwide Interoperability for Microwave Access); green technology; cloud
computing; IPv6; and virtualization. Teamwork assessment and self/peer evaluation are becoming an
essential part of BT2 unit, to encourage students to learn, improve and develop communication skills (such as
writing and oral presentation), to develop excellent relationships with their peers, and to exchange ideas and
knowledge in relation to the unit in general and the assignment in particular. The teamwork assessment and
self/peer evaluation were released in week one of the semester, with the necessary information from the
90
marking guide, strategies on the presentation and structure of reports, and guidelines on plagiarism and
referencing style. The self/peer evaluation aims to encourage each student to think about and understand how
well s/he and her/his partner performed the team activity, and to develop their teamwork skills. Each pair
submitted one peer-reflective and one personal (self-reflective) comment on what s/he has done for this
assignment, and a copy of her/his notes on the resources that s/he developed or located in this assignment.
The self/peer reflection questions encompassed the following headings: (1) Individual roles and
responsibilities; (2) Generating and creating ideas/strategies; (3) Research; (4) Collegiality; (5)
Organizational skills; (6) Final outcome/product.
Furthermore, students were asked to complete the self/peer reflection as well as the peer reflection
comments section. The former included three questions or tasks – (1) what did s/he learn from this group
activity (in terms of the process of working in teams – not content/information)? (2) What would s/he need to
change in the way s/he worked with her/his partner in order to make future team activities more successful?
(3) List three strengths that you have in team working and two weaknesses that you need to continue to
improve, while the latter included one question that asked the student to list three strengths and two
weaknesses that s/he had identified in her/his partner from this team-working assignment. Each student
uploaded their evaluation and draft resource material to the blackboard FLECS facility, and later the lecturer
checked and assessed the group’s work and identified how well the pair functioned. Finally, this assessment
provided the students with an opportunity to inform the lecturer if there were any problems with the pair
interactions, the division of workload, or other aspects. On the basis of this information, the lecturer had the
capacity to assign each student a different mark based on his or her efforts to complete the assignment well.
This paper will present an Australian perspective on integrating teamwork assessment and self/peer
evaluation in higher education. This study is organized as follows: (1) teamwork assessment and self/peer
evaluation; (2) methodology and research question(s); (3) BT2 Participations, (4) BT2 Assessments, (5)
Results, (6) Discussion and Lessons Learned, (7) Conclusion.
2. TEAMWORK ASSESSMENT AND SELF/PEER EVALUATION

Teamwork assessment and self/peer evaluation have become widely used in university courses, including
those on computing and information systems (Lejk, Wyvill & Farrow 1996). It has been noted that the
integration of teamwork assessment and self/peer evaluation in the higher education curriculum would
develop skills in communication, leadership, time management, problem solving and decision making
(Gielen, Dochy & Onghena 2011; Oldfield & MacAlpine 1995; Weaver & Esposto 2011). Furthermore, this
assessment will provide more interesting, effective, and exciting learning experiences for students and the
lecturer, as plenty of new ideas and perspective will be generated, and more discussion and debate will occur
(Lanning et al. 2011; Nulty 2011; Weaver & Esposto 2011; Webb 1995). Furthermore, the literature review
confirmed (Diggins 2004; Gillies & Ashman 2003; Lanning et al. 2011; Pavolvich, Collins & Jones 2009;
Steensels et al. 2006) that students who have completed assessments in teamwork mode would be able to
work collaboratively with others in the university and in the workplace in future; the significant concept
behind this assessment is to develop and promote students’ skills for the lifelong learning process.
BT2 is a core unit in the double/single major in the Bachelor of Commerce degree (information systems
and information technology, information technology and digital design, information systems and internet
communications, information systems, and information technology). The BT2 unit focuses on hardware,
software, operating systems, networking, LAN protocols, network services, and UNIX scripting. According
to the university policy (Curtin University n.d.), each unit must promote two to three professional skills: i.e.
communication (writing, interpersonal interaction and cultural awareness, and presenting), critical and
creative thinking (problem solving and decision making), teamwork, IT literacy and information literacy. The
BT2 unit promotes the following professional skills: (1) Critical and creative thinking – problem solving; (2)
Critical and creative thinking – decision-making; and (3) Teamwork. Integrating these skills in BT2 assists
students to (1) identify problems and analyze the main features; (2) identify, implement and evaluate
strategies for the resolution of problems; (3) generate pioneering solutions; (4) accept each other’s ideas
within the group and gather all these ideas to generate a logical argument to solve the problem of the
assignment; (5) manage the time effectively, which involves meeting requirements set by others; (6) manage
projects effectively, which involves the organization and co-ordination of group work; finally, (7) negotiate
91
ISBN: 978-989-8533-12-8 © 2012 IADIS
successfully with others, which involves getting people to move from a position they prefer to a position that
suits all parties. As for the teamwork assessment and self/peer evaluation in BT2, it allows students to
develop the following skills: (1) communication, information literacy, information technology literacy and
teamwork skills, (2) evaluating how teamwork is beneficial in distributing the workload (ensuring individual
accountability), creating participation and involvement, making better decisions and generating a diversity of
ideas. To allow the smooth development of these skills amongst the students, an icebreaker technique was
adopted in the lab session in which the students were asked to introduce themselves to the lecturer and the
other students simultaneously. The icebreaker questions included name, degree, current occupation (if any),
experience in computers, and education in computers; ‘Why did you select this unit?’, and ‘What do you
expect to gain from this unit?’ This technique was very useful in the lab session for allowing students to gain
more information about their peers with respect to studies, knowledge and work experience. In week two, the
students chose their peer for the team working assessment from the same lab cohort, to monitor and check
their progress for the assessment. Furthermore, the procedures for conducting this assessment were explained
in the lab session and in the lecture with respect to the report writing task, plagiarism and referencing. Also,
to emphasize the importance of these procedures, a presenter from the communications skill center in the
university was invited to emphasize the importance of report writing, plagiarism and referencing, as a useful
handout showing different examples with respect to the above procedures was presented to the students.
Students were monitored during the whole process, and if any conflicts occurred between team members in
relation to teamwork assessment, or if students did not appear to be getting along, a contact would be
established with students in order to assist them to finalize and submit the teamwork assessment and the peer-
and self-reflection on time. Furthermore, after the assessment was submitted the self/peer evaluations were
checked and compared to decide if the teamwork contract between the pair was workable. Nevertheless, if a
major difference occurred in the self/peer evaluation, then students would be contacted to clarify the situation
and later each student would receive a different mark for the assessment.
Furthermore, based on students’ formal and informal feedback, slight changes were carried out to the
teamwork assessment and self/peer evaluation. These changes included allocating marks to the self/peer
evaluation to emphasize the value of this assessment, and two approaches were introduced and integrated into
the teamwork assessment and self/peer evaluation: (1) student expectation process; (2) teamwork agreement
and timeline. The former was intended to assist students to choose partners who had similar expectations to
themselves and who were aiming for similar grades for the group assignment, as the icebreaker technique
became unpopular in the lab session. The latter assessment was introduced since in previous semesters many
students had trouble in finalizing the assessment because of the deadline. To prevent this problem, a
teamwork agreement and timeline were signed by the pair and submitted to the lecturer during week three of
the semester. During the lab session, the lecturer tracked the students’ progress on the basis of the teamwork
agreement, as students identified which task activities should be completed in each week. The lecturer
provided formative feedback to tackle any problems immediately and to improve the presentation, structure,
and layout of the assessment. Some outstanding feedback was received from the students regarding the recent
changes, as the majority of the students agreed that these approaches allowed students to manage their time
efficiently and to complete the assessment on time. Finally, more comments from the students indicated that
the teamwork agreement and timeline allow them to: (1) finalize the assessment promptly, and achieve better
planning and preparation, (2) learn new skills and knowledge from the other group members, (3) split the
workload effectively according to the agreement and (4) organize the assessment work and reduce time
pressure.
3. METHODOLOGY AND RESEARCH QUESTION

The mixed methods approach (quantitative and qualitative methods) was used in this study to minimize
discrepancies in the findings and provide a substantial amount of data to identify the positive and negative
aspects of the adoption of teamwork assessment and self/peer evaluation in the BT2 unit (Creswell 2003;
Crump & Logan 2008; Gilbert 2006; Harrison & Reilly 2011; Hesse-Biber 2010; Maudsley 2011;
Molina-Azorin 2011; Teddlie & Tashakkori 2009). Currently most researchers and scholars are in favor of
mixed methods in a single research study, since this mixture allows researchers to have a comprehensive
understanding of the data collected through their research, by a combined analysis of text and statistical data.
92
Creswell (2003) proposes that mixed methods are an approach for collecting, analyzing and integrating both
quantitative and qualitative data in a single study to understand the problems and to identify the strengths and
weaknesses behind the study. By the same token, Wiggins (2011, p.47) claims that “the strengths of each
methodology (flexibility and intimacy for qualitative, generalizability for quantitative) make up for the
other’s weaknesses (low generalizability for qualitative and rigidity and objectivity for quantitative)”. In this
study, the data were collected using quantitative and qualitative methods via anonymous students’ formal and
informal feedback. Informal feedback was collected during the fourth week of the semester. The informal
feedback was designed to give the lecturer some idea of how students felt about the unit and learning
experiences so far in the semester. The informal feedback was analyzed and shared with the students in order
to carry out the changes before the end of the semester. Questions in the informal feedback included: What
are the positive aspects of this unit? What do you like about the lecturer? Do you have any suggestions or
recommendations for how the learning experience can be improved? (Please provide specific suggestions.)
What do you need the lecturer to continue doing, to stop doing, and to start doing? As for formal feedback,
‘eVALUate’ is an anonymous university feedback system that usually runs at the end of the semester.
This feedback mainly focuses on the unit as well as satisfaction with the teaching. The formal feedback
included responses to the following statements: the learning outcomes in this unit are clearly identified; the
learning experience in this unit helps me to achieve the learning outcomes; the learning resources in this unit
help me to achieve the learning outcomes; the assessment tasks in this unit evaluate my achievement of the
learning outcomes. Moreover, the feedback on my work in this unit helps me to achieve the learning
outcomes; the workload in this unit is appropriate to the achievement of the learning outcomes; the quality of
teaching in this unit helps me to achieve the learning outcomes. Further, I am motivated to achieve the
learning outcomes in this unit; I make best use of the learning experience in this unit; I think about how I can
learn more effectively in this unit; and finally, overall, I am satisfied with this unit. In this study the
researcher will examine and investigate if teamwork assessment and self/peer evaluation will assist students
to improve and promote their learning skills (Curtin University n.d.). The learning skills are communication
(writing, interpersonal interaction and cultural awareness, and presenting), critical and creative thinking
(problem solving and decision-making), and Information Technology literacy and information literacy skills,
as they have become crucial for information systems graduates.
4. BT2 PARTICIPANTS
In this study, the participants are 267 students who completed BT2 between 2006 and 2010. Most of these
students are studying towards a double/single major in the Bachelor of Commerce degree (information
systems and information technology, information technology and digital design, information systems and
internet communications, information systems, and information technology). The BT2 focus is on operating
systems and the networking basics, e.g. local area networks. A majority of BT2 students come directly from
high school, and thus has little or no work experience. BT2 students are mainly from Australia and Asia
(China, Hong Kong, Indonesia, Iran, Japan, Malaysia, Pakistan, Singapore, Sri Lanka, Taiwan, Thailand, and
Vietnam); 75% are Asian, while 25% are Australian. It was noted that nationality and culture played an
important role in the BT2 unit via the teamwork assessment, discussion, and debates in the lab session. The
Asian students shared their knowledge in information technology and their computer experience by
uploading the latest technology news and information on the blackboard, while the Australian students shared
their work experience and understanding through discussions and debates. Feedback on BT2 confirmed that
“learning to work with people with English being their second language always establishes a challenge but it
has taught me some skills which I can use later when communicating with people from different countries”,
and from the unit you “gain experience in how to work ‘with people of different cultures and countries’”.
5. BT2 ASSESSMENTS
The assessments for BT2 are divided into three: teamwork assessment and self/peer evaluation (30%),
mid-semester test (30%), and the final exam (40%). These assessments focus on critical and creative
thinking, problem solving and decision-making, and teamwork. The teamwork assessment consists of a set of
93
ISBN: 978-989-8533-12-8 © 2012 IADIS
evaluation exercises (three to four questions) to familiarize students with the newest technology in the
marketplace. These exercises are selected based on the most recent technology in the market to motivate and
challenge students to explore and investigate the strengths and weaknesses of the new technologies in the
marketplace, while the self/peer evaluation is designed to encourage students to think about and understand
how well s/he and her/his partner performed the team activity, in order to develop communications and other
skills. The mid-semester test and final exam consist of short-answer questions and case studies, in order to
assess and examine students’ higher-level thinking skills and knowledge, on the basis of the unit learning
objectives, the practical case study, and the lecturer’s formative and summative feedback during the
semester.
6. RESULTS
Integrating teamwork assessment and self/peer evaluation in the BT2 unit was essential to promote and
improve learning skills among BT2 students. This assessment assisted BT2 students to develop excellent
relationships with their peers, sharing and exchanging ideas and knowledge in relation to the unit in general,
and specifically to the assignment. This assessment has become the favorite for BT2 students, since it was
well monitored and structured by the lecturer. From week one, the lecturer released the assessment
information, namely assessment questions, presentation standard, guidelines for report writing, self/peer
evaluation schedule, teamwork agreement and timeline, and finally the marking guide. In week three, the
teamwork agreement, and timeline were submitted during the lab session to the lecturer. The lecturer and
students must keep a signed copy of the agreement, and the lecturer monitored the agreement periodically.
Integrating this assessment into BT2 prompts and enhances learning skills, develops good relationships
between students, and encourages them to learn other skills besides the teamwork. It was noted from
students’ formal and informal feedback that the teamwork assessment and self/peer evaluation was well liked
among the students. Students confirmed that new knowledge and problem-solving, decision-making and
technology skills were promoted and improved: “The teamwork assessment taught us how we should behave
in a group discussion and how we should work to get the tasks finished quicker but with good quality
results.” “Teamwork is the synergy that really helps in producing great stuff.” “Team work is about using
other members' ideas along with your own to resolve a problem – as the saying goes, ‘two heads are better
than one’”. Additionally, students were very pleased with the teamwork assessment, as they indicated:
“Teamwork assessments are dynamic, and basically teach you how to handle people and their personalities.”
“We made good friends for the future and have learned more about the topic.” Students learned and
developed specific skills from this assessment beside the teamwork skills, and these skills will assist them in
the current study and for the future workplace: i.e. problem solving, decision-making, cultural awareness,
presentation, communication skills, and meeting a deadline. Students stated, “From this group activity we
learn that it is very important to get along with someone. We need to support each other in our work. No
matter from what background they come, everyone should be treated equally, since in real life experience,
different people might have different points of view and it is very important to try to understand each other
and support each other’s decision.” Furthermore, “although team work can be difficult, it helps with learning
patience and being open to other peoples' ideas,” and “I have learned that showing respect and being patient
is very useful in team working. Listen first and then give your opinion and try to show your appreciation for
the effort that your team member has made.”
Furthermore, other skills such as time management and meeting a deadline were promoted and developed
among BT2 students by completion of the teamwork assessment. Students commented: “Doing team work
and team assignment is a good way for me/us to learn how to prepare, how to manage my/our time, and it
develops my/our ability to socialize and meet new people.” Students also confirmed that “working with the
team, especially if s/he is a shy student, our member group will encourage me/us to ask questions during the
meeting.” “It is important to determine pre-arranged deadline schedules. Communication is important; plan
to finish work well before the due date to allow time to fix potential problems.” “We work together not only
in the BT2 unit, but also in other units as a group, to complete other assessments.” BT2 students did not
restrict their feedback to developing and promoting the skills area, but confirmed that completing this
assessment will prepare them for the workplace and for their careers in the future. “The type of information
generated from the assignment will help me throughout my career.” “The Team Working Assessment will
94
help me in participating in teamwork as we are going to face this teamwork in our real life.” Furthermore, to
confirm the teamwork assessment’s success, students must listen to and respect their partner’s opinions, and
students commented that “the Team Working Assessment teaches me how to work in a team and do work
effectively to make a satisfactory outcome”. “Supporting and listening to other group members’ opinions is
an important factor in group success.” From the informal feedback, students indicated that, without the
lecturer’s commitment, encouragement, and formative feedback, the assessment could have failed. Students
stated that the lecturer is “very energetic and full of current information”; “she tries hard to make the unit
interesting; at the beginning of each lecture she always tells us what is happening in the news regarding IT.”
In addition, “I like her for being able to resolve our learning to the practical in a real.” “She explains
materials well and is friendly and approachable.” “She helps everyone who is in difficulties or having
problems with the computer or the unit.” Finally, she “provides good real life situation news, articles and
updates as well”, and, finally, our lecturer is “witty, clear, precise, and humorous”. Furthermore, it was noted
that students were pleased with the BT2 learning experience and feedback, as well as expressing overall
satisfaction (see Table 1). Table 1 indicates that the BT2 average in items 2, 5 and 11 in unit evaluation are
higher than the university average. These outcomes provide the lecturer with more confidence to carry on
with the current assessments, including the teamwork assessment and self/peer evaluation, and to impart
formative and summative feedback to the students aligned to promote and improve their communication
skills.
Table 1. eVALUate results – Unit Evaluations for BT2 from S1 2006 – S2 2010 (93% response rate)
Year/Semester Students Response Item 2 Item 5 Item 11

enrolled rate Learning experience Feedback Overall satisfaction
Unit University Unit University Unit University
Average Average Average
2006/S1 54 19% 100 81 90 70 100 78
2007/S1 32 47% 100 82 93 73 93 82
2007/S2 25 56% 93 82 86 74 86 81
2008/S1 29 59% 100 83 94 76 100 82
2008/S2 26 69% 89 82 83 77 89 83
2009/S1 36 47% 100 84 100 76 100 83
2009/S2 29 62% 89 84 83 77 88 83
2010/S2 16 56% 100 85 100 79 100 84
Beside the quantitative results, students were keen to provide some qualitative responses in relation to the
BT2 unit: “This is a pretty tough unit, but I think we can learn a lot of potential information from this unit for
future use as an IS student.” Furthermore, “it is useful in application to real life”; “the unit is very helpful for
the students who would like to continue their career in the IT area”; it is “very well organized, the material
taught is well structured, organized, and well taught; lots of useful documents, lab notes etc. … given to
students; useful windows functions, facts, features taught”. Students also perceived the BT2 unit as “looking
at the real life situation of the technology world and providing up-to-date information”; it has a “very logical
and paced structure, the work is not over the top and you still are doing things.” The feedback from
eVALUate (formal feedback) was not limited to the unit evaluation; students also presented their views on
the teaching evaluation (see Table 2).
Table 2. eVALUate results –Teaching Evaluations for BT2 from S1 2006 – S2 2010 (25% response rate)
Question: The Lecturer: S1 06 S 1 07 S2 07 S1 08 S2 08 S1 09 S2 09 S2 10

Is well organized 100 100 100 100 100 100 100 100
Communicates clearly 100 100 78 100 77 83 90 100
Is approachable 100 100 100 100 100 100 100 100
Provides useful feedback 100 100 100 91 85 100 90 100
Appears knowledgeable in this subject area 100 100 100 100 100 100 90 100
Is enthusiastic in teaching this unit 100 100 100 91 93 100 100 100
Is an effective teacher 100 100 100 100 85 100 90 100
Students offered the following comments about the lecturer: “The best aspect about the lecturer is that she
is specific about her areas of assessment; this enables students to review those areas, and get back to her if
95
ISBN: 978-989-8533-12-8 © 2012 IADIS
need be.” “She explains the lecture well and offers help when asked, and she is great when confronted to
discuss a topic on an individual basis as she observes the level of the student and is able to adjust her
explanations and examples to suit the student.” “She is very up to date with industry-relevant information
regarding the unit, and is able to explain the concepts being taught in the unit clearly.” “The best lecturer I
have ever seen. Easy to approach, provides very effective information for us, and she never delayed in
response to our queries.” “[Our lecturer] is one of the best lecturers I have ever had! I am sure that all of the
students will feel the same as me, she is the best lecturer at the university, no one is better than her.” The
results from the mixed methods endorsed the study research question, as students confirmed, after completing
the BT2 assessments, that they had learned new skills in the areas of communication (writing, interpersonal
interaction and cultural awareness, and presenting), critical and creative thinking (problem solving and
decision making), Information Technology literacy, and information literacy. Finally, students corroborated
that the new skills will be useful in university life, the future workplace and life in general.
7. DISCUSSION AND LESSONS LEARNED

This experience was interesting and an outstanding achievement from the lecturer's perspective, as the
majority of the students indicated in their informal and formal feedback that completing the BT2 assessments
allowed them to promote and develop their communication skills for university, future workplace, and life in
general. The teamwork assessment and self/peer evaluation was designed on the basis of the literature review
and the regular feedback from students. This assessment was a challenging exercise for the lecturer and the
students simultaneously, as the lecturer provided formative feedback to her students in order to tackle any
problems immediately and improve the presentation, structure, and layout of the assessment, while students
were keen to present a weekly report regarding their assessment progress based on the team agreement.
Additionally, from the lecturer's perspective, regular feedback prevents students from repeating the same
mistakes and improves their learning behaviour and thinking skills, especially concerning report writing,
research, and using Endnote software. On the other hand, summative feedback informs students about their
level of performance when they reach the end of a topic or specific assessment task, or at the conclusion of
the unit. As a result of using both formative and summative feedback, the number of requests for extensions
dropped dramatically, and the extra time and effort that were put into the planning and organization of
cooperative learning tasks were rewarded by an increase in students’ levels of satisfaction with the teaching
and learning experiences. On the basis of the literature review and students’ feedback (Bates 2010 ; Gul &
Boman 2006; Jaques & Salman 2007; Neus 2011; Nulty 2011; Wolcott et al. 2002 ; Yang, Newby & Bill
2008), the author definitely recommends the integration of teamwork assessment and self/peer evaluation and
time agreement in higher education. These assessments will encourage students to work as a team, and to
improve their communication skills, including interpersonal interaction, negotiation, conflict resolution,
listening, and patience with others, and their skills in the areas of leadership, brainstorming, research, writing,
problem solving, and decision-making. Finally, on the basis of the students’ outstanding overall satisfaction,
the author is now considered a teacher-leader in teamwork assessment in the school, and she now works with
her colleagues to support and implement this assessment in the school curriculum strategy to match the
university graduate attributes and to promote and improve students’ learning skills.
8. CONCLUSION
This study has investigated the incorporation of teamwork assessment and self/peer evaluation in the higher
education sector, specifically at the undergraduate degree level. The literature review confirmed the
significance of this assessment in higher education, as students would improve their collaboration,
cooperation, and communication skills, which have become essential for the lifelong learning process and
future workplace. This paper discussed the strategies and techniques that were adopted for the teamwork
assessment and self/peer evaluation in the BT2 unit. This assessment was a challenging exercise for both the
students and the lecturer, as the lecturer always provided formative feedback to her students in order to tackle
any problems immediately and improve the presentation of the assessment. The regular formative feedback
prevents students from repeating the same mistakes and improves their critical thinking, self-confidence and
96
learning behaviour; also, students were keen to complete their tasks on time according to the team agreement.
Furthermore, it was confirmed that students improved their communication skills, including interpersonal
interaction, negotiation, conflict resolution, listening, and patience with others, and their skills in the areas of
leadership, brainstorming, research, writing, problem solving, and decision making. Finally, it is
recommended that this assessment should be implemented in higher education to meet the needs of students,
university, and the workplace, as these skills are essential and critical in higher education in our time. In the
future, the same exercise will be implemented for a postgraduate unit to examine the similarities and
differences between students’ perception of teamwork assessment and self/peer evaluation. Furthermore,
more research will be carried out to discuss the self/peer evaluation template, especially the strengths and
weaknesses of the partner.
ACKNOWLEDGEMENT
Special thanks to Dr. Shelleyann Scott for her support during this study.
REFERENCES
Bates, T 2010 'New Challenges for Universities: Why They Must Change', in U-DEaDS (eds) (ed.), Changing Cultures
in Higher Education, Springer-Verlag Berlin Heidelberg.
Creswell, JW 2003, Research Design Qualitative, Quantitative, and Mixed Methods Approaches, Second edn, SAGE
Publications, USA.
Crump, B & Logan, K 2008, 'A Framework for Mixed Stakeholders and Mixed Methods', The Electronic Journal of
Business Research Methods, vol. 6, no. 1, pp. 21-8.
Curtin University n.d., Graduate Attributes and Professional Skills Program Retrieved 14 May 2012 from
http://www.business.curtin.edu.au/business/current-students/unit-and-course-information/graduate-attributes-and-
professional-skills-program
Diggins, M 2004, Teaching and Learning Communication Skills in Social Work Education Social Care Institute for
Excellence.
Gielen, S, Dochy, F & Onghena, P 2011, 'An Inventory of Peer Assessment Diversity ', Assessment & Evaluation in
Higher Education, vol. 36, no. 2, pp. 137 - 55.
Gilbert, T 2006, 'Mixed Methods and Mixed Methodologies - The Practical, the Technical and the political ', Journal of
Research in Nursing vol. 11, no. 3, pp. 205 - 17.
Gillies, RM & Ashman, AF 2003, Co-operative Learning - The social and intellectual outcomes of learning in groups
RoutledgeFalmer New York
Gul, R & Boman, J 2006, 'Concept Mapping: A strategy for teaching and evaluation in nursing education ', Nurse
Education in Practice vol. 6, pp. 199 - 206
Harrison, R & Reilly, T 2011, 'Mixed Methods designs in Marketing Research ', Qualitative Market Research: An
International Journal vol. 14, no. 1, pp. 7 - 26.
Hesse-Biber, S 2010, 'Emerging Methodologies and Methods Practices in the Field of Mixed Methods Research',
Qualitative Inquiry, vol. 16, no. 6, pp. 415 - 8.
Jaques, D & Salman, G 2007, Learning in Groups, A Handbook for face to face and online environments Routledge New
York
Lanning, SK, Brickhouse, TH, Gunsolley, JC, Ranson, SL & Willett, RM 2011, 'Communication Skills Instruction: An
Analysis of self, peer-group, student instructors and faculty assessment ', Patient Education and Counseling, vol. 83,
no. 145 - 151.
Lejk, M, Wyvill, M & Farrow, S 1996, 'A Survey of Methods of Deriving Individual Grades from Group Assessments ',
Assessment & Evaluation in Higher Education, vol. 21, pp. 267 - 80.
Maudsley, G 2011, 'Mixing it but not mixed-up: Mixed methods research in medical education (a critical narrative
review) ', Medical Teacher vol. 33, pp. 92 - 104.
Molina-Azorin, J 2011, 'The Use and Added Value of Mixed Methods in Management Research', Journal of Mixed
Methods Research, vol. 5, no. 1, pp. 7 - 24.
Neus, JL 2011, 'Peer Assessment Accounting for Student Agreement ', Assessment & Evaluation in Higher Education,
vol. 36, no. 3, pp. 301 - 14.
97
ISBN: 978-989-8533-12-8 © 2012 IADIS
Nulty, DD 2011, 'Peer and Self-Assessment in the First year of University ', Assessment & Evaluation in Higher
Education, vol. 36, no. 5.
Oldfield, KA & MacAlpine, JMK 1995, 'Peer and Self-Assessment at the Tertiary level- An experiential report ',
Assessment & Evaluation in Higher Education, vol. 20, no. 1, pp. 125 - 32.
Pavolvich, K, Collins, E & Jones, G 2009, 'Developing Students' Skills in Reflective Practice: Design and Assessment',
Journal of Management Education, vol. 23, no. 1, pp. 37 - 58.
Steensels, C, Leemans, L, Buelens, H, Laga, E, Lecoutere, A, Laekeman, G & Simoens, S 2006, 'Peer Assessment: a
Valuable tool to differntiate between student contributions to group work?', Pharmacy Education vol. 6, no. 2, pp.
111 - 8.
Teddlie, C & Tashakkori, A 2009, Foundations of Mixed Methods Research - Integrating Quantitative and Qualitative
Approaches in the Social and Behavioral Sciences, SAGE Publisher USA.
Van Deursen, AJAM & Van Dijk, JAGM 2009, 'Using the Internet: Skill related problems in users' online behavior',
Interacting with Computers, vol. 21, no. 5-6, pp. 393-402.
Weaver, D & Esposto, A 2011, 'Peer Assessment as a Method of improving student engagement ', Assessment &
Evaluation in Higher Education, pp. 1 - 12
Webb, NM 1995, 'Group Collaboration in Assessment: Multiple Objectivies, Processes and Outcomes', Educational
Evaluation and Policy Analysis vol. 17, no. 2, pp. 239 - 61.
Wolcott, S, Baril, C, Cunningham, B, Fordham, D & Pierre, K 2002 'Critical thought on critical thinking research ', J. of
Acc. Ed, vol. 20 pp. 85 - 103.
Yang, Y, Newby, T & Bill, R 2008, 'Facilitating interactions through structured web-based bulletin boards: A quasi-
experimental study on promoting learners' critical thinking skills ', Computers and Education vol. 50 pp. 1572 - 85
98
EVALUATION OF EPE VIDEOS IN DIFFERENT PHASES

OF A LEARNING PROCESS
Line Kolås, Robin Munkvold & Hugo Nordseth

University College of Nord-Trøndelag
ABSTRACT
The goal of the paper is to present possible use of EPE videos in different phases of a learning and teaching process. The
paper is based on an evaluation of EPE (easy production educational) videos. The evaluation framework used in this
study, divides the teaching and learning process into four main phases: 1) The preparation phase, 2) The learning
activities, 3) Follow up work and 4) Assessment. The evaluation of 132 EPE videos shows that the videos mainly are
used in the preparation phase and/or as part of the learning activities. Videos used as part of formative or summative
assessment are rare. The paper presents an illustration, which provides an overview of pedagogical use of educational
videos, based on the four learning phases. Our findings show that all pedagogical opportunities concerning EPE videos
still are not extensively used.
KEYWORDS
Educational video, evaluation framework
1. INTRODUCTION
The interest in short videos used for educational purposes, called EPE (easy production educational) videos is
increasing. The definition of an EPE video is a pedagogical video clip (lasting up to 5 minutes) produced by
the use of “low-effort” video equipment like mobile phones or small cameras. EPE videos should have a
pedagogical purpose and be used in a pedagogical context. The goal of the paper is to present possible use of
EPE videos in different phases of a learning and teaching process. The learning and teaching process is
divided into four main phases: 1) The preparation phase, 2) The learning activities, 3) Follow up work and 4)
Assessment. During the preparation phase, one motivates and raises interest in the learning objective(s), and
the students prepare for the next phase. The next phase is traditionally connected to the classroom situation,
where the teacher has prepared for a variety of learning activities, e.g. presentations, demonstrations,
collaborative learning activities etc. During the 3rd phase “Follow up work”, the students continue the work,
e.g. through homework. The 4th phase includes both formative assessment (assessment for learning) and
summative assessment (assessment of learning).
There are a variety of examples on how to use videos in the classroom, e.g. students watching
films/videos, which presents a specific topic (Torgersen, 1998), active viewing of films (Newby et al, 2006),
teacher recording classroom interactions e.g. the Marte Meo Method (Axberg et al, 2006), student- or
teacher-generated videos e.g. digital storytelling (Lambert, 2009; Høiland & Wølner, 2007; Olsen & Wølner,
2003) and “low effort” use of mobile video technologies in the classroom (Kolås et al, 2011). Outside the
classroom, videos can be used for e.g. preparation, using Khan Academy (http://www.khanacademy.org) or
through methodology like “the flipped classroom” (Bennett et al., 2011).
The technology and software necessary to create, edit and present educational videos are widespread.
Both teachers and students usually have access to a mobile phone camera, which can be used for recording
educational videos. 90% of all Norwegian children aged 9-16 have their own mobile phone, and 97% of all
Norwegian children aged 13-16 have their own mobile phone (Norwegian Media Authority, 2010). The
Norwegian Monitor (Hatlevik et al, 2011) found that 75% of 13 years old children, 81% of 15 years old
children and 86% of 18 years old Norwegians use their own computer at home. 99% of 13 years old children,
98% of 15 years old children and everyone at the age of 18 have access to a computer at home. Editing
software e.g. Windows Live movie maker, screencast software e.g. Screencast-o-matic.com or CamStudio
99
ISBN: 978-989-8533-12-8 © 2012 IADIS
and publishing services like “YouTube”, are free and widely accessible. This is a good starting point for the
use of EPE videos in a pedagogical setting, as the technology is already in their pockets or on their
computers, and many of the students and teachers already are familiar to technology.
The background of this paper is connected to two projects; “Nærproduksjon av video” (Developing EPE
videos) 2011-2012, a national project carried out with the support of Norway Opening Universities focusing
on how to produce and use EPE videos in teaching and learning settings; and “CoTech”, an international
NordPlus project on collaboration technologies in education across borders.
The structure of the paper is as follows. In section 2, we describe the methods, which the development of
the evaluation framework is based upon. Then, a theoretical background of “educational videos” is provided.
The next section briefly presents the evaluation framework before the findings of the study are presented in
section 4 through figures and explanations. We then discuss the findings and present an overview of
pedagogical use of videos in the fifth section, before the paper concludes in section 6.
2. METHOD
2.1 The Evaluation Framework

The evaluation framework for EPE videos was developed based on literature review, evaluation of student
tasks and through prototype testing by an expert panel. First, the literature review covered international
literature on educational videos, with focus on categories/types of educational videos. Secondly, teacher
students in three countries (Estonia, Denmark and Norway) as part of the CoTech project, first performed a
task of planning, recording, producing and publishing an educational video, then through a follow-up task
peer reviewed each other’s videos. Teachers from teacher education institutions in the three countries have
assessed both the student videos and the students’ peer reviews. The experiences of performing these tasks
were important, as they visualized the needs of teachers who do not have experience of producing and using
educational videos. Thirdly, an expert group was presented to a prototype of the evaluation framework that
they tested on a variety of educational videos. The expert group consisted of ten persons with a technical
and/or pedagogical background, who have experience with educational videos. The expert group’s feedback
was implemented into a new version of the evaluation framework.
2.2 Evaluation of EPE Videos

The evaluation was performed by 30 teachers/teacher students. They were told to find five educational
videos, with the maximum length of 5 minutes, which they had used or would like to use in their classroom.
During January 2012, the teachers and teacher students evaluated 4-5 educational videos each using the
evaluation framework. 132 EPE videos were evaluated. The evaluation framework was available in digital
form (through an online questionnaire), which was to be filled out, partly by free text, partly by the use of
radio buttons and check boxes. To ensure a common understanding of all parts of the evaluation framework,
the evaluation framework was introduced to all the participants using a “walk through” technique where the
group evaluated the same educational video together. Any questions to the evaluation framework were
discussed in the group to ensure a common understanding of the evaluation framework.
3. THEORY
3.1 Video Vitamins and the Flipped Classroom

Educational videos can change the structure of teaching. Rismark et al (2007) describe how the teacher in a
biology course prepared “video vitamins” (short video productions) with the goal that the students would
show up better prepared in class compared to using traditional reading assignments. The videos were made
available for the students through their learning management system and the students used mobile phones to
watch the videos. The teacher prepared a video of 4-6 minutes together with the university multimedia centre,
100
using “greenscreen” technology. An alternative method of using educational videos for preparation is the
“flipped classroom” methodology (Bennet et al, 2011), which is an “innovative classroom structure that
moves the lecture outside the classroom via technology and moves homework and practice with concepts
inside the classroom via learning activities” (Strayer, 2007).
3.2 Formative and Summative Assessment

Formative assessment means assessment for learning (Harlen, 2006) and is used to improve a student’s
learning process and learning outcome. Examples of formative assessment techniques are peer assessment,
self-assessment, formative tests, visualization of demands, criteria, and progress. Summative assessment
means assessment of learning (Harlen, 2006), and includes tests, exams, portfolios etc.
3.3 Pre-, while-, post- Viewing Activities

According to Lehiste (2012), videos are powerful classroom resources when employed appropriately. In
order to exploit videos fully in the classroom, she suggests integrating pre-viewing, while-viewing and post-
viewing activities. Examples of pre-viewing activities are to predict the content of the video based on the title
or some keywords, to ask true-false statements, to generate questions about the topic (which the video is
supposed to answer), and to brainstorm. With while-viewing activities, it is recommended that students watch
carefully, not to miss important visual clues. The video clip can be played more than once if necessary.
While-viewing activities include work sheets (e.g. with questions that can be answered during the video clip
is showed), tables or schemes that can be filled out (Lehiste, 2012). Torgersen (1998) describes four types of
worksheet activities: Detail-oriented tasks, problem-oriented tasks, describing tasks and visually oriented
tasks. While-viewing activities can also be performed through alternative viewing techniques e.g. silent
viewing (watching a video clip/scene with the sound off), sound only (listening for aural clues to the actions),
freeze frame (to point out parts of the video, to ask questions etc), backwards viewing and jigsaw viewing
(student partners will each know different, but incomplete versions of a story) (Lehiste, 2012). Examples of
post-viewing activities are summarizing the video in own words, answering comprehension questions,
comparing, creating a mind map, discussing, role-playing etc (Lehiste, 2012).
3.4 Pedagogical Methods

Heinich et al. (2002) categorize pedagogical methods into 10 main categories: drill and practice, presentation,
tutorial, gaming, demonstration, discovery, problem solving, simulation, discussion and cooperative learning.
In drill and practice, the learner is led through a series of exercises designed to practice/improve existing
skills. The method assumes that the learner previously has been introduced to the theme. Drill exercises
should include feedback to reinforce correct answers and remediate errors that learners might make (Heinich
et al, 2002). In the presentation method a source tells, dramatizes, or otherwise presents information to
learners. It is a one-way communication controlled by the source (teacher, book, video etc), with no
immediate response from or interaction with learners (Heinich et al, 2002). Using the tutorial method a tutor
(e.g. a person or computer software) presents the content, poses a question, requests a learner’s response,
analyzes the response, supplies feedback and provides practice exercises until the learner demonstrates a
predetermined level (Heinich et al, 2002). Gaming as a teaching method provides a playful environment
where learners follow prescribed rules while working toward a goal. Gaming can be motivating, especially
for tedious and repetitive content (Heinich et al, 2002). Alessi and Trollip (2001) define six game categories:
adventure games, business games, board games, combat games, logical games, and word games.
By demonstration, the learner views a real or lifelike example of the skill or procedure to be learned. The
goal may be for the learner to imitate a physical performance or to adopt attitudes or values exemplified by
someone who serves as a role model (Heinich et al, 2002). The discovery method uses an inductive approach
to learning: it presents problems to be solved through trial and error. The problem solving method confronts
the learner with a problem situated in the real world and the learner develops, explains and defends a solution
on the problem (Heinich et al, 2002). This study considers discovery and problem solving as one pedagogical
method. Learning through discovery and problem solving requires that learners search for information.
Simulations involve learners confronting a scaled-down version of the real-life situation. It allows realistic
101
ISBN: 978-989-8533-12-8 © 2012 IADIS
practice without the expense or risks otherwise involved (Heinich et al, 2002). Different types of simulations
are physical, iterative, procedural, and situational simulations (Alessi & Trollip, 2001).
As a pedagogical method, discussion involves the exchange of ideas and opinions among learners and
instructor (Heinich et al, 2002). Cooperative learning involves learning together and learning from each
other. Learners can collaborate, not only to discuss, but also by producing media (Heinich et al, 2002).
4. FINDINGS
The findings are based on the evaluation framework for EPE videos, which includes 24 questions. Theory
and previous experiences are implemented into the evaluation framework. The evaluation framework focuses
on five main topics: 1) Background information of the educational video. This includes name and duration of
the video, where the video is published, subject and context of the video. 2) Added values of EPE videos.
3) Different types of EPE videos using different technology, genres, technical effects, cognitive types, modes
and learning phases. 4) The pedagogical methods the EPE videos are based upon. 5) The student versus
teacher productions of EPE videos. This paper mainly presents results from topics 3 and 4, focusing on the
pedagogical methods and use of the EPE videos in different teaching phases.
4.1 Teaching/Learning Phases

The teaching/learning process is divided into four main phases: 1) The preparation phase, 2) The learning
activities, 3) Follow up work and 4) Assessment. During the preparation phase, the teacher’s goal is to
motivate and raise interest in the learning objective(s), and the students prepare for the next phase. The
second phase is traditionally connected to the classroom activities, where the teacher has prepared for a
variety of learning activities, e.g. presentations, demonstrations, collaborative learning activities etc. During
the third phase “Follow up work”, the students continue the work, e.g. through homework. The fourth phase
is assessment, and includes both formative and summative assessment. Figure 1 illustrates that most of the
EPE videos were possible to use in the preparation phase and during learning activities, and a smaller number
of videos can be used for assessment.
Figure 2. Number of EPE videos that can be used in

Figure 1. Number of EPE videos possible to use in the four several teaching phases.
different teaching phases.
The evaluation framework allowed multiple values to be filled in when categorizing the use of EPE
videos into different teaching phases. Figure 2 illustrates how only 12 videos fit only one teaching phase. 118
out of 132 videos were described to fit 2-4 different phases and two videos did not fit any of the teaching
phases. This means that many of the videos can be used in e.g. both preparation phase and phase of learning
activities.
102
4.1.1 Preparation Phase

The evaluation framework asked about the intention of the EPE video if the video can be used in the
preparation phase, and the results show that the main application areas in the preparation phase was to use the
video for introduction and/or to motivate the students. Some videos were used to initiate reflections, while a
small number of videos were used for reading support and/or as an artistic perspective to the learning goals.
Figure 3. Application areas of videos used in the preparation phase.

The evaluation framework allowed multiple values to be filled in when evaluating the application area of
the videos use in the preparation phase. 76 out of 132 videos were described to fit more than one of the
categories for the preparation phase, which means that a video can be used for e.g. both motivation and
reading support.
The “Other” category include explanations like “step by step instruction”, “preparation before using a
tool”, “an example”, “visualization”, “introduction of a person” and “show the right technique”.
4.1.2 Learning Activity Phase
The EPE videos used in the learning activity phase were based on different pedagogical methods. The main
pedagogical methods of the EPE videos were presentation and demonstration. Few videos are based on the
following pedagogical methods: drill & practice, exploration, simulation, discussion, collaborative learning
and game-based learning.
Figure 4. Pedagogical methods used in the EPE videos in the learning activity phase.
4.1.3 Follow up Work

The ”follow up work” phase includes e.g. the use of videos for homework. The findings show that half of the
EPE videos are used for repetition (figure 5), while some EPE videos are used to summarize the classroom’s
learning activities or to prepare for exams. 55 out of 132 videos were not possible to use as follow up work.
103
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 5. Application areas of EPE videos used for follow up work.

The evaluation framework allowed multiple values to be filled in when evaluating the intention of the
EPE videos in follow up work. 29 out of 132 videos were described to fit more than one of the descriptions,
which means that a video can be used for e.g. both repetition and preparation for exam.
4.1.4 Assessment Phase
The assessment phase was divided into two categories: formative and summative assessment. 23 of the
videos were found applicable for formative assessment (figure 6). Some of the videos could be used as video
blogs for reflection (6 videos), self-evaluation situations (6 videos) or self-testing purposes (5 videos). Three
of the videos also were adequate for video-based feedback from the teacher. Just two videos were adequate as
pre- and post-video to visualize and measure the learning progress, and one video was possible to use for
peer assessment purpose. Four of the videos had multiple possibilities for formative assessment.
Figure 6. Use of EPE videos for formative assessment Figure 7. Use of EPE videos for summative assessment
The evaluation shows that 33 of the videos were applicable for summative assessment (figure 7). Most
applicable was use of video-based hand-in (11 videos) and use of video as a part of multiple-choice questions
(8 videos). Seven of the evaluated videos could be used as for teacher generated observation and analysis and
five could be used as a video-based journal for documentation. Six of the videos had multiple possibilities for
summative assessment.
5. DISCUSSION
The results presented in this paper are based on an evaluation framework for EPE videos, which is developed
as a tool for researchers to be able to characterize EPE videos for learning situations and a tool for
teachers/teacher students who want to learn about multiple ways of varying the use of EPE videos in their
teaching practice. This evaluation focuses on video resources chosen by the teacher in a process of
104
planning/teaching. It does not include student-generated videos, which also can be valuable learning
resources. The results are based on the evaluation of 132 EPE videos. The number should be increased if the
goal is to specify exact use of the educational videos. Six of the chosen EPE videos were evaluated twice;
these videos are included because different teachers have different perceptions on how to use the videos in a
teaching and learning situation. The findings presented should be considered as an attempt to start the work
of defining the variety of EPE video use. The evaluation indicates that EPE videos tend to be used more in
the preparation phase and the learning activities, than as follow up work and assessment.
One of the reasons why many EPE videos are used for the preparation phase might be the influence of
methodologies like the flipped classroom (Bennet et al, 2011) and video vitamins (Rismark et al, 2007)
together with video web sites like Khan Academy and TED talks. Video is a well-known and useful tool for
web-based tutorials, presentations and demonstrations. These are methods often used in a preparation phase.
Another reason why many EPE videos are used for the preparation phase is that we want students to be
prepared and to make the cognitive awareness start working on a certain topic.
Based on the evaluation and our findings we present an illustration (figure 8), which provide an overview
of opportunities when and how to use EPE videos.
Figure 8. The variety of EPE video use in different phases of the teaching and learning process.
Flipped classroom methodology (Bennett et al, 2011) grows popular and represents the use of videos in
the preparation phase. The flipped classroom methodology focuses mainly on the teacher’s
presentation/introduction of new learning material. There are other pedagogical methods than presentation,
which should be considered when planning to use EPE videos. The flipped classroom methodology mainly
focuses on video use in the preparation phase, but EPE videos should also be considered in other phases of a
learning process, e.g. as follow up work or assessment.
Using EPE videos as part of the learning activities e.g. in the classroom, our findings indicates that EPE
videos usually are based on presentation and demonstration as pedagogical methods. There are a number of
other pedagogical methods that are not often used today. One may explain this by considering video as a
useful tool when it comes to demonstration. When teaching e.g. a skill, the procedures can be described
through text and still images in a book or through audio, as the teacher is explaining. Using a video one may
include both a visual, oral and textual explanation of how to perform the skill. Motion pictures are usually
more powerful than still images when the goal is to demonstrate a skill.
Hopefully, the use of a variety of pedagogical methods will increase in EPE videos in the phase of
learning activities. Teachers should become aware of the possibilities in the use of EPE videos, e.g. within
mathematics, the use of simulation as pedagogical method might work as a tool for improving the cognitive
understanding of a certain topic. Using EPE videos in order to visualize mathematical simulations –will
hopefully make students better prepared for their classes and give them a better foundation to understand
mathematical topics. Research within the field of mathematics shows that it is more efficient to work with the
subjects based on an inquiry-based approach, making the students ask questions and wonder about why
different mathematical formulas are defined the way they are, giving them a greater chance to understand the
issue rather than just making use of a mathematical formula about which they do not understand the logics
(Fuglestad, Goodchild & Jaworski, 2007, Jaworski, 2006).
We also believe that EPE videos can be useful in follow up work. Examples of EPE videos as follow up
work are to make EPE videos summarize students’ assessments, give them an overview of rights and wrongs
and show good examples. This will give students the opportunity to use videos in different phases of their
learning process, and provide visually-oriented students the opportunity to use their advantages for learning.
105
ISBN: 978-989-8533-12-8 © 2012 IADIS
One of the reasons why only a few EPE videos evaluated are used for assessment might be because these
videos are not necessarily widely published, as they mainly are useful only for a few persons. Feedback on
hand-ins in many cases would be considered personal feedback, which is not relevant to publish on the web.
A video blog for formative assessment, however, would probably be published open for anyone.
6. CONCLUSIONS
The paper has based on the results of an evaluation of 132 educational videos, presented possible use of EPE
videos in different phases of a learning and teaching process. Our findings show that all opportunities
concerning EPE videos still are not extensively used. We believe that teachers who in the future want to use
EPE videos in their classes, should be aware of the opportunities of video use in different phases of a
student’s learning process and the variety of different pedagogical methods an EPE video can be based upon.
The paper presents an illustration, which describes an overview of pedagogical use of EPE videos.
REFERENCES
S.M. Alessi and S.R. Trollip, Multimedia for learning – Methods and development, 3rd edition. Allyn & Bacon – A
Pearson Education Company. ISBN 0-205-27691-1, 2001.
Axberg, U. et al., 2006. The Development of a Systemic School-Based Intervention: Marte Meo and Coordination
Meetings, Family Process, Oxford: Sep 2006, Vol 45, Iss. 3, pg. 375.
Bennett, B., Spencer, D., Bergmann, J., Cockrum, T., Musallam, R., Sams, A., Fisch, K. and Overmyer, J. (2011).
The Flipped Class Manifest. The Daily Riff 12/1/2011. http://www.thedailyriff.com/articles/the-flipped-class-manifest-
823.php
Fuglestad, A. B., Goodchild, S., & Jaworski, B. (2007). Utvikling av Inquiry community for å forbedre undervisning og
læring i matematikk: Didaktikere og lærere arbeider sammen. In M.B.Postholm (Ed.), Forsk med! Lærere og forskere
i læringsarbeid (pp. 34-73). Oslo: Damm & Søn A/S.
Harlen, W. (2006). On the Relationship Between Assessment for Formative and Summative Purposes. In Gardner, J. (ed).
Assessment and Learning, pp 103-117. Sage. ISBN 978-1-4129-1050-7.
Heinich, R., Molenda, M., Russell, J.D. and Smaldino, S.E. Instructional media and technologies for learning 7th edition.
(Merrill Prentice Hall. 2002).
Høiland, T. and Wølner, T.A., 2007. Fra digital ferdighet til kompetanse – om didaktikk for arbeid med digitale medier i
skolen. Gyldendal akademisk. ISBN 978-82-05-37656-4.
Jaworski B. (2007). Learning Communities in Mathematics: Research and Development in Mathematics Teaching and
Learning. In C. Bergsten, B. Grevholm, H. S. Måsøval and F Rønning (Eds.), Relating Practice and Research in
Mathematics Education. Proceedings of NORMA 05, Fourth Nordic Conference of Mathematics Education.(pp 71 -
96) Trondheim: Tapir Academic Press
Kolås, L. et al., 2011. “Low-effort” use of mobile technologies and video in the classroom. Ed-Media 2011 Proceedings
Lambert, J. 2009. Digital storytelling: capturing lives, creating community. Berkeley, Calif.: Digital Diner Press. ISBN
978-1-61623-849-0.
Lehiste, P. (2012). Video in the classroom. http://portal.sliderocket.com/AUEOA/Video
Newby, T.J. et al. 2006. Educational technology for teaching and learning, 3rd edition, Pearson Merrill Prentice Hall.
ISBN 0-13-046714-6.
Norwegian Media Authority, 2010. Mobiltelefon. http://www.medietilsynet.no/Documents/Trygg%20bruk/Rapporter/
Barn%20og%20digitale%20medier/100319_Del_3.pdf
Olsen, K.R. & Wølner, T.A. (2003). Storyline for ungdomstrinnet. Universitetsforlaget. ISBN 82-15-00442-3.
Rismark, M. et al. (2007). Using mobile phones to prepare for university lectures: students experiences. The Turkish
Online Journal of Educational Technology, ISSN: 1303-6521, vol 6, issue 4, art. 9.
Strayer, J.F (2007). The effects of the classroom flip on the learning environment: A comparison of learning activity in a
traditional classroom and a flip classroom that used an intelligent tutoring system. The Ohio State University
http://etd.ohiolink.edu/send-pdf.cgi/Strayer%20Jeremy.pdf?osu1189523914
Torgersen, G.E. (1998). Læring med IT – Teori og metode for undervisning med informasjons- og
kommunikasjonsteknologi. Opplysningsfilm. ISBN 82-994777-0-0.
106
FACTORS IMPACTING CORPORATE E-LEARNERS’

LEARNING FLOW, SATISFACTION, AND LEARNING
PERSISTENCE
Young Ju Joo1, Sunyong Joung2, Nam Hee Kim1 and Hyun Min Chung1
1
Ewha Womans University
2
Kookmin University
ABSTRACT
This study aimed to investigate the structural relationships among teaching presence, cognitive presence, usage, learning
flow, satisfaction, and learning persistence in corporate e-learners. The research participants were 462 e-learners
registered for e-lectures through an electronics company in South Korea. First, the sense of teaching presence, sense of
cognitive presence, and usage affect flow. Second, the sense of teaching presence, sense of cognitive presence, usage, and
flow affect satisfaction. Third, the sense of teaching presence, sense of cognitive presence, usage, flow, and satisfaction
affect learning persistence. Fourth, learning flow intermediates the sense of teaching presence, sense of cognitive
presence, usage, and satisfaction. Fifth, satisfaction intermediates the sense of teaching presence, cognitive presence,
usage, flow, and persistence. We confirmed that learning flow significantly intermediated among sense of teaching
presence, sense of cognitive presence, and satisfaction but not between usage and satisfaction. In addition, satisfaction
intermediated among sense of teaching presence, sense of cognitive presence, usage, flow, and learning persistence.
These findings demonstrate the importance of sense of teaching presence, sense of cognitive presence, and usage for e-
learners. We expect that the results will contribute to the formation and improvement of fundamental learning strategies
for successful e-learning.
KEYWORDS
Corporate e-learning, sense of teaching presence, sense of cognitive presence, usage, learning flow, satisfaction
1. INTRODUCTION
In an information society, where knowledge is a core resource, as the importance of human resource
development increases incrementally, many enterprises emphasize employee education in order to improve
their employee capabilities. Moreover, as people acknowledge the benefits of e-learning—namely, that they
can overcome time and space constraints through use of the Internet and other information and
communication technologies—e-learning rapidly has become diffused and generalized. However, regardless
of such quantitative growth, it can be difficult to actively engage online learners. Achieving flow in the e-
learning process is challenging because e-learning differs from traditional education, which is conducted in a
separate and private space. Thus, online learners may not be motivated to continue due to low learning
satisfaction. As advocates have called for investigation of the e-learner’s experience in the e-learning process
with the aim of improving its quality, researchers have sought to understand the sense of presence and its role
in e-learning.
Sense of presence is expressing not what exists in the physical environment, but rather what one
experiences and perceives (Witmer & Singer, 1998). Sense of teaching presence refers to a learner’s
perception regarding a general teaching phenomenon, along with aspects of designing and systemizing
instruction (Arnold & Ducate, 2006). Sense of cognitive presence is defined as the degree of consistent and
confirmed meaning in a learner’s reflection and discourse (Garrison & Anderson, 2007). The current study
did not adopt the notion of sense of social presence, but did adopt senses of teaching presence and cognitive
presence as effective variables for learning outcomes.
First, this study attempted to clarify the cause and effect relationships among sense of teaching presence,
sense of cognitive presence, flow, satisfaction, and persistence, within an integrated view. Despite the
107
ISBN: 978-989-8533-12-8 © 2012 IADIS
importance of sense of teaching presence, sense of cognitive presence, and usefulness for e-learners, the
research investigating their effects on learning outcomes from an integrated view remains insufficient.
Previous studies related to successful e-learning investigated simple correlation or effects between two
variables rather than integrating the variables related to learning persistence. For example, studies have
examined the relationship between teaching presence and satisfaction (Wu & Hiltz, 2004) as well as the
effect of cognitive presence on satisfaction and presence (Kang, 2005), of teaching presence on persistence
(Shin, 2003), of usage on satisfaction and persistence (Roca, Chiu, and Martinez, 2006), of usage on
persistence (No, Lee & Chung, 2008), of flow on satisfaction (Kim, 2005), of cognitive presence on
satisfaction and achievement (Kang, 2005), and of flow on satisfaction and achievement (Kang, 2006). Since
learning outcomes in cyber-environments, such as satisfaction and learning persistence, are complex
phenomena affected by assorted variables(Willging & Johnson, 2004), observers should consider the relevant
variables’ causal relationships with an integrative structural view.
Second, previous studies on the relationships among flow, satisfaction, and persistence, using SEM, have
not addressed important variables in predicting successful e-learning. Joo, Kim, and Park (2009) did not
include the sense of teaching presence in e-learning but did suggest the necessity of investigating it. Kim
(2006) investigated the relationships among usefulness, ease of use, and behavioral and attitudinal flow but
did not include the outcome variables of flow. Chiu, Hsu, Sun, Lin, and Sun (2005) investigated the
relationships among usefulness, satisfaction, and persistence with the intermediation effect of satisfaction but
did not include the teaching and cognitive sense of presence prior to looking at usefulness and outcome
variables. Moreover, the results of previous studies are somewhat contradictory. Thomas’ (2000)
investigation of the effects of students’ social networks on learning persistence in a university setting found
that the relationships among social networks, scholastic integration, social integration, grades, flow in
objectives, flow in educational organization, and learning persistence were not significant. In research
investigating the relationships among satisfaction, confidence, flow, and intention to repurchase on 615 e-
learners in various institutional settings, Lee (2006) showed that flow does not affect the persistence to
purchase.
The purpose of this study is to explain the relationships among the sense of teaching presence, usage, and
learning outcomes, which facilitate learner flow in corporate e-learning, by integrating all the variables in a
single structural model. Based on a review of the previous literature, we established research hypotheses and
hypothetical research models of corporate e-learning. First, the sense of teaching presence, sense of cognitive
presence, and usage affect flow. Second, the sense of teaching presence, sense of cognitive presence, usage,
and flow affect satisfaction. Third, the sense of teaching presence, sense of cognitive presence, usage, flow,
and satisfaction affect learning persistence. Fourth, learning flow intermediates the sense of teaching
presence, sense of cognitive presence, usage, and satisfaction. Fifth, satisfaction intermediates the sense of
teaching presence, cognitive presence, usage, flow, and persistence. Based on a review of the previous
literature, we established research hypotheses and hypothetical research models of corporate e-learning.
These are shown in Figure 1.
First, the sense of teaching presence, sense of cognitive presence, and usage affect flow. Second, the
sense of teaching presence, sense of cognitive presence, usage, and flow affect satisfaction. Third, the sense
of teaching presence, sense of cognitive presence, usage, flow, and satisfaction affect learning persistence.
Fourth, learning flow intermediates the sense of teaching presence, sense of cognitive presence, usage, and
satisfaction. Fifth, satisfaction intermediates the sense of teaching presence, cognitive presence, usage, flow,
and persistence.
Figure 1. Hypothetical Research Model of Corporate e-Learning
108
2. METHOD
2.1 Subjects
We chose to investigate a single enterprise for the consistency of its registration system and learning
management system. We issued a survey to 462 participants to measure sense of teaching presence, cognitive
presence, usage, learning flow, satisfaction, and learning persistence. Male participants numbered 375
(81.2%), and females, 87 (18.8%). The participants’ ages ranged from 24 to 54. Their job statuses were as
follows: 29.5% staff, 21.4% deputy section chiefs; 17.3% section chiefs, 15.1% deputy department heads,
7.4% department heads, and 9.2% other.
2.2 Research Instrument

To measure the sense of presence, we used the validated instrument designed specifically for measuring
sense of presence by Garrison, Cleveland-Innes, and Fung (2004), extracting only the sense of teaching
presence and sense of cognitive presence. The sense of cognitive presence is measured by eight items. For
both variables, the inter-item consistency had a Cronbach’s α of .94. We removed the second item in one
duplicated items out of the 13 items measuring sense of teaching presence. Thus, the final measurement used
12 items for sense of teaching presence and 8 for sense of cognitive presence. The construct reliability of
sense of teaching presence was .99, and the average extracted variance was .99. The construct reliability for
sense of cognitive presence (the reliability of the dormant variable) was .95, and the average extracted
variance was .91. We measured usage by extracting items from Davis’(1989) Technology Acceptance Model
instrument. Usage consisted usefulness and ease of use. Usefulness and ease of use consisted of four items
each. Inter-item consistency had a Cronbach’s α of .87 calculated for usefulness and .86 for ease of use. The
construct reliabilities of usefulness and ease of use were both .99, and the average extracted variance was .97.
We used 9 items from the Flow state scale instrument validated by Martin and Jackson (2008) to measure
learning flow. The inter-item consistency for learning flow items had a Cronbach’s α of .83 for participants in
high school gym class and .84 for those in high school music class. The construct reliability was .99, and the
average extracted variance was .99. To measure this variable, we revised Shin’s (2003) measurement
instrument for corporate settings. The instrument consists of 8 items rated on a 5-point Likert scale. The
inter-item consistency had a Cronbach’s α of .96, the construct reliability in the current study was .99, and
the average extracted variance was .99. The instrument measuring learning persistence consisted of 4 items.
The construct reliability in the current study was .97, and average extracted variance was .99.
2.3 Research Procedure

To collect data, we conducted an online survey for e-learners enrolled in courses at Enterprise A. We selected
Enterprise A because it uses the same registration and management systems, learning service, and evaluation
and grade generation systems for its various sub-branches. We administered the survey the week prior to the
e-learners completing the four-week course. The main instructional methods were lectures given by the
instructor, delivered by video. To investigate the causal relationship between sense of teaching presence and
sense of cognitive presence in corporate e-learning, we established the hypothetical research model shown in
Figure 2 (see Section 1.5) and the statistical model in Figure 1. As seen in the statistical model, we
established each mathematical dormant variable using index variables from the research model.
109
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 2. Statistical Model of Corporate e-Learning

In establishing the model, we used an item parcel method to avoid overweighting on the measurement
model, since there are single-factor measurement variables in exploratory factor analysis results among sense
of teaching presence, sense of cognitive presence, usage, learning flow, satisfaction, and learning persistence.
To determine the prediction method for the statistical model, we examined multivariate normal distributions
of 8 variables of the SEM, using AMOS 6.0. As a result, we were able to satisfy the conditions of skewnesses
and kurtoses for single variables. We predicted the model fitness and parameters using a Maximum
Likelihood Estimation (MLE) procedure, given that the multivariate normal distribution assumption was
satisfied. We evaluated the model fitness through CMIN, TLI, CFI, and RMSEA.
3. RESULTS
The variable means ranged from 3.55 to 4.22, standard deviations from .66 to .73, skewnesses from .12 to .67,
and kurtoses from .10 to .30. This satisfied the basic assumptions of SEM, as the skewnesses of the
measurement variables were less than 3, and their kurtoses were less than 10. Therefore, the variables
satisfied the basic assumptions of a multivariate normal distribution for SEM examination.
Since the initial fitness of the structural model was TLI = .977, CFI = .986, and RMSEA = .068 (.054-
.082), we confirmed that the fitness index of the initial structural model indicated it was a good model.
Accordingly, we examined the direct effects among sense of teaching presence, sense of cognitive presence,
usage, flow, satisfaction, and learning persistence. The results were as follows: sense of teaching presence on
flow was β = .413 (t = 8.200, p < .05); sense of cognitive presence on flow, β = .411 (t = 6.174, p < .05), and
usage on flow, β = .122 (t = 2.027, p < .05). Second, the effects of sense of teaching presence on satisfaction,
β = .109 (t = 2.091, p < .05); cognitive presence on satisfaction, β = .272 (t = 4.054, p < .05); and usage on
satisfaction, β = .144 (t = 2.550, p < .05). Third, the effects of sense of teaching presence on learning
persistence, β =.136 (t = 12.687, p < .05); and cognitive presence on learning persistence, β = .797 (t =
12.687, p < .05). The effect of usage on learning persistence was not statistically significant. The initial
structural model of this study reveals the significant fact that removing the paths between sense of cognitive
presence and learning persistence, between usage and learning persistence, and between learning flow and
learning persistence did not affect the model fitness. Accordingly, we established the simple model seen in
Figure 3, in which we removed the paths mentioned.
110
Figure 3. Revised Research Model of Corporate e-Learning

Because the initial structural model and revised model have hierarchical relationships, we conducted a
chi-square test to determine whether there was a statistically significant difference between the two. The
result showed no difference between the models in their goodness of fit (CMIND = 5.389, p = .145).
Accordingly, although there was no difference in goodness of fit between the models, we selected the revised
model and estimated the goodness of fit and parameters, since the revised model was simpler. <Table 1>
shows the results of the revised structural model’s goodness of fit. By confirming the goodness of fit index,
we confirmed that the model fitness was good (CMIND = 5.389, p = .145).
Table 1. Examination Results of Fitness of Corrected Model (n = 462)
CMIN df TLI CFI RMSEA (90% Confidence Interval)
Corrected Structural Model 126.815 42 .978 .986 .066 (.053~.080)
Initial Structural Model 121.426 39 .977 .986 .068 (.054~.082)
As seen in <Table 2>, the model’s overall fitness index appeared similar to the initial structural model,
but the absolute fitness index value x2 (chi-square) was 5.389 higher. The fitness indices were statistically
significant; all the fitness indices of the revised structural model satisfied the fitness criteria. In addition, the
sense of teaching presence and satisfaction had significant effects on learning persistence. <Table 2>
summarizes the direct model effects.
Table 2. Direct Model Effects
Direct effect Non-standardization Error of C.R. p Standardization
Flow ← Teaching presence .40 .05 8.22 * .41
Cognitive presence .42 .07 6.18 * .41
Usage .14 .07 1.97 * .12
Satisfaction ← Teaching presence .11 .05 2.01 * .11
Cognitive presence .27 .07 3.84 * .26
Usage .19 .07 2.73 * .16
Flow .46 .06 7.14 * .44
Persistence ← Teaching presence .14 .04 3.65 * .16
Satisfaction .75 .04 17.92 * .83
*
p < .05
111
ISBN: 978-989-8533-12-8 © 2012 IADIS
These results of this study show as followings: first, the sense of teaching presence, sense of cognitive
presence, and usage had significant effects on learning flow. Second, sense of teaching presence, sense of
cognitive presence, and usage had significant effects on satisfaction. Third, satisfaction also had significant
effects on learning persistence. Also, learning flow had significant effects on satisfaction, and satisfaction
had significant effects on learning persistence. Fourth, we found that sense of teaching presence, sense of
cognitive presence, and usage had significant effects on satisfaction by intermediating learning flow. Fifth,
sense of teaching presence, sense of cognitive presence, and usage were found to have significant effects on
learning persistence by intermediating satisfaction. This shows the possibility that learning flow has
significant effects on learning persistence by intermediating satisfaction. Accordingly, the Sobel test was
applied to examine the significance of the indirect effects. <Table 3> displays the indirect effect analysis of
the variables affecting learning outcomes.
Table 3. Direct and Indirect Effects of the Modified Model (n = 462)

Relevant variables Total effect Direct effect Indirect effect
Teaching presence .40 .40* -
Flow ← Cognitive presence .42 .42 *
-
*
Usage .14 .14 -
*
Teaching presence .30 .11 .18*
Cognitive presence .46 .27* .19*
Satisfaction ←
Usage .25 .19* .06
*
Flow .46 .46 -
*
Teaching presence .22 .14 .08*
Cognitive presence .20 - .20*
Persistence ← Usage .14 - .14*
Flow .35 - .35*
*
Satisfaction .75 .75 -
*
p < .05
4. DISCUSSION
Based on the research results, we find the following: First, we confirmed that sense of teaching presence,
sense of cognitive presence, and usage have significant effects on learning flow in corporate e-learning. The
significant effects between teaching and cognitive presence on learning flow are consistent with previous
research findings (Barfield, Zeltzer, Sheridan, & Slater, 1995; Wang & Kang, 2005). The significant effect of
usage on learning flow is also consistent with previous research results (Kim & Oh, 2005).
Second, we confirmed that sense of teaching presence, sense of cognitive presence, usage, and flow have
significant effects on satisfaction in corporate e-learning. The significant effects of teaching presence
(Garrison & Cleveland-Innes, 2005; Wu & Hiltz, 2004; Shin, 2003) cognitive presence (Joo, Kim, & Park,
2009; Kim, 2008), usage (Chiu, Hsu, Sun, Lin & Sun, 2005; Roca, Chiu, & Martinez, 2006), and flow on
satisfaction (Shin, 2006) are consistent with previous research results.
Third, we confirmed the significant effects of sense of teaching presence and satisfaction on learning
persistence in corporate e-learning. The significant effect of teaching presence on learning persistence is
consistent with previous research findings (Shin, 2003). The significant effect of satisfaction on learning
persistence is consistent with previous studies that reported if learners are satisfied with overall aspects, such
as the instructor, teaching method, process, and learning environments, they are likely to continue their
learning after completing the course. On the other hand, in this study, sense of cognitive presence, flow, and
usage did not directly affect persistence. This finding is not consistent with previous studies, which reported
112
opposite results (Joo, Kim, & Park, 2009). This suggests the possibility that the sense of cognitive presence
does not directly affect learning persistence. Further, findings on the effect of usage on persistence is not
consistent with previous studies (No, Lee, Chung, 2008). Although previous studies (Chiu, Hsu, Sun, Lin, &
Sun, 2005; Roca, Chiu, & Martinez, 2006) did not observe direct effects of usage on learning persistence,
they reported that usage significantly affects learning persistence by intermediating satisfaction. Then, the
findings regarding the effect of learning flow on learning persistence are also not consistent with previous
studies, which reported opposite results, namely, that flow does affect persistence in sports environments
(Kim & Lee, 2008). These results likely differ due to the environmental distinction: Flow in a sports
environment means that participants are deeply engaged in attitudes or behavioral aspects of the sport rather
than in cognitive processes.
Fourth, satisfaction intermediated among sense of teaching presence, sense of cognitive presence, usage,
learning flow, and learning persistence. These results mean that when learners’ perceived sense of teaching
and cognitive presence improve, they can experience flow. Accordingly, learners’ overall satisfaction
improves, and learning persistence in taking relevant courses increases even after the students complete the
current course. This finding is consistent with the previous research results (Chiu, Hsu, Sun, Lin, & Sun,
2005).
The suggestions and contributions from the current research results are as follows: First, e-learning
designers, therefore, should increase learners’ sense of teaching presence by providing them with
opportunities to ask questions about class content as part of the learning process, in order to confirm what the
learners know and to correct any misunderstandings. Educators should also encourage each learner’s
persistent participation by managing his or her learning process through e-mail, short message services, or a
webpage. Second, to increase the sense of cognitive presence, e-learning designers should structure learning
content appropriately, allow learners themselves to generate new knowledge, and develop systems to help
learners manage their learning resources and time. Third, e-learning designers, therefore, must improve usage
by providing practical cases closely related to the students' work, so learners can feel that the system is very
useful and easy for them to use. Fourth, e-learning designers, therefore, should improve usage by devising
strategies to raise flow in the learning process, improve learning outcomes, avoid distractions for learners,
and consider the learners’ convenience so that they are not hindered in their studies. Fifth, thus, e-learning
designers, should increase learner satisfaction by emphasizing strategies that use the e-learning design and
management elements, learners’ interest in the learning process, and a systematic educational process to
improve the quality of educational content and to avoid inconveniences for learners.
The limitations of the study and our suggestions for further research studies are as follows. First, the
results of this study have a limited generalizability: we used 462 participants, all employees of Enterprise A
in South Korea, who received an employment insurance refund of the course cost. Future studies should
investigate whether different e-learning settings or cyber-universities produce the same results. Second, since
we expect that the perceived degree of a sense of cognitive presence differs for individual learners, further
research is needed to investigate various motivational variables, rather than simply reflect learner
characteristics. Third, we administered the study to reflect corporate e-learning characteristics, after
removing sense of social presence and focusing on sense of teaching presence and sense of cognitive
presence. However, future studies should consider the sense of social presence that learners experience in e-
learning environments, because successful learning occurs through the integration of the three: sense of
teaching presence, sense of cognitive presence, and sense of social presence. Fourth, we removed learning
achievement from the learning outcome variable because learners received the employment insurance refund
from Enterprise A only if they gave more than 70% correct answers. Therefore, we decided that it would be
somewhat difficult to explain learning achievement as a learning outcome. However, since achievement is an
important variable for measuring learning outcomes, further studies should include achievement as a learning
outcome variable. The ultimate purpose of corporate educational training is not a better grade but to improve
employees’ performance through their application of the knowledge and skills obtained in e-learning. We
suggest future research include studies on learning transfer or Return On Investment.
113
ISBN: 978-989-8533-12-8 © 2012 IADIS
REFERENCES
Arnold, N., & Ducate, L. (2006). Future foreign language teachers' social and cognitive collaboration in an online
environment. Language Learning & Technology, 10(1), 42-66.
Barfield, W., Zeltzer, D. M., Sheridan, T., & Slater, M. (1995). Presence and performance within virtual environments. In
W. Barfield & T. Furness (Eds.), Virtual Environments and Advanced Interface Design (pp.473-513). New York,
NY, USA: Oxford University Press, Inc.
Chiu, C. M., Hsu, M. H., Sun, S. Y., Lin, T. C., & Sun, P. C.(2005). Usability, quality, value and e-learning continuance
decisions. Computers & Education, 45, 399-416.
Davis(1989). Perceived usefulness, Perceived ease of use and User Acceptance of Information Technology. MIS
Quarterly, 13(3), 319-340.
Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future
directions. Internet and Higher Education, 10(3), 157-172.
Garrison, D. R., & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not
enough. The American Journal of Distance Education, 19(3), 133-148.
Garrison, D. R., Cleveland-Innes, M., & Fung, T. (2004). Student role adjustment in online communities of inquiry:
Model and instrument validation. Journal of Asynchronous Learning Networks, 8(2), 61-74.
Joo, Y. J., Kim, E. K., & Park, S. Y. (2009). The structural relationship among cognitive presence, flow and learning
outcome in corporate cyber education, The Journal of Educational Information and Media, 15(3), 21-38.
Kang, M. H. (2005). Cybergogy for Engaged Learning. Seoul: Ewha Womens University Academic-industrial
Cooperation Association.
Kim, M. R. (2005). Factors affecting learners' flow and satisfaction in e-learning graduate courses. Journal of Korean
Education, 32(1), 165-201.
Lee, M.M. (2006). Research on factors affecting repurchasing intention in the e-Learning cites. Knowledge Research,
4(1), 30-69.
Matin, A. J., & Jackson, S. A. (2008). Breif approaches to assessing task absorption and enhnaced subjective experience:
Examining 'short' and 'core' flow in diverse performance domains. Motivation and Emotion, 32, 141-157.
No, M. J., Lee, W. B., & Chung, K. S. (2008). Factors affecting the continuous use intention of e-learning site. Journal of
Vocational Education & Training, 11(3), 237-261.
Roca, J. C., Chiu, C. M., & Martinez, F. J. (2008). Understanding e-learning continuance intention: An extension of the
Technology Acceptance Model. Human-Computer Studies, 64, 683-696.
Shin, N. (2003). Transactional presence as critical predictor of success in distance learning. Distance Education, 24(1),
48-58.
Wang, M. J., & Kang, M. (2006). Cybergogy for engaged learning: A framework for creating learner engagement
through information and communication technology. In M. S. Khine (Eds.), Engaged Learning with Emerging
Technologies(pp.225-253): New York: Springer Publishing.
Willging, P. A. & Johnson, S. D. (2004). Factors that influence students’ decision to dropout of online courses. Journal of
Asynchronous Learning Networks, 8(4), 105-118.
Witmer, B. G., & Singer. M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. MIT Press
Journal, 7(3), 225-240.
Wu, D., & Hiltz, S. R. (2004). Predicting learning from asynchronous online discussions. Journal of Asynchronous
Learning Networks, 8(2),139-152.
114
LINGOBEE AND SOCIAL MEDIA: MOBILE LANGUAGE

LEARNERS AS SOCIAL NETWORKERS
Emma Procter-Legg1, Annamaria Cacchione2 and Sobah Abbas Petersen3

1
Study Group, Bellerbys College, Oxford, UK
2
University of Molise, Dept. of Social Science and Linguistic Centre, Italy
3
Norwegian University of Science & Technology, Trondheim, Norway
ABSTRACT
This paper presents language learners as social networkers and describes and discusses the types of users that can be
identified by analysing the content created by them using a situated mobile language learning app, LingoBee, based on
the idea of crowd sourcing. Borrowing ideas from other studies conducted on social network users, we can identify that
language learners use LingoBee as a social network and they behave as social networkers by creating content, acting as
Conversationalists, Critics and displaying other behaviours shown by social networkers. In addition to this, from our user
studies, it can be seen that the language learners are stimulated by the contributions of other users as well as welcoming
competition among users. LingoBee users as social networkers were analysed and discussed based on Luckin's idea of the
Zone of Proximal Assistance and the Zone of Available Assistance (Luckin, 2009).
KEYWORDS
Mobile Language Learning, Crowdsourcing, Social Media, Community of Learners
1. INTRODUCTION
Social Networks and crowd sourcing have recently received a lot of attention in the area of Technology
Enhanced Learning, introducing new concepts along with the ideas of learning as a social activity and
communities of learners (Haythornwaite, 2011). Learners who already engage with social networks such as
Facebook expect similar capabilities in their learning environments, sometimes even expecting the learning
processes to be connected in some way to their social networks. This opens up a whole new dimension to the
evaluation of learning and learning processes, drawing in ideas from Social Network Analysis and other
disciplines to the area of Technology Enhanced Learning. This brings new insights and new means of
understanding learning and the behaviour of individual learners as well as communities, or more
appropriately, networks or crowds of learners (Altshuler, Fire, Aharony, Elovici, & Pentland, 2012).
In this paper, we present some user studies that we have conducted in several different countries, where
the learners were provided with a mobile language learning app, LingoBee, to support their learning process.
LingoBee is based on ideas that have been explored in an earlier project, in a mobile app called Cloudbank
(Pemberton, Winter, & Fallahkair, 2010), which has a wiki-like functionality to crowd source language
content from anonymous users. User studies conducted on Cloudbank showed that users were also social
networkers. They showed that experience of applications such as Facebook played an important role in
learners’ expectation; e.g. some users expressed their wish to have an identity and desired functionalities to
support multiple contributions, comments and ratings to "bubble up" good content (Pemberton & Winter,
2011). This experience highlights the role that social media plays in our lives and learning processes and how
users' expectations are being affected by it. Thus, LingoBee included functionality to support social
networking among users such as a user profile and identity as well as wiki-like contents and rating of
content.
The aim for this paper is to identify whether LingoBee users act as a true social network or some kind of
hybrid and to identify and describe the types of LingoBee users and the implications this has on how best to
support teaching and learning, to positively impact on learners’ engagement and use of LingoBee.
115
ISBN: 978-989-8533-12-8 © 2012 IADIS
The rest of this paper is organized as follows: Section 2 provides an outline of the functionality of
LingoBee; Section 3 outlines the general study design for the three user studies; Section 4 describes the data
collection methods and the types of social networkers that are relevant for LingoBee; Section 5 presents
evidence of users as social networkers; Section 6 presents and discusses implications for learning; Section 7
summarises the paper.
2. LINGOBEE MOBILE APP
Figure 1. LingoBee functionality. a) LingoBee repository; b) definition of an entry; c) editor to enter a definition d) user
profile; e) browsing: search, filter, sort; c) a flagged definition
LingoBee is a mobile app to support situated mobile language learning and to help the learners in linguistic
and cultural diversity. Based on the ideas of situated learning (Lave & Wenger, 1991) and contextualized
learning, e.g. (Luckin, 2010), it is designed to capture language elements that learners come across in their
everyday lives, whenever and wherever. Ideas of crowd sourcing and social networking are used to collect,
116
share and annotate the contributions of all learners in a shared online repository as shown in Figure 1a).
Users are able to add entries, which may be words or phrases, to the LingoBee repository, which can be
accessed by other users of LingoBee, see Figure 1b), which shows a definition containing a picture, and
Figure 1c) where the user can enter new definitions. Learners are also able to add new definitions to existing
entries and rate existing definitions; e.g. in Figure 1b) the entry "lenticchie" has one definition and users have
rated it as five stars. Each entry can contain multimedia elements such as a picture and/or audio content as
well as web links. In addition, a text-to-speech functionality is available for the correct pronunciation of the
entries. Language articles are co-constructed through the use of LingoBee, where students add items and
build meaning together through dialogue created via multiple entries.
To support social networking, LingoBee provides the capability for the users to define their profiles such
as the username and contact details as shown in Figure 1d). Users are able to view or browse through the
contents of the LingoBee repository and the functionalities to support browsing are shown in Figure 1e).
Other ideas from social networks have been included such as peer rating as shown in Figure 1b) and flagging
an entry as a form of feedback as shown in Figure 1f).
LingoBee was developed as part of an EU LLP project SIMOLA, with partners from six different
countries, (SIMOLA, 2012b). Thus, LingoBee is supported in 6 different European languages and Japanese.
It is a mobile app developed for the Android platform.
3. USER STUDIES
User studies of LingoBee have been conducted in several locations (Italy, UK, Norway, Lithuania, Hungary
and Japan) since July 2011. The user studies presented in this paper were carried out in three different
European educational establishments: Bellerbys College, Oxford, part of Study Group UK (Study Group),
The University of Molise (Unimol), Italy, the Department of Social Science and Linguistic Centre and
Department of Languages and Communication at the Norwegian University of Science and Technology
(NTNU). The users were all language learners, enrolled on university or pre-university courses.
The study design involved introducing LingoBee to international students learning English, Italian and
Norwegian, and providing them with a free smartphone for use in their daily lives. LingoBee was formally
introduced to the students in a classroom setting, although it did not always form any part of the classroom
activities. In general, the students were provided smartphones with the LingoBee app pre-installed. LingoBee
was introduced in different ways: by presenting the basic functionalities for adding content to LingoBee,
through demonstrating the functionality through a video, through the help guides uploaded onto a VLE and
through activities designed to encourage students to explore the functions of the app.
Considering the users as motivated, independent learners, which is one of the findings of the studies
conducted by (Pemberton & Winter, 2011), one of the aims when introducing LingoBee was to show the
students the basic functionality without influencing them with our views of the app and its usage. Another of
the aims of evaluation was to see how the users perceived the systems and used the functionality.
4. METHOD
The results presented in this paper are based on the content in the LingoBee repository and the pre- and post-
intervention questionnaires and interviews conducted with the participants in the studies at Study Group,
Unimol and NTNU. In addition to these, data logs created by the LingoBee system were used to determine
some of the activities of the users that were not visible from the LingoBee mobile app interface. Google
Analytics were also used to detect activities of users. However, for the results that are discussed in this paper,
we have not analysed the data from Google Analytics in detail. Our focus so far has been on the content
created by the users in the repository.
In this paper, the main analysis has been to identify LingoBee users as social networkers. Based on the
motivations for the design of LingoBee and the work conducted in analyzing the types of Social Networking
users such as The Social Technographic Ladder by the Forrester Research Inc. ("The Social Technographics
Ladder," 2011-2012), we have identified the following types of users as relevant for LingoBee and
potentially other social networking based language learning apps:
117
ISBN: 978-989-8533-12-8 © 2012 IADIS
 Creators: users that create entries in the LingoBee repository by adding new words, phrases, additional
definitions and multimedia content.
 Conversationalists: users that add entries onto other users' definitions.
 Critics: users that provide peer reviews by using the rating and flagging functionality on content in the
LingoBee repository.
 Collectors: users that add other users' definitions from the LingoBee repository to their favourites list. In
LingoBee, an individual user's wordlist or favourites are on their mobile device. These are entries
created by them and the entries that they have downloaded from the repository to their favourites list.
 Spectators: users that viewed content in the LingoBee repository.
 Inactives: users that were none of the above or users who were active users at the beginning and then
stopped.
5. LINGOBEE USERS AS SOCIAL NETWORKERS

Based on the results of the studies at Study Group, Unimol and NTNU, we can categorise LingoBee users as
social networkers, using the categories described in Section 4. We will illustrate the different types of users,
using examples sourced from the LingoBee repository, the post-intervention questionnaires and interviews.
Figure 2. Definitions of “mint”: a conversation between a learner and the teacher

The obvious category of user is the Creator as evidenced by the LingoBee repository, which is accessible
from (SIMOLA, 2012a) and examples of which are presented in Figure 1a), b) and f). It is interesting to note
that Creators were of two main types: i) true creators that created entries autonomously and adopted
118
LingoBee as a natural part of their language learning, adding an entry when they saw something new and ii)
those that required prompting to add new content. The analysis of our results concluded that the initial
studies at all three locations experienced the second type of Creators. Thus, prompts were used to stimulate
the creation of more content, such as tasks set by the teacher, e.g. scavenger hunts, or a Facebook group to
support each other.
In the same way, there seem to be two types of Conversationalists: i) true conversationalists that interact
and exchange content taking into consideration what others previously said, just like in any other real life
conversation, and ii) those who simply add entries in a row, without being influenced by others’ content. The
example in Figure 2 shows a "Conversation" where users have created collective definitions in a wiki-style.
Such users are Conversationalists, where one user's definition of an entry is complemented by another user’s
definition. In this particular example, a conversation between the teacher and a student is shown. This is also
a prime example of how a conversation between the learner and a native speaker could take place through
LingoBee, where LingoBee users include native speakers as well as learners. There are several other
examples where one user's entries have additional definitions. For example, in the Unimol user group the
following entries were made:
3 different entries for SEDIA (chair):
04/04/2012,B,sedia,
22/04/2012,F,sedia,
03/04/2012,Si,Se, sedia
4 different entries for POLLO (chicken)
19/11/2011,E,pollo al curry,
29/11/2011,P,Pollo,
01/12/2011,P,Pollo,
19/04/2012,S,Pollo,
These are not true conversations; they are multiple attempts at one single definition by different users in
the Unimol trials. It is interesting to see such examples. Here, the users are in fact Creators of new entries.
However, it appears that they are unaware of existing entries, thus creating a new entry rather than adding to
existing definitions, i.e. a conversation. Had the entry "Pollo" existed before the user wished to add the entry
"pollo al curry", would it have been a conversation? These are the types of questions that we focused on
answering through our interviews. With regards to this point, we can observe that this is due to a feature of
LingoBee, it detects automatically identical entries and links them together, but it does not inform the user
that there is already an entry in the repository. If it did warn the user, they may be less inclined to make a
separate entry.
Examples of Critics can be seen in Figures 1, 2 and 3; users have rated entries in the LingoBee repository,
e.g., in Figure 1b) and Figure 2a), the entries have a rating of five stars. Similarly, users have flagged content;
e.g. in Figure 1f), the entry is flagged as it is spelt incorrectly. Flagging could indicate several things such as
incorrect spellings, inappropriate or abusive content. Both learners and teachers acted as Critics; e.g. the
entry in Figure 1f) was flagged by the teacher to draw the learners' attention to the incorrect spelling.
Collectors are not obvious from the LingoBee interface. However, we asked in the post-intervention
questionnaire ‘Is it helpful to see the words and phrases added to LingoBee by other users’ rated on a Likert
scale of 1-7, where 7 indicates the highest and 1 indicates the lowest level of agreement. The combined
results, across all three studies showed that 68% of all respondents agreed and that 42% strongly agreed
(rated 7) with this statement. This, combined with the logs created by the system, identified entries from the
repository that have been added to other users' favourites lists. Examples of Collectors are shown in
Figure 3; both the entries have only one definition, but you can see from c) from the data collected by
LingoBee, that they have been added to favourites 6 and 3 times respectively. It is not possible to detect
Spectators from the LingoBee interface. However, based on the data from the Google Analytics (which we
are analysing at the time of writing), the post-intervention questionnaires and interviews, it is clear that all
three studies had users who were Spectators that browsed the LingoBee repository. The Google Analytics
data will also help to find out how many times an entry has been viewed but not added to favourites. The
post-intervention questionnaire (which we are currently analysing) had questions such as the number of hours
a learner used LingoBee during the day and the level of LingoBee usage in different locations such as home,
city, etc., on a Likert scale of 1-7. The data from the questionnaires shows that some students reported a
higher number of hours of using LingoBee per day compared to the level of activity shown on the logs in
terms of them as Creators, Conversationalists, Critics and Collectors. We believe this is an indication that
119
ISBN: 978-989-8533-12-8 © 2012 IADIS
some users are Spectators. In addition to this, some of the learners that have been interviewed reported
face-to-face discussions related to LingoBee content with other LingoBee users in their language class.
Figure 3. a) Definitions of Train spotter b) Definition of Crème de la crème c) Extracts from the logs showing number of
times these entries have been added to a users favorites list
Some users were Inactives as they joined the study and accepted a smartphone with LingoBee, but either
never used LingoBee or stopped using it during the course of the user study, as they stopped attending the
language classes for various reasons.
We are currently conducting an in-depth analysis of the data and therefore we have not supported our
observations reported in this section with statistics. In addition to the types of users discussed so far, the
response to the post-intervention question "What additional functionalities would you like in LingoBee to
support language learning?" provided a few interesting responses such as "widget that shows best
definition/information ranked, so people can compete (against) each other, to be ranked on the table….", "It
could be more interactive like a network. Maybe using questions about the name of some objects and doing a
competition between the users with a game" and "motivation". These responses from the learners illuminate
two key points, the first that students like the idea of using LingoBee in a similar way to which they use other
social media, e.g. number of likes on Facebook or Instagram or playing competitive games via Facebook and
they could possibly be motivated by this. If students tend to think it could be more “like a network”, maybe
LingoBee is not a real social network, at least not in the sense they would like. It currently has basic social
network functionalities, but it could allow more interaction and be more integrated with other networks – e.g.
allowing automatic integration with Facebook. Furthermore, within the growing family of social media,
LingoBee appears to resemble interest-focused networks like Instagram or Pinterest, where you do not
necessarily search for friends but for people sharing the same interest.
The second point is that a competitive spirit is indicated as a means of motivation and recognition among
peers. This identifies language learners as a new type of social networker: Competitors and Motivators.
120
6. IMPLICATIONS FOR LEARNING

In this section, we analyse the types of LingoBee users from the perspectives of the work produced in the
area of Technology Enhanced Learning, in particular, learning contexts. An interesting approach is
developed by Luckin, whose ideas are based on the Zone of Proximal Development proposed by (Vygotsy,
1978). Luckin added two additional concepts by defining a Zone of Proximal Adjustment and a Zone of
Available Assistance, (Luckin, 2009). The Zone of Available Assistance describes a variety of assistance that
can be made available to a learner, whilst the Zone of Proximal Adjustment represents a selection from the
Zone of Available Assistance for a given learner and the educational situation.
Figure 4. The Zone of Collaboration around LingoBee users adapted from (Luckin 2010)
LingoBee users as social networkers or as a community of learners can be considered in the context of the
ideas proposed by Luckin; as shown in Figure 4. The tools available to users within the Zone of Available
Assistance are: other LingoBee users; native speakers; the teacher as well as a myriad of tools and
technologies, such as online dictionaries providing potential assistance to the learners. The key to engaging
learners in the user studies, was ensuring LingoBee and other LingoBee users moved from the Zone of
Available Assistance into the Zone of Proximal Adjustment. The first trial group at Study Group struggled to
have any tools in their Zone of Proximal Adjustment, they had never used mobile phones in formal learning,
they weren’t able to use the VLE and they were too new to the college to understand the importance of being
able to use these. Their main tools or available assistance was their teacher, email and text messages and the
LingoBee app (only when supported to use it). In the second Study Group user study, through class
discussions, it became apparent that the entries by one particular user was liked by the other participants of
the study and stimulated them to learn the language and use LingoBee, thus this user became a More Able
Partner (Luckin, 2010) or at least another tool in their Zone of Proximal Adjustment. Similarly, some of the
entries by users in the NTNU user study were rated by the teacher, encouraging and promoting those users
and similar entries in the LingoBee repository. Similarly, the example in Figure 2 shows how the teacher can
provide support to a learner. What is important is that the network of learners and through ideas of social
networking, LingoBee users could provide implicit support to one another in their learning process. Thus, the
LingoBee users can play a significant role in both the Zone of Proximal Adjustment and the Zone of
Proximal Development as a More Able Partner.
The Luckin diagram works as a powerful tool for teachers and researchers. It helps to assess the
preliminary situation in order to prepare the setting to introduce the app. It also highlights the constraints that
could have a negative impact on the learning process. In any case, the experiences of our studies show that
current users’ expectations of social networking cannot be ignored as they are part of their mind-set and
orientate them in creating, selecting and sharing learning objects.
To enable LingoBee users to provide relevant support to one another, there is a need "to have a big group
(using LingoBee)", as stated by one of the users in their post-intervention questionnaire. Our earlier user
studies indicated that the learners required assistance in starting to use LingoBee, thus requiring scaffolding
initially by the teacher in various ways. For example, in addition to the teacher being a Conversationalist to
121
ISBN: 978-989-8533-12-8 © 2012 IADIS
prompt the users, Study Group introduced various activities to support learning, e.g. a scavenger hunt,
walking tour, show and tell activities (often around food), access to a VLE and tasks to engage with the target
culture such as watching specific television programmes. Similarly, the second study conducted at Unimol
introduced a Facebook support group to motivate and encourage the users. Our studies suggest, there is a
need for scaffolding when using LingoBee and when engaging users with the other tools in their Zone of
Proximal Assistance (Wood, Bruner, & Ross, 1976), (Vygotsy, 1978) and (Luckin, 2010).
7. SUMMARY
This paper presents LingoBee users as social networkers and describes and discusses the types of users that
can be identified by analysing the content created in the LingoBee repository and the data that is available.
Borrowing ideas from other studies conducted on social network users, we can identify that LingoBee
language learners use LingoBee as a social network. As social networkers they are: Creators of content;
Conversationalists, Critics of other users' entries; Collectors that download entries created by other users; as
Spectators that browse the content as well as Inactives. In addition to this, from the post-intervention
questionnaires and interviews, it can be seen that the language learners are stimulated by the contributions of
other users and welcome competition. LingoBee users as social networkers were analysed and discussed
based on Luckin's idea of the Zone of Proximal Assistance and the Zone of Available Assistance.
The next stage of our work is to analyse the data that has been gathered in more detail and to provide
statistical evidence. In addition, we will analyse the data for a better understanding of language learners as
social networkers and the role of crowdsourcing and social networking in language learning to provide better
support to language learners.
ACKNOWLEDGEMENT
This work has been conducted within the European Union's Lifelong Learning Programme, SIMOLA,
Project number LLP 511776-LLP-1-2010-1-UK-KA3-KA3. We would also like to thank Ole-Torfinn Fagerli
and Sissel Nefzoui for providing access to their students for the NTNU user studies.
REFERENCES
Altshuler, Yanis, Fire, Michael, Aharony, Nadav, Elovici, Yuval, & Pentland, Alex. (2012). How many makes a crowd?
On the Evolution of Learning as a Factor of Community Coverage. Paper presented at the Int. Conference on Social
Computing, Behavioural and Cultural Modelling, and Prediction.
Haythornwaite, Caroline. (2011). Learning Networks, Crowds and Communities. Paper presented at the Learning and
Analytics Conference LAK'11, Banff, Canada.
Lave, Jean, & Wenger, Etienne. (1991). Situated Learning: Legitimate Peripheral Participation: Cambridge University
Press.
Luckin, Rosemary. (2009). Learning, Context and the Role of Technology. London: Institute of Education, University of
London.
Luckin, Rosemary. (2010). Re-designing Learning Contexts. London and New York: Routledge Taylor & Francis Group.
Pemberton, Lyn, & Winter, Marcus. (2011). Learning from Formative Evaluation in Use: a Case Study of a Mobile
Application for Language Learners. Paper presented at the MLearn 2011, Beijing, China.
Pemberton, Lyn, Winter, Marcus, & Fallahkair, Sanaz. (2010). Collaborative Mobile Knowledge Sharing for Language
Learners. Journal of the Research Centre for Educational Technology, 6(1).
SIMOLA. (2012a). LingoBee Repository Retrieved June 2012, from
http://itrg.brighton.ac.uk/simola.org/lingobee/index.php
SIMOLA. (2012b). Situated Mobile Language Learning, June 2012, from http://itrg.brighton.ac.uk/simola.org/
The Social Technographics Ladder. (2011-2012) Retrieved June 2012, from http://empowered.forrester.com/ladder2010/
Vygotsy, L. S. (1978). Mind in Society. The Development of Higher Psychological Processes: Harvard University Press.
Wood, D. J., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychiatry and
Psychology, 17(2), 89-100.
122
INVESTIGATING THE EFFECT OF USERS’ TAGGING

MOTIVATION ON THE DIGITAL EDUCATIONAL
RESOURCES METADATA DESCRIPTIONS
Panagiotis Zervas1, Demetrios G. Sampson1 and Maria Aristeidou2

1
Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH) and Department of
Digital Systems, University of Piraeus, 150, Androutsou St., 185 32 Piraeus, Greece
2
Department of Digital Systems, University of Piraeus, 150, Androutsou St., 185 32 Piraeus, Greece
ABSTRACT
Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide aiming to
create, share and reuse digital educational resources among educational communities. As a result, organizing, offering
and accessing these resources over the web have been key issues for both the research and the educational community.
Traditionally, a popular way for characterizing digital educational resources is by using a formal and centrally agreed
classification system, such as the IEEE Learning Object Metadata (LOM). On the other hand, with the emerging Web 2.0
applications, the issue of characterizing digital educational resources tends to move from the expert-based description
based on formal classification systems to a less formal user-based tagging referred to as social tagging. As a result, a
number of studies have been reported in the field of Technology-enhanced Learning (TeL) aiming to evaluate whether
social tagging can improve the discovery of new digital educational resources and the retrieval of known ones stored in
web-based repositories. Additionally, recent studies in the field of social tagging systems has provided initial evidence
that users’ tagging motivation has a direct influence on the properties of the resulted tags and folksonomies. Thus, in this
paper we aim to propose a methodology for investigating this issue and examine whether different end-users’ tagging
motivations (that is teachers) could enlarge the metadata descriptions of digital educational resources. The results of our
study provided us evidence that there is a direct influence of users’ tagging motivation on the enlargement of metadata
descriptions of digital educational resources, as well as to the resulted folksonomy when it is compared with formal
structured vocabularies used by metadata experts or content providers for characterizing digital educational resources.
KEYWORDS
Educational resources, social tagging, folksonomy, user tagging motivation, evaluation methodology, evaluation results
1. INTRODUCTION
Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide
towards the provision of open access to digital educational resources, in the form of Learning Objects (LOs)
(McGreal, 2008). UNESCO (2002) has defined Open Educational Resources (OERs) as the “technology-
enabled, open provision of educational resources for consultation, use and adaptation by a community of
users for non-commercial purposes”. The main objective of OERs initiatives is to support the process of
organizing, classifying and storing digital educational resources and their associated metadata in web-based
repositories which are referred to as Learning Object Repositories (LORs) (McGreal, 2004). Within this
context, a popular way for describing digital educational resources is by using a formal and centrally agreed
classification system, such as the IEEE Learning Object Metadata (LOM) (IEEE LTSC, 2005).
On the other hand, the emerging Web 2.0 applications have led to an enormous increase of the digital
resources available on the web today. As a result, both the discovery of new resources and the retrieval of
known ones on the web, become an increasingly complex problem (Heymann, Koutrika and Molina, 2008;
Yande, Jatowt, Nakamura and Tanaka, 2007). Therefore, the issue of characterizing digital resources tends to
move from the expert-based description based on formal classification systems to a less formal user-based
tagging (that is, adding keywords to digital resources) (Derntl, Hampel, Motschnig-Pitrik and Pitner, 2011;
Bi, Shang and Kao, 2009). Adding keywords, also known as tags, to any type of digital resource by users
123
ISBN: 978-989-8533-12-8 © 2012 IADIS
(rather than resources’ authors) is referred to as Social Tagging (Bonino, 2009). The term of social tagging
has emerged for those applications that encourage groups of individuals to openly share their private
descriptions (or tags) of digital resources with other users, either by using a collection of tags created by the
individuals for their personal use (referred to as folksonomy) (Anderson, 2007).
In the field of Technology Enhanced Learning (TeL), social tagging has also been proposed as a mean for
describing digital educational resources (Cho, Yeh, Cheng and Chang, 2011; Bateman, Brooks, McCalla and
Brusilovsky, 2007). As a result, the generation of metadata is now done by individual users, who might
primarily see private benefits, like an easy way to search and retrieve from LORs already used and known
resources using meaningful terms (Dahl and Vossen, 2008). Thus, a number of studies have been reported
aiming to evaluate the added value of socially tagging digital educational resources stored in LORs and
compare it with the traditional approach of expert-based formal description based on centrally agreed
classification systems, such as IEEE LOM (Trant, 2009b; Vuorikari and Ayre, 2009). Additionally, recent
studies in the field of social tagging systems suggests that users’ tagging motivation has a direct influence on
the properties of resulting tags and folksonomies (Korner, et al., 2010; Korner, 2009) but there are not
existing studies for investigating this issue in the field of TeL.
To this end, in this paper we aim to investigate this issue and we propose a methodology that aims to
evaluate whether users’ tagging behaviour can influence (a) the enlargement of metadata descriptions of
digital educational resources and (b) the resulted folksonomy when it is compared with formal structured
vocabularies used by metadata experts or content providers for characterizing digital educational resources.
The application of the proposed evaluation methodology in an existing LOR, namely the
OpenScienceResources Repository (http://www.osrportal.eu/) provided us evidence that ‘verbose’ taggers
can produce tags that are significantly different from formal metadata generated by metadata experts or
content providers.
The paper is structured as follows. Following this introduction, Section 2 provides an overview of
existing studies from the literature, which have investigated the potential benefits of socially tagging digital
educational resources. In Section 3, we present our proposed methodology for identifying different types of
users’ tagging motivation and evaluating their possible influence to the metadata descriptions of digital
educational resources, as well as to the resulted folksonomy when it is compared to formal structured
vocabularies used for metadata authoring by metadata experts or content providers. Section 4, presents the
tagging approaches adopted by the existing LOR (namely OpenScienceResources) where we applied our
proposed methodology, the dataset of the OpenScienceResources repository that was available at the time of
our study, as well as the evaluation results and discussion of our findings. Finally, we present our conclusions
and ideas for future work.
2. LITERATURE REVIEW
Within the TeL literature there are existing works that have examined the added value of socially tagging
digital educational resources. An initial study towards addressing this issue has been conducted in the
framework of the EU-funded project “MELT: a Metadata Ecology for Learning and Teaching”
(http://info.melt-project.eu/) (Zens et al., 2009; Vuorikari and Ayre, 2009). More specifically, MELT project
developed a LOR including around 153.000 digital educational resources. These resources were
characterized by the resources’ authors with educational metadata following an application profile of the
IEEE LOM standard, as well as with social tags added by the end-users of the repository (that is students and
teachers). During the evaluation study of the project, the MELT Repository included 10131 social tags,
which had been added to 2008 digital educational resources. This means that on average 4 social tags were
added per digital educational resource. The main issues that were investigated during the MELT project
evaluation study were the following:
I1: usefulness of social tags for searching digital educational resources.
I2: usefulness of social tags as metadata descriptors of digital educational resources.
The evaluation results in respect to the aforementioned issues showed that:
I1: 23% of searches in MELT Repository were performed based on social tags and 40% of these
searches were found very useful. However, for accurate searches within MELT Repository, social
tags were considered not useful.
124
I2: 25% of social tags contained additional to authors’ information, 26% unnecessary information
and 49% no new information. Moreover, social tags were found to be more useful than formal
vocabulary terms and most of the end-users wanted to change the original metadata description to
adopt some of the social tags as indexing terms.
Another important study about social tagging has been conducted in the framework of the US-funded
project “STEVE: The Museum Social Tagging Project” (http://www.steve.museum/) (Trant, 2009a). More
specifically, the project developed a repository including around 97.000 digital resources of cultural heritage.
These resources were characterized with metadata by professional museum experts as well as with tags added
by the end-users of the repository (that is students and teachers). During the evaluation study of the project,
the STEVE Repository was including 36981 tags which had been added to 1792 digital cultural heritage
resources. This means that on average 20 social tags were added per cultural heritage resource. The main
issues that were investigated during the STEVE project evaluation study were the following:
I1: social tags correlation with museum metadata.
I2: usefulness of social tags as museum metadata descriptors of the digital cultural heritage
resources.
The evaluation results in respect to the aforementioned issues showed that:
I1: 86% of social tags didn’t match with museum metadata added by professional museum experts.
I2: 88% of social tags were considered useful by the professional museum experts
As we can notice from the aforementioned studies, there is a strong interest for investigating the added
value of social tagging on enlarging the metadata descriptions of digital educational resources, as well as the
formal vocabularies used in expert-based metadata. However, both issues have been investigated without
considering the possible implications of users’ tagging motivation to the enlargement of resulting tags and
folksonomies. Next, we aim to address this issue and we present our proposed evaluation methodology.
3. PROPOSED EVALUATION METHODOLOGY

In this section, we present our proposed methodology for identifying different types of users’ tagging
motivation and evaluating their possible influence to the metadata descriptions of digital educational
resources, as well as to the resulted folksonomy when it is compared to formal structured vocabularies used
for metadata authoring by metadata experts or content providers. More specifically, our methodology
includes the following steps:
Step 1 – Identify different underlying behaviours for users’ tagging: This step includes the
discrimination of the users based on their tagging motivation. For this purpose, we adopt two types
of user motivations proposed by Korner (2009):
o Categorizers, who are motivated by categorization and use tags to construct and maintain a
navigational aid to the resources they annotate. For this purpose, categorizers aim to
establish a stable and bounded vocabulary based on their personal preferences and
motivation.
o Describers, who are motivated by description aim to describe the resources they annotate
accurately and precisely. As a result, their tag vocabulary typically contains an open set of
tags.
In order to discriminate between categorizers and describers we adopt a set of measures proposed by
Korner et al. (2010):
o Tag/Resource Ratio: relates the vocabulary size of a user to the total number of digital
educational Resources tagged by this user. Describers, who use a variety of different tags
for their resources, score higher values for this measure than categorizers, who use fewer
tags.
o Orphaned Tag Ratio: characterizes the degree to which users produce orphaned tags (that is
tags assigned to few resources only, and therefore are used infrequently). The orphaned tag
ratio captures the percentage of tags in a user's vocabulary that represent such orphaned
tags. Categorizers vocabulary scores values closer to 0 because orphaned tags would
introduce noise to their personal vocabulary, whereas describers vocabulary scores values
125
ISBN: 978-989-8533-12-8 © 2012 IADIS
closer to 1 due to the fact that describers tag resources in a more verbose and descriptive
way.
o Overlap Factor: measures the phenomenon of an overlap produced by the assignment of
more than one tag per resource on average. Categorizers are interested in keeping this
overlap relatively low. On the other hand, describers produce a high overlap factor since
they do not use tags for navigation but instead aim to best support later retrieval.
o Tag/Title Intersection Ratio: measures how likely users choose tags from the words of an
educational resource title. Categorizers use tags taken from the title and they score values
closer to 1, whereas describers rarely use tags from the title and they score values closer to
0.
Step 2 – Calculate similarity between social tags and educational metadata: During this step we
calculate the similarity between social tags (offered by end-users, that is teachers) and educational
metadata (offered by metadata experts or content providers). The similarity is calculated for social
tags added by describers, as well as for social tags added by categorizers based on the users’
discrimination performed in step 1. At the end of this step, we expect to identify digital educational
resources enlarged with social tags offered by describers and/or categorizers that are different by the
formal metadata descriptions offered by metadata experts or content providers.
Step 3 - Compare folksonomy with formal vocabularies of educational metadata: During this
step, we compare the resulted folksonomy produced by the social tags with formal structured
vocabularies of educational metadata. The comparison is performed with the folksonomy produced
by describers, as well as with the folksonomy produced by categorizers following the users’
discrimination performed in step 1. At the end of this step, we would be able to identify new tags
offered by describers and/or categorizers that can enlarge the formal structured vocabularies of
educational metadata.
4. APPLYING PROPOSED METHODOLOGY TO AN EXISTING

LEARNING OBJECT REPOSITORY
In this section, we apply our proposed evaluation methodology to an existing LOR, namely
OpenScienceResources Repository (http://www.osrportal.eu/). The OpenScienceResources Repository was
developed in the framework of an EU-funded project, referred to as “OpenScienceResources: Towards the
development of a Shared Digital Repository for Formal and Informal Science Education”
(http://www.openscienceresources.eu/). It provides access to openly licensed (through Creative Commons)
science education resources, which can be used by science teachers connecting formal science education in
schools with informal science education activities taken place in European Science Centres and Museums
(Sampson, Zervas and Sotiriou, 2011).
The OpenScienceResources Repository currently includes around 3000 science education resources
characterized by content providers (European Science Centres and Museums) with educational metadata
following an application profile of the IEEE LOM standard (Sampson, Zervas and Sotiriou, 2011). More
specifically, the application profile of the OpenScienceResources Repository includes formal structured
vocabularies at the Classification Element (Nr. 9) of the IEEE LOM standard for describing (a) the subject
domain of science education resources that is related with the science curriculum (the vocabulary includes
400 terms) and (b) the educational objectives that science education resources intend to target (the
vocabulary includes terms selected by Bloom’s Taxonomy of Educational Objectives and its subsequent
revisions and extensions (Anderson and Krathwohl, 2001)).
The science education resources of the OpenScienceResources Repository are also characterized with
social tags added by the end-users of the repository (that is teachers). The social tags includes the following
categories: (a) free tags, that are mainly used by the end-users, so as to describe the topic and/or the subject
domain of a science education resource related with the science curriculum and (b) educational objectives
tags, that are used by the end-users, so as to express the educational objective(s) that a science education
resource can target. The educational objectives tags are selected from a predefined vocabulary following
revised Bloom’s Taxonomy of Educational Objectives and it is the same with the formal vocabulary, used by
content providers for characterizing science education resources with IEEE LOM compliant educational
126
metadata. Table 1 presents the two tagging approaches of the OpenScienceResources Repository and how the
metadata elements added by content providers can be compared with the social tags added by end-users.
Table 1. OpenScienceResources Repository Tagging Approaches
Educational Metadata (according to IEEE LOM) Social Tags

Metadata Element Value Space Tags Categories Value Space
1.5 General.Keyword Free text describing the topic of Free Tags Free text describing
the science education resource the topic and/or the
9.2.2.2 Classification.Taxon Structured vocabulary describing subject domain of a
Path.Taxon.Entry (when purpose the subject domain of the science science education
metadata element has the value education resource related with resource related with
“discipline”) the science curriculum the science
curriculum
9.2.2.2 Classification.Taxon Structured vocabulary describing Educational Structured vocabulary
Path.Taxon.Entry (when purpose the educational objectives that a Objectives Tags describing the
metadata element has the value science education resource intends educational objectives
“educational objective”) to target that a science
education resource
intends to target
Next, we present the snapshot of the OpenScienceResources Repository dataset at the time of our study
(March 2012) that was used for applying our proposed evaluation methodology described in Section 3.
Table 2. OpenScienceResources Dataset (March 2012)
Variables Value
Taggers 434
Tagged Science Education Resources 1877
Social Tags 14707
Free Tags 13117
Educational Objectives Tags 1590
4.1 Describers and Categorizers

In order to cluster the taggers of the OpenScienceResources Repository we applied the set of measures
described in our proposed methodology in Section 3. Table 3 presents the number of categorizers and
describers resulted by applying these measures.
Table 3. Describers and Categorizers
Type of Taggers Value % per Total Taggers

Categorizers 226 52,07 %
Describers 208 47,92 %
Total 434 100,00 %
Additionally, Table 4 presents quantitative data about the number of tags used and the number of science
education resources tagged by each type of tagger.
Table 4. Quantitative Data per Type of Tagger
Type of Number Tags Average Tags Science Education Average Tagged Average Tags
Taggers of Contributed per Tagger Resources Tagged Science Education per Science
Taggers Resources per Education
Tagger Resources
Categorizers 226 1960 8,67 1852 8,19 1,05
Describers 208 12647 60,8 630 3,02 20,07
As it was expected we can notice from Table 4 that describers contributed the vast majority of social tags
of the OpenScienceResources dataset but they have tagged less science education resources than the
categorizers. This means that describers mainly use a small set of digital educational resources, which they
127
ISBN: 978-989-8533-12-8 © 2012 IADIS
want to accurately and precisely describe for later searching and retrieval. On the other hand categorizers
contributed a small amount of social tags to a large set of digital resources aiming to support future browsing
to as many educational resources of the repository.
4.2 Educational Metadata vs. Social Tags

The next step of our proposed evaluation methodology includes similarity calculation between educational
metadata and social tags. More specifically, we calculated (a) the similarity of the metadata values added by
content providers to the element Nr. 1.6 Keyword of the IEEE LOM standard with the free tags category of
the social tags added by describers and categorizers and (b) the similarity of the metadata values added by
content providers to the element Nr. 9.2.2.2 Classification.TaxonPath.Taxon.Entry (when purpose metadata
element has the value “educational objective”) of the IEEE LOM standard with the educational objectives
tags category of the social tags added by describers and categorizers. Table 5 presents average similarity
results for the different type of taggers, as well as the number of science education resources that achieved
low similarity score in average (<0,5). This similarity threshold was selected by considering relevant
thresholds used in other social tagging evaluation studies from the literature (Vuorikari and Ayre, 2009).
Table 5. Educational Metadata vs. Social Tags
Type of Taggers Keyword vs. Classification.Taxon Science Education % per total tagged
Free Tags Path.Taxon.Entry vs. Resources with low science education
Educational Objectives similarity score (<0,5) resources
Tags
Categorizers 0,79 0,71 184 out of 1852 9,93%
Describers 0,76 0,45 307 out of 630 48,73%
As we can notice from Table 5, describers added social tags (mainly to the educational objectives tags
category) that were significantly different from the educational metadata added by the content providers.
Therefore, they have significantly contributed to the enlargement of the metadata descriptions of 307 science
education resources. This was the 48,37% of the science education resources that they have tagged. On the
other hand, categorizers added tags that were quite similar with the educational metadata added by the
content providers. Therefore, this distinction between categorizers and describers is very important because it
facilitates capturing enlarged metadata descriptions of digital educational resources.
4.3 Folksonomy vs. Formal Structured Vocabulary

During the final step of our proposed evaluation methodology, we compared the resulted folksonomy of free
tags category of social tags added by describers and categorizers with the structured formal vocabulary used
by content providers to characterize science education resources with terms related with science curriculum.
In order to achieve that, we calculated the similarity of the describers folksonomy and categorizers
folksonomy with the formal structured vocabulary and we kept those tags that they achieved similarity score
zero. The number of these tags was 202 (from describers) and 94 (from categorizers). Finally, we excluded
the semantic noise from these tags, that is synonyms and subjective tags and we concluded to 46 (describers)
and 10 (categorizers) possible new terms that could enlarge the structured formal vocabulary used at the
element Nr. 9.2.2.2 Classification.TaxonPath.Taxon.Entry (when purpose metadata element has the value
“discipline”) of the IEEE LOM standard. Finally, we investigated whether there were new terms contributed
by both describers and categorizers. Table 6 presents the distribution of the new terms contributed.
Table 6. Distribution of new terms per type of tagger
Type of Taggers New terms Contributed % per Total Terms

Only by Describers 46 82,14 %
Only by Categorizers 10 17,85 %
By Categorizers & Describers 0 0,00 %
As we can notice from Table 6, describers outperformed categorizers by contributing 36 more new terms.
Moreover, there were not common terms contributed by both describers and categorizers. These results
128
provided us evidence that describers have a stronger influence on the enlargement of formal structured
vocabularies, whereas categorizers’ contribution was limited. These findings could be relevant in LORs for
indexing digital educational resources with describers’ tags towards facilitating search and retrieval of digital
educational resources.
5. CONCLUSIONS AND FUTURE WORK

In this paper, we investigated the influence of different tagging styles, namely describers and categorizers on
enlarging the metadata descriptions of digital educational resources. The results of our study performed with
the dataset of an existing LOR, namely OpenScienceResources Repository showed that describers produce
tags that are significantly different from formal metadata, whereas categorizers mainly follow the formal
metadata generated by metadata experts or content providers. As a result, considering tagging motivation
during folksonomies analysis could facilitate capturing enlarged metadata descriptions of digital educational
resources. Future work includes deeper analysis to the results of our study, so as to identify the effect of
describers’ and categorizers’ social tags to the enlargement of metadata descriptions for digital educational
resources with different granularity levels and different formats.
REFERENCES
Anderson, L. W. and Krathwohl, D. R., 2001. A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom’s
Taxonomy of Educational Objectives. New York: Addison Wesley Longman.
Anderson, P., 2007. What is Web2.0? Ideas, technologies and implications for education. [online] JISC Technology and
Standards Watch. Available at: <http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf> [Accessed 28
August 2012].
Bateman, C., Brooks, G., McCalla, T. and Brusilovsky., P., 2007. Applying collaborative tagging to e-learning. [online]
Proceedings of ACM WWW. Available at: <www2007.org/workshops/paper_56.pdf> [Accessed 28 August 2012].
Bi, B., Shang, L. and Kao, B., 2009. Collaborative Resource Discovery in Social Tagging Systems. In: 18th ACM
Conference on Information and Knowledge Management, Hong – Kong, China, 2-6 November.
Bonino, S., 2009. Social Tagging as a Classification and Search Strategy. Germany: VDM.
Cho, C.W., Yeh, T.K., Cheng S.W. and Chang, C.Y., 2011. A Social Tagging System for Online Learning Objects.
Advanced Science Letters, 4(11-12), pp. 3362-3365.
Dahl, D. and Vossen, G., 2008. Evolution of learning folksonomies: Social Tagging in e–learning repositories.
Technology Enhanced Learning, 1(2), pp. 35-46.
Derntl, M., Hampel, T., Motschnig-Pitrik, R. and Pitner, T., 2011. Inclusive social tagging and its support in Web 2.0
services. Computers in Human Behavior, 27(4), pp. 1460-1466.
Heymann, P., Koutrika, G. and Molina, H., 2008. Can Social Bookmarking Improve Web Search?. In:.1st International
Conference on Web Search and Data Mining (WSDM 2008), pp. 195-205, Palo Alto, California, USA 11-12
February.
IEEE Learning Technology Standards Committee (LTSC), 2002. Final Standard for Learning Object Metadata, [online]
IEEE Learning Technology Standards Committee. Available at: <http://ltsc.ieee.org/wg12/> [Accessed 28 August
2012].
Korner, C., 2009. Understanding the motivation behind tagging. In: 20th ACM Conference on Hypertext and Hypermedia,
Torino, Italy, 29 June - 1 July.
Korner, C., Benz, D., Strohmaier, M., Hotho, A. and Stumme, G., 2010. Stop thinking, start tagging - tag semantics
emerge from collaborative verbosity. In: 19th International World Wide Web Conference (WWW 2010). Raleigh, NC
26-30 April. USA: ACM.
McGreal, R., 2004. Online Education Using Learning Objects. Washington, D.C.: Falmer Press.
McGreal, R., 2008. A typology of learning object repositories. In: H.H. Adelsberger, Kinshuk, J. M. Pawlovski and D.
Sampson, eds. International Handbook on Information Technologies for Education and Training, pp. 5-18. 2nd
Edition, Springer.
Sampson, D., Zervas, P. and Sotiriou, S., 2011. Science Education Resources Supported with Educational Metadata: The
Case of the OpenScienceResources Web Repository. Advanced Science Letters, Special Issue on Technology-
Enhanced Science Education, 4(11/12), pp. 3353-3361 (9).
129
ISBN: 978-989-8533-12-8 © 2012 IADIS
Trant, J., 2009a. Studying Social Tagging and Folksonomy: A Review and Framework. [online] Journal of Digital
Information. Special Issue on Digital Libraries and User Generated Content. Available at:
<http://arizona.openrepository.com/arizona/bitstream/10150/105375/1/trantstudyingFolksonomy.pdf> [Accessed 28
August 2012].
Trant, J., 2009b. Tagging, Folksonomy and Art Museums: Results of steve.museum’s Research. [online] Archives &
Museum Informatics. Available at: <http://conference.archimuse.com/files/trantSteveResearchReport2008.pdf/>
[Accessed 28 August 2012].
UNESCO, 2002. Forum on the Impact of Open Courseware for Higher Education in Developing Countries, [online]
Final Report. Available at: <http://unesdoc.unesco.org/images/0012/001285/128515e.pdf. 2002> [Accessed 28
August 2012].
Vuorikari, R. and Ayre, J., 2009. MELT Final Report of Phase II. [online] European Schoolnet. Available at:
<http://info.melt-project.eu/shared/data/melt/D5_5_PhaseIIreport_Final.pdf> [Accessed 28 August 2012].
Yande, Y., Jatowt, A., Nakamura, S. and Tanaka, K., 2007. Can Social Bookmarking En-hance Search in the Web?. In:
Joint Conference on Digital Libraries (JCDL 2007), pp. 107-116, Vancouver, British Columbia, Canada 18-23 June.
Zens, B., Morgeditsch., W, Meisinger, E., Reiterer, S. and Baumgartner, P., 2009. MELT Deliverable 7.3 Final
Evaluation Report. [online] European Schoolnet. Available at: <http://info.melt-
project.eu/shared/data/melt/MELT_D7_3_Final_Evaluation_Report.pdf> [Accessed 28 August 2012].
130
USING WIKIS TO FOSTER COLLABORATIVE WRITING:

EXPLORING INFLUENCING FACTORS TO SUCCESSFUL
IMPLEMENTATION
Said Hadjerrouit
University of Agder
Serviceboks 422, 4604 Kristiansand, Norway
ABSTRACT
Wiki technology provides new opportunities to foster collaborative learning in various educational settings. To
empirically examine the impact of wikis on learning, this article explores students’ collaborative writing activities
performed on MediaWiki. The activities were analyzed using a taxonomy with ten categories (clarify content, add
content, delete content, add link, delete link, fix link, grammar, spelling, style/typography, and formatting). The work
also analyses students’ comments posted on the discussion page of the wiki. The results show important differences in the
types of contributions across the categories investigated. The results also reveal that the level of collaboration and
discussion was relatively low compared with other activities performed on the wiki. Finally, the article suggests a number
of factors influencing wiki-based collaborative writing in teacher education.
KEYWORDS
Collaboration, collaborative learning, collaborative writing, MediaWiki, wiki.
1. INTRODUCTION
With the emergence of wiki technology, opportunities for fostering group interaction and participation are
greatly enhanced. Combined with the collaborative learning theory and the socio-constructivist view of the
learning process, wikis are increasingly becoming potentially powerful tools to foster collaboration among
participants (Caple & Bogle, 2011; Grant, 2009; Li & Zhu, 2011; Tetard, Packalen & Patokorpi, 2009;
Thomas, King, & Minocha, 2009). Wikis offer possibilities for teachers and learners to collaborate on joint
assignments and collective writing tasks. They enable active participation of contributors by editing and
revising each other’s writings, and hence contributing to the sharing of knowledge and collective
development of wikis. However, although teachers and students are encouraged to exploit the possibilities
offered by wikis in their practices, they may not be confident or experienced in supporting collaborative
writing, or adapt the technology to the characteristics of the learners and target audience. This paper reports
on a study that focuses on wiki technology as a tool for collaborative writing in teacher education. The
overall aim is to assess the activities carried out on the wikis by means of the history function that tracks all
students’ contributions. Particular attention was devoted to students editing one another’s contributions. The
activities are analyzed quantitatively using a taxonomy proposed by Pfeil, Zaphiris, and Ang (2006). The
work also analyses students’ comments posted on the discussion page of the wiki. Finally, influencing factors
are discussed to exploit the opportunities offered by wiki technology to foster collaborative writing in teacher
education.
131
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. BACKGOUND
2.1 Wiki Technology

This work used one of the most popular wiki platforms – MediaWiki - to perform collaborative writing
activities (Kasemvilas, & Olfman, 2009). MediaWiki uses a simplified HTML language and provides an
extensive functionality for user authentication (Su & Beaumont, 2010). Another important functionality of
MediaWiki is the history function that keeps track of students’ edits by name, date, and colour coding (Lund
& Smørdal, 2006). In addition, MediaWiki provides a discussion page for reflecting on the wiki content.
2.2 Collaborative Writing

Collaboration is an activity that enables participants to accomplish a task collectively (Ta-Elhasid & Meishar-
Tal, 2007; Witney & Smallbone, 2011). Wikis offer a new way to work collaboratively by creating collective
content, and as such, they facilitate collaborative writing and group discussion. Collaborative writing is a
coordinated activity that enables participants to edit and revise each other’s contribution to the wiki task
(Chao & Lo, 2009; Meishar-Tal & Gorsky, 2010; Trentin, 2009; Witney & Smallbone, 2011), as opposed to
simply splitting up the task, work independently of each other, and then assemble individual contributions to
a final wiki. Collaboration is grounded in the social-constructivist learning theory (Vygotsky, 1978), and
assumes that participants can achieve more in terms of learning benefits than individuals.
2.3 A Taxonomy to Analyze Students’ Actions Performed on Wikis

Taxonomies have been proposed in the research literature to classify and analyze collaborative writing
activities performed on wikis (Meishar-Tal & Gorsky, 2010; Pfeil, Zaphiris, & Ang, 2006). The taxonomy
used in this paper draws on the one developed by (Pfeil, Zaphiris, & Ang, 2006). This included originally 13
categories, of which the following 10 were identified as important for this work (Table 1).
Table 1. Taxonomy and categorization of activities (Pfeil, Zaphiris, & Ang, 2006, pp. 101).
Category Explanation
Add Information Addition of topic-related information (the information must not consist only of links).
Add Link Addition of links to an existing set of listed links or linking of a word within an existing sentence
to a page (links to other Wikipedia pages or to external Internet pages).
Clarify Information Rewording of existing information without adding new information. Rewording done in order to
clarify the content (e.g., substitution of certain words for a better understanding, change of the
word order or deletion/addition of words in order to clarify).
Delete Information Deletion of topic-related information (the information must not consist only of links).
Delete Link Deletion of links from the set of listed links or removal of the linking function from a word within
an existing sentence (links to other Wikipedia pages or to external Internet pages).
Fix Link Modification of an existing link (can be an alteration of the linked URL or the name of the link).
Format Contributions that affect the appearance or structure of the whole page (e.g., addition of space
lines, sorting/moving of paragraphs or links and addition of subtitles in order to structure the
content).
Grammar Alterations of the grammar (e.g., change of punctuation).
Spelling Correction of spelling mistakes (e.g., reversed letters or capital letter).
Style/Typography Contributions that affect the presentation/appearance of the text (e.g., bold/italic/underlined text).
132
2.4 Related Work

Most research work on wiki reports on students’ perceptions of collaborative learning and writing by means
of qualitative methods. A number of researchers (Arnold et al., 2009; Britcliffe, & Walker, 2007; Cole 2009;
Ebner et al., 2008; Minocha & Thomas, 2007; Karasavvidis, 2010; Lund & Smørdal, 2006) reported that
students do not collaborate when they use wiki and rarely edit each other’s contributions. Several hypotheses
were raised to explain the low level of collaboration: limited student contribution, reluctance and resistance
to use wiki, dominant learning paradigm, problem of ownership, lack of appropriate pedagogy, etc. In
addition, according to Pifarre and Fisher (2011), there is relatively little research on successful
implementations of wikis supporting collaborative writing. In quantitative terms, a small but growing number
of studies have recently drawn on the data log generated by the history function of wikis. Hadjerrouit (2011)
reported that most students do not collaborate when they use wiki to edit collective documents. Instead, they
focused mostly on adding content to existing pages and technical aspects. Similarly, Hadjerrouit (2012)
highlighted the problems and difficulties of using wikis to edit each other’s contributions. Leung and Chu
(2009) also reported that students worked individually most of the time, and edited each other’s contributions
if necessary. Likewise, Judd, Kennedy, & Cropper (2010) provided evidence against a general tendency to
collaborative writing. In some contrast, Meishar-Tal and Gorsky (2010) indicated that adding text was carried
by a large majority of students, but the percentage of editorial changes was higher than adding sentences,
because the students were required to edit each other’s work.
3. METHODOLOGY
3.1 Objective
This work aims at exploring the extent to which students collaborated to perform wiki projects associated
with collaborative writing in teacher education. The work focuses particularly on students editing each
other’s contributions to the wikis and comments posted on the discussion page.
3.2 Participants
An eight-week wiki project was introduced in a Web 2.0 technology course. The participants were 16
students divided into 6 groups of 2 to 4. None of the students were experienced wiki-based collaborative
writing. Some possessed good technical skills, and some had background in pedagogy. The wiki topics were
chosen by the students themselves. The specificities and technical features of wikis were introduced to the
students during the first week of the projects. Lectures on collaborative writing were given in the following
two weeks. The students were required to submit their wikis for continuous supervision.
3.3 Learning and Assessment Goals

To perform collaborative writing using MediaWiki, the teacher provided a set of three learning goals. First,
the wikis should follow general usability criteria such as technical layout, formatting, and style. Second, the
wikis must contain information of good quality, without linguistic, grammar, and spelling errors. The content
should draw on recent curricular development, and include study material that is well structured with heading
and subheading, images, tables, lists, and references. Third, the wikis should be self-explaining, and offer
information that is relevant to the target audience. Some of these tasks could be done individually, for
example adding or deleting content. However, developing an overall wiki requires collaborative work such as
arguing, discussing, and reflecting on the content through editing each other’s contributions, adapting the
language to the needs of the target audience, designing an overall structure of the wiki, and making
cross-linking. Given these requirements, the students were encouraged to edit each other’s contributions, and
take actively part in discussion. Finally, in line with the wiki philosophy based on collaborations, the students
were not assessed individually, but as a group working collaboratively. Nevertheless, the history function can
be used to look at the students’ individual contributions to the wiki.
133
ISBN: 978-989-8533-12-8 © 2012 IADIS
3.4 Data Collection and Analysis Methods

The work used both quantitative and qualitative methods. Firstly, the total number of actions per group and
category, including their frequencies were collected and analyzed, such as whether the action was an
addition, deletion or modification of content; addition, deletion, or fixation of a link; formatting, spelling,
style, or grammar etc. Secondly, the comments raised in the discussion page were categorized by increased
level of criticality, and analyzed both quantitatively in terms of number of comments, and qualitatively in
terms of quality of the comments provided.
4. RESULTS
This section describes the results achieved in terms of actions carried out on the wiki across the categories
investigated, and students’ comments posted on the discussion page. Contribution to collaborative writing
and discussion is also assessed.
4.1 Contribution to Collaborative Writing

Table 2 shows the frequency of actions that fell under each of the 10 categories investigated. The total
number of actions was 2856, which means an average of 178.5 actions per student (n = 16). Note that a single
edit may involve several actions, for example a student could add content and delete a link. In this case, both
actions in the categories add content and delete link were recorded. The table reveals that the most important
category performed on the wiki in terms of average frequency relative to the total number of actions was
formatting, followed by addition of content and links, clarification of content and fixing of links, deletion of
content, style/typography, deletion of links, spelling, and finally grammar. In terms of clarifying content, and
thus editing each other’s contribution, two subcategories can be distinguished: a student clarifying his/her
own content, which is not the focus of this work, and clarifying each other’s content, which is the main
concern of this study. The statistics does not indicate the frequency of each subcategory. However, even if all
actions associated with clarifying content fell under the second subcategory, which is the best possible
scenario, the average frequency of 12.04 % cannot be considered as high compared with the percentages
achieved for formatting, addition of content and links. This is the case of four groups (1, 2, 5, and 6), where
the average frequency for clarifying content is lower than 12.04 %, and group 4 with a slightly higher
frequency (14.93 %). The only exception is group 3 that achieved a percentage of 23.79 %, which in itself
cannot be regarded as very high. Considering these frequencies as the best possible results that can be
achieved, it can be implied that only a few actions fell under the category clarify content by editing each
other’s contributions.
Table 2. Types of actions in each category in terms of frequencies and total number of actions in ascending order.
Total no. of Frequency
Group Group Group Group Group Group
Category actions per of actions
1 2 3 4 5 6
category in %
Formatting 155 113 80 163 109 48 668 23.39 %
Add content 65 51 81 259 89 44 589 20.62 %
Add link 79 36 136 119 82 53 505 17.68 %
Clarify content 24 21 114 138 44 3 344 12.04 %
Fix link 69 8 20 68 39 17 221 7.73 %
Delete content 24 19 18 86 35 25 207 7.25 %
Style/Typography 33 9 6 19 38 29 134 4.69 %
Delete link 23 0 5 29 14 6 77 2.70 %
Spelling 13 7 8 29 6 5 68 2.39 %
Grammar 3 3 11 14 9 3 43 1.51 %
Total no. of
448 260 479 924 465 233 2856 100 %
actions per group
No. of students
2 3 4 2 3 2 n = 16
per group
134
4.2 Contribution to Discussion

A detailed analysis of the comments posted on the discussion page showed that the students made comments
on a range of issues. Following (Su & Beaumont, 2010), the content of the comments were analyzed and
categorized by increased level of criticality (Table 3):
• Comments on technical issues of the wiki
• Comments on the wiki content
• Comments on collaborative writing
Table 3. Comments posted on the wikis derived from a content analysis.
Level of No. of comments Frequency
Categories
criticality per category in %
Comments on technical layout, structure, formatting,
Low 106 55.21 %
images, tables, lists, paragraphs, headings, subpages.
Comments on content, proof reading, corrections,

Middle 74 38.54 %
references, linking, wiki length.
Comments on reflections related to usefulness of
High information, critical review of literature, adaption of 12 6.25 %
language to target audience, editing each other’ content
Total no. of comments 192 100 %

Most comments were posted the last two-three weeks before the project deadline. Otherwise it was
difficult to follow the discussion treads of the respective wikis, because the date of contribution or/and name
of contributor were missing. As the table shows, most comments were low-level criticality comments
(55.21%). These related mostly to technical layout, such as insertion and placing of tables and figures,
formatting of headings and subheadings, etc. Middle-level criticality comments related mostly to addition or
deletion of content or links, corrections, wiki length, etc. Finally, high-level criticality comments were
associated with critical issues of the wiki content, such as usefulness of information sources and study
material used to design the wiki. Only one student emphasized the need to work collaboratively. None of the
students discussed the issue of how to adapt the language to the characteristics of the target audience.
There were several comments with low-level criticality, of which the following are representative:
Now we have nice images and I think there are enough. We can probably insert a picture or two to
the section on (…).
Fixed a positioning of images, but I am still not particularly happy (…). Need some more lists to
meet the requirement. Suggestions would be appreciated.
The following were typical comments associated with middle criticality:
We have to at least make sure that none of the links are "empty". We now have about 3000 words
(…). Thus, 1000 words are still missing. Any suggestions on what we can write more detailed?
I guess today we will deliver the wiki, so we have to get it done. Who takes care of creating tables? I
thought I was writing the page about (…), and proofread the entire wiki. We also need to arrange
the reference list.
There were few high-level criticality comments. The following illustrates the type of students’ comment:
The next will probably be that we meet and work collaboratively with the wiki, do you agree?
Considering that the category with a high-level of criticality is the one that requires most collaboration in
terms of reflection and critical discussions, it can be asserted that the level of collaboration was rather low.
This is in accord with the results achieved for the categories of actions carried out on the wikis. In addition,
the number of comments (192) is very low compared with the number of actions (2856), which means an
average score of only one comment per 14.87 actions.
135
ISBN: 978-989-8533-12-8 © 2012 IADIS
5. DISCUSSION
5.1 Summary of Results

As the results clearly show (Table 2) the most frequent action was formatting, followed by adding content
and links. Clarifying each other’s content was an action that the students did not carry out much. As a result,
collaboration among participants was rather low compared with the frequencies of other actions. This is
confirmed by the comments posted on the discussion page (Table 3).
5.2 Factors affecting Collaborative Writing

Although the students were encouraged to edit each other’s work, and reflect on their ideas and concerns, the
results indicate that they preferred to focus more on formatting, technical layout, addition of content, or
making links to pages than collaboration. In spite of the fact that some issues were discussed face-to-face or
by means of traditional communication channels, there is little evidence that the students identified gaps in
their knowledge by reflecting on the content of the wikis by means of literature review and adapting the
content to the characteristics of the target audience. In addition, most discussions happened the last two-three
weeks before the project deadline. Likewise, the history function also shows that all group worked much as
the deadline approached, approximately two weeks before the delivery of the wiki projects. This is in accord
with previous research indicating the students' tendency to postpone much of their work until just before the
deadline (Judd, Kennedy, & Cropper, 2010; Leung & Chu, 2009; Meishar-Tal & Gorsky, 2010). This
behavior somehow undermined the students’ opportunities to fully collaborate, edit each other’s
contributions, and discuss with their peers. A possible explanation is that the students were more concerned
about passing the course, while doing as little work as possible than learning collaboratively. The fact that
they mentioned several times the length of the wiki, the linking of words, uploading of images, insertion of
tables, and technical layout, is an indication that the students were more concerned about passing the course
than critically evaluating their knowledge. However, this is not the only explanation for low collaboration.
Adapting the content to the characteristics of the users was an important requirement, but this issue was
not raised in the discussion page. This is confirmed by the final wiki products, which failed to thoroughly
address this issue. Indeed, the analysis of the wikis shows that the students did not identify gaps in their
knowledge in order to make the wikis more relevant and attractive to the users. Rather, the students focused
more on selecting content from Wikipedia and other Web sites than reflecting on their knowledge. Adapting
the content to the characteristics of the target audience is a demanding task that requires knowledge about the
reliability and usefulness of information sources, including central facts and concepts of the topic, and how to
connect them to foster meaningful understanding. It is obvious that this issue requires drafting, rephrasing,
and reworking the language recursively until it fits the needs of the target audience. In addition, an acceptable
level of language proficiency needs to be taken into account (Li & Zhu, 2011). In turn, drafting and
reworking the wiki content cannot be done properly without editing each other’s contributions. Another
explication raised by (Grant, 2009) is that students do not perceive their wikis as an authentic activity that
requires an authentic audience. They think that their wikis will not continue after the end of the project. As a
result, combined with the importance of grades, the most important audience for them is their teacher.
Grant (2009), in line with the results of previous research work (Elgort et al., 2008; Forte & Bruckman,
2007; Lund & Smørdal, 2006), also suggests that students appear to use practices of individualised written
work they were accustomed, rather than collaborating to realize shared knowledge, particularly in the
absence of collaboration models to draw on. The wiki history function seems to confirm the view that
students tended to approach the writing task more individually than collaboratively by splitting up the wiki
task into subtasks, not only in the very beginning of the project, which is reasonable and understandable, but
also throughout the entire project period. This behaviour may be explained by the fact that collaborative
writing is more challenging in terms of cognitive efforts, active participation, group interactions, and time
management than just splitting up the wiki task into subtasks, working individually without collaborating,
and finally putting all the subtasks together to create a final wiki (Hadjerrouit, 2011).
Another possible explanation may be the assessment form being used (Harsell, 2010; Tetard, Packalen, &
Patokorpi, 2009). Since the students were not assessed individually, but as a group working collaboratively to
achieve a common goal, students tended to focus more on the final wiki product than the collaborative
136
process. As a result, the students did not consider editing each other’s contributions as useful or desirable,
and they preferred more individual work than collaboration. Nevertheless, wikis offer a possibility to assess
individual contributions by means of the history function, which records all students’ activities, particularly
when they are required to demonstrate regular contributions and discussion (Grant, 2009, Harsell, 2010).
However, being aware of the usefulness of the history function to assess individual contributions may not
automatically facilitate collaboration, if students are not able to develop effective collaborative and
discussion strategies.
Finally, usability obstacles of wiki technologies may disrupt the students’ learning experience (Minocha
& Thomas, 2007). Likewise, the absence of a WYSIWYG editor may prevent students from fully using wiki
for project collaboration (Chao, 2007). As a consequence, although students are positive when they use
wikis, they are reluctant to fully collaborate, in accordance with several previous research studies. However,
for Selwyn (Cited in Grant (2009)) collaboration is less a technological problem than a cultural and
pedagogical issue. Focusing on removing technological barriers to realizing wiki potentialities is a
“reductionist thinking equivalent to technological determinism” (Grant, 2009, p. 113). As a result, it appears
that while wikis might support group work, collaborative writing is not reductible to the technology. Rather
the role of the teacher, the nature of the task, time management, motivation, assessment, pedagogy, and
technology integration are crucial elements in encouraging students to work more collaboratively (Caple &
Bogle, 2011; Lund & Smørdal, 2006, Tay & Allen, 2011).
6. CONCLUSIONS
Although this work has its limitations, because of the small sample size (n = 16) and short duration of the
wiki projects (8 weeks), it can serves as a basis for further explorations in wiki-based collaborative writing.
Given these considerations, some conclusions can be drawn. Firstly, the results are line with research that
reports on students’ reluctance to edit each other’s contributions to the wikis. Secondly, the history function
of the wiki provides an excellent research tool to analyze students’ contributions using an appropriate
taxonomy of activity categories (Judd, Kennedy, & Cropper, 2010; Meishar-Tal & Gorsky, 2010; Pifarre &
Fisher, 2011). Finally, factors influencing collaborative writing are identified. These need to be considered by
teacher educators to successfully exploit the potentialities of wikis to foster collaborative writing. In future
work, a longitudinal study will be undertaken to explore students’ collaborative writing activities over a
period of three years to confirm the results of this work. Future research will also be undertaken with a larger
population of students to strengthen the validity and reliability of the results.
REFERENCES
Arnold, N., Docate, L. and Kost, C., 2009. Collaborative Writing in Wikis. In: Lomicka, L. and Lord, G. (Eds.). The Next
Generation: Social Networking and Online Collaboration in Foreign Language Learning. CALICO Monograph, Vol.
8, pp. 115–114.
Britcliffe, W. and Walker, R., 2007. Making Wikis Work: How Do We Create the Conditions for Effective Collaborative
Learning? In: ALT-C 2007. Nottingham, UK, 4–6 September, pp. 91–92.
Caple H. and Bogle, M., 2011. Making Group Assessment Transparent: What Wikis Can Contribute to Collaborative
Projects. Assessment & Evaluation in Higher Education, iFirst Article, pp. 1-13.
Chao, J., 2007. Student Project Collaboration Using Wikis. Proceedings of the 20th Conference on Software Engineering
Education and Training (CSEE&T). Dublin, Ireland, pp. 255-261. Washington, DC: IEEE Press.
Chao, Y-C.J. and Lo, H-C., 2009. Students’ Perceptions of Wiki-Based Collaborative Writing for Learners of English as
a Foreign Language. Interactive Learning Environments, iFirst Article, pp. 1–17.
Cole, M., 2009. Using Wiki Technology to Support Student Engagement: Lessons from the Trenches. Computer &
Education, Vol. 52, pp. 141-146.
Ebner, M., Kikckmeier-Rust, M. and Holzinger, A., 2008. Utilizing Wiki-Systems in Higher Education Classes: A
Chance for Universal Access? Universal Access in the Information Society, Vol. 7, pp. 199–207.
Elgort, I., Smith, A.G. and Toland, J., 2008. Is Wiki an Effective Platform for Group Course Work? Australian Journal of
Educational Technology, Vol. 24, No. 2, pp. 195–210.
137
ISBN: 978-989-8533-12-8 © 2012 IADIS
Forte, A. and Bruckman, A., 2006. Constructing Task: Wiki as a Toolkit for (Collaborative?) Learning. Proceedings of
the 2007 International Symposium on Wikis. Montreal, Quebec, Canada, 2007, ACM, pp. 31–42.
Grant, L., 2009. ‘I don’t’ Care Do Our Own Page!’ A Case Study of Using Wikis for Collaborative Work in a UK
Secondary School. Learning, Media, and Technology, Vol. 34, No. 2, pp. 105-117.
Hadjerrouit, S., 2012. Using Wikis for Collaborative Learning: Critical Evaluation and Recommendations. Proceedings
of IADIS International Conference on e-Society 2012. Berlin, Germany, 10-13 March 2012, pp. 91-98.
Hadjerrouit, S., 2011. Collaborative Writing with Wikis: Evaluating Students' Contributions. Proceedings of the IADIS
International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA 2011). Rio de Janeiro,
6-8 November 2011, Brazil, pp. 173-180.
Harsell, D. M., 2010. Wikis in the Classroom: Faculty and Student Perspective. Journal of Political Science Education,
Vol. 6, No. 3, pp. 310-314.
Judd, T., Kennedy, G. and Cropper, S., 2010. Using Wikis for Collaborative Learning: Assessing Collaboration through
Contribution. Australasian Journal of Educational Technology, Vol. 26, No. 3, pp. 341-354.
Karasavvidis, I., 2010. Wiki Uses in Higher Education: Exploring Barriers to Successful Implementation. Interactive
Learning Environments, Vol. 18, No. 3, pp. 219-231.
Kasemvilas, S. and Olfman,L., 2009. Design Alternatives for a MediaWiki to Support Collaborative Writing. Journal of
Information, Information Technology, and Organizations, Vol. 4, pp. 87–104.
Li, M. and Zhu, W., 2011. Patterns of Computer-Mediated Interaction in Small Writing Groups Using Wikis. Computer
Assisted Language Learning, iFirst Article, pp. 1-22.
Leung, K. and Chu, S. K.W., 2009. Using Wikis for Collaborative Learning: A Case Study of an Undergraduate
Students’ Group Project in Hong Kong. Available online: http://www.ickm2009.org/snews/upload/ickm_2009
Lund, A. and Smørdal, O. 2006. Is there a Space for the Teacher in a Wiki? Proceedings of WikiSym’06, August 21–23,
2006, Odense, Denmark, pp. 27–45.
Meishar-Tal, H. and Gorsky, P., 2010. Wikis: What Students Do and Do not Do When Writing Collaboratively. The
Journal of Open and Distance Learning, Vol. 25, No. 11, pp. 25-35.
Minocha, S. and Thomas, P. G., 2007. Collaborative Learning in a Wiki Environment: Experiences from a Software
Engineering Course. New Review of Hypermedia and Multimedia, Vol. 13, No. 2, pp. 187-209.
Pifarré; M. and Fisher, R., 2011. Breaking up the Writing Process: How Wikis Can Support Understanding the
Composition and Revision Strategies of Young Writers. Language and Education, Vol. 25, No. 5, pp. 451-466.
Pfeil, U., Zaphiris, P. and Ang, C.S., 2006. Cultural Differences in Collaborative Authoring of Wikipedia. Journal of
Computer-Mediated Communication, Vol. 12, No. 1, pp. 88–113.
Selwyn, N., 2008. Developing the Technological Imagination: Theorising the Social Shaping and Consequences of New
Technologies. In: Livingstone, S. (2008). Theorising the Benefits of New Technology for Youth: Controversies of
Learning and Development. London/Oxford: London School of Economics/University of Oxford.
Su, F. and Beaumont, C., 2010. Evaluating the Use of a Wiki for Collaborative Learning. Innovations in Education and
Teaching International, Vol. 47, pp. 417-431.
Tay, E. and Allen, M., 2011. Designing Social Media into University Learning: Technology of Collaboration or
Collaboration of Technology. Educational Media International, Vol. 48, No. 3, pp. 151-163.
Tal-Elhasid, E. and Meishar-Tal, H., 2006. Models for Activities, Collaboration and Assessment in Wiki in Academic
Courses. Available online: http://www.biu.ac.il/bar-e-learn/eden2007/tal_tal.doc
Tetard, F., Patokorpi, E. and Packalen, K., 2009. Using Wikis to Support Constructivist Learning: A Case Study in
University Education Settings. Proceedings of the 42nd Hawaii International Conference on System Sciences, pp. 1–
10.
Thomas, P., King, D. and Minocha, S., 2009. The Effective Use of a Simple Wiki to Support Collaborative Learning
Activities. Computer Science Education, Vol. 19, No. 4, pp. 293-313.
Trentin, G., 2008. Using a Wiki to Evaluate Individual Contribution to a Collaborative Learning Project. Journal of
Computers Assisted Learning, Vol. 25, pp. 43-55.
Vygotsky, L.S., 1978. Mind and Society: The Development of Higher Mental Processes. Cambridge, MA, Harvard
University Press.
Witney, D. and Smallbone, T., 2011. Wiki Work: Can Using Wikis Enhance Student Collaboration for Group
Assignment Task? Innovations in Education and Teaching International, Vol. 48, No. 1, pp. 101-110.
138
DEVELOPING TECHNOLOGICAL AND PEDAGOGICAL

AFFORDANCES TO SUPPORT COLLABORATIVE
INQUIRY SCIENCE PROCESSES
Manoli Pifarré1, Rupert Wegerif2, Alba Guiral1 and Mercè del Barrio1
1
Universitat de Lleida , Av. de l’Estudi General, 4. E 25001 Lleida - Spain
2
University of Exeter, St. Luke’s Campus, Heavitree Road. UK EX1 2LU
ABSTRACT
One key feature of scientific community is collaborative shared enquiry and problem solving mediated by electronic
networks. We report on the development and evaluation of an ICT tool to support learning how to learn together (L2L2)
in Science Education as part of an EC funded project called ‘Metafora’ (FP7-ICT-2009.4.2/257872). Through literature
review we isolated some key features of L2L2 and inquiry science processes. We turned these into icons within an ICT
environment to support planning and reflection of inquiries stimulated by real-world challenges. In this paper we report
the design-based research study carried out in secondary schools in Spain in order to gain understanding about the
Metafora’s technological and pedagogical affordances to support students’ awareness of the key aspects of learning
together and the key scientific inquiry processes. Our findings suggest that the ICT environment may help rise students’
awareness of key collaborative scientific processes. These findings may contribute to the development of tools to support
more web-mediated collaborative learning.
KEYWORDS
Learning to learning together (L2L2), Computer Supported Collaborative Learning (CSCL), Knowledge-age skills,
Technology Enhanced Learning (TEL), Secondary Education, Inquiry-based science learning (IBSE).
1. INTRODUCTION
Collaboration is a central tenet of knowledge era. In 2.0 societies, citizens are massively engaged in social
networking activities to exchange information, collaborate in virtual games and create multimedia artifacts.
In the economic world, collaborative technologies are mediating knowledge creation processes and project
management. There can be no doubt that the society requires people to possess the adequate skills to
participate actively and constructively in collaborative and creative practices (Minocha & Thomas, 2007).
Although, young people could gain from various kinds of informal computer supported collaborative
experience, there are no specific web-based tools designed to support them to learn how to learn together in
subject-specific domains. Furthermore, the current education systems do little to equip children and young
people with the complex competence of problem solving and learning together with others online.
The Metafora project1 funded by the EC Framework 7 ICT program aims to design a pedagogy using
web-based tools to support learning to learn together in the context of Maths and Science for schools. A key
technical and pedagogic innovation of the project will be to support learning to learn within a community. To
gain deep knowledge of any specific domain learners need to integrate learning experiences with shared
reflection through dialogue in a way that builds conceptual understanding cumulatively.
1
‘Metafora’, Learning to learn together: A visual language for social orchestration of educational activities. FP7-ICT-2009.4.2
Technology Enhanced Learning contract no. 257872
139
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. THE WEB-BASED LEANING ENVIRONMENT: METAFORA

Metafora aims to provide a holistic environment in which students will collaboratively plan and organize
their work, as well as collaborate in solving science challenges and problems over a relatively long time
period.
A web-based Metafora learning environment is being developed that includes 4 main tools: a) the
challenge; b) the planning and reflection tool; c) the argumentation tool and d) the microworlds. Next we
describe each of these tools integrated in the Metafora platform:
2.1 The Challange

Challenge-based learning methodology was pioneered by education staff at Apple Inc. and aims to engage
learners in meaningful learning context; authentic connection with multiple disciplines, multiple points and
multiple possible solutions and focus on the development of 21st century skills (Johnson & Adams, 2011).
Metafora project incorporates challenge-based learning objectives and at the beginning of a typical
Metafora-based activity, a group of students is formed and receives a relatively complex assignment –the
challenge. The challenge is built in way that will require the students to plan how they are going to approach
the solution in order to reach it on time. After planning, the group begins with an iterative process entailing
enactment – discussion - revision of the plan, until the peers obtain a solution for the challenge.
2.2 The Planning and Reflection Tool

The planning and reflection implements an understanding of the key features of learning together in a visual
language intended to facilitate greater awareness of the process of learning together.
The visual language was designed after a detailed research carried out during the first year of the
Metafora project. The consortium used literature review and design-based research in order to refine and
develop a stable visual language for implementation in the Metafora planning and reflection tool (Wegerif et
al. 2012). The visual language included in the planning and reflection tool has 6 types of components:
- Main activity stages implemented as icons in blue boxes: Explore, Reach agreement, Define
questions, Build model, Find hypothesis, Test model, Refine model, Draw conclusions, Prepare presentation,
Reflect on process, Blank (for students to define if required)
- Activity processes that occur within each activity stage implemented as icons in green circles that
attach to the boxes: Build, Experiment, Hypothesize, Make notes, Propose an alternative, Report, Reach
agreement, Anticipate, Brainstorm, Evaluate, Gather information, Present, Reflect, Discuss, Simulate,
Analyze, Allocate roles, Blank (for students to define if required)
- Attitudes implemented as icons in different colored hats: Open, Positive, Critical, Creative, Ethical,
Rational, Intuitive, Blank (for students to define if required)
- Roles, implemented as icons in grey circles: Manager, Evaluator, Note taker, Blank (for students to
define if required)
- Resources, implemented as an icon that represent the specific available resource: LASAD and the
Microworlds.
- Connectors, implemented as black, blue and red arrows indicating any relationship, causal
relationship and temporal relationship.
In figure 1 we present the visual appearance of the Metafora platform and the planning and reflection
tool.
2.3 The Argumentation Tool – LASAD

An online dialogue support system using a dynamic concept mapping tool called LASAD was integrated in
Metafora platform. LASAD is web-based argumentation tool that enables groups of learners to discuss their
work in a structured way (Loll, Pinkwart, Scheuer, & McLaren, 2009; Scheuer, McLaren, Loll, & Pinkwart,
2009). LASAD is a collaborative, shared workspace containing a graphical argumentation environment and a
140
chat tool. Students can use this space to share ideas and organize their thoughts in order to solve the
challenge.
2.4 Microworlds
Various microworlds (Kynigos, 2007) which support constructionist learning in mathematical, scientific and
socio-environmental domains are also integrated in Metafora platform. Students, in order to solve specific
math and science challenges might use one of these microworlds.
Figure 1. Screenshot of Metafora Platform

The research study we present in this paper focuses on the implementation and evaluation of the planning
and reflection tool using a visual language representing the key components and features required for L2L2
and for shared scientific enquiry
3. OBJECTIVES AND RESEARCH QUESTIONS

In our research study we had two main objectives:
1. To understand and specify the Metafora’s potential affordances to promote the learning and reflection
about scientific enquire processes.
2. To study how Metafora’s potential affordances may support students’ development of L2L2 skills.
This study was conducted as design-based research (Wang & Hannafin, 2005) in which our research
questions were:
RQ1: How does the visual language help students to solve the challenge using key scientific processes?
RQ2: How does the visual language stimulate discussion and reflection about scientific processes?
RQ3: Does the visual language help students to develop collaborative learning processes?
141
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. METHOD
4.1 Participants
Eleven secondary students of year 11 participated in our study. The students were the entire class of a
Science subject. Students do not work in groups regularly and they were used to embed technology to support
learning. Students work in 3 groups to solve a challenge-based science project. Students work on the
challenge during 9 class sessions.
The Science teacher responsible of this class participated in the design-based research study as a co-
researcher.
4.2 Procedure
Teacher began introducing the challenge and the visual language to the students. She used the interactive
blackboard.
The challenge was:
The water and environmental European committee has fixed in its normative 2000/60/CE that all
European rivers have to be in good ecological conditions in 2015.
A study of this European committee realized in 2008 found that Segre River (Lleida, Spain) was in good
ecological condition only in 75% of its course. The most polluted section of the river is when the river
crossed the town of Lleida.
What scientific and rigorous proposals could you think about to influence on the society on solving the
rivers’ problem. Your ideas and actions might be at different levels: authorities, media, society and
peers-secondary schools.
Students were provided with some net resources about main causes that pollute more the river. These
resources were selected by the Science teacher. However, students could check the open net.
Afterwards, students planned and solved the challenge using the Metafora planning and reflection tool.
The pedagogy used during these sessions was:
• Students worked in small groups during all sessions.
• Work-in progress presentations and group debate sessions were carried out. During three times
along the workshop, every group presented their working progress. In this presentation, students were asked
to present the work done so far but also the group thinking process: reflect and present their discussions,
problems, how they overcame them, use of visual language, collaboration…
• Final group work presentation and whole class discussion. Every group presented the whole work
and the group proposal to influence on the society on solving the rivers’ problem.
4.3 Data Collection

• We collected all students’ group work realized on the computer and students’ group discussion
during small group work were video-audio-recording using video recorder programme –CAMSTUDIO.
• Video recording sessions of work-in progress presentation and final presentation.
• Video recording of students’ dialogue while working together. For the purposes of this paper, we
have transcribed and analyzed the dialogue of one group.
5. FINDINGS
5.1 How does the Visual Language Help Students to Solve the Challenge Using
Key Scientific Processes?
To answer this research question we analyzed the small group work in the planning tool and their work-in
progress presentations to the whole group class -in which students present what they did, for which purposes,
142
how they were thinking in order to better solve the challenge. All the groups organized their challenge
resolution process around the “activity stage” icons which represented a scientific objective to solve the
challenge.
Analyzing the planning and the icons used by the three groups, we observed students took in
consideration the next five scientific enquiry stages:
• Define the problem
• Hypothesis
• Methodology – Experimental design
• Analysis of results – discussion
• Conclusions and proposals to solve the challenge
These findings show that Metafora planning and reflection tool supported students’ creation of an enquire
process because students establish the main scientific enquire stages highlighted in the literature (e.g.
Hakkarien, 2010; Shimoda, White & Frederiksen, 2002).
Besides, students used the “activity processes” to unpack the processes and actions of the scientific
activity stages. The use of the “activity processes” helped students to better define and fulfill the scientific
objectives of each activity stages.
Furthermore, the analyses of the data showed that “activity processes” icons were mainly used for the
next three purposes: a) as an aid to start thinking in possible actions: brain storming; b) to reflect about what
they did and consequently plan the next step to solve the challenge and c) to organize and structure their
actions. Next we present three excerpts of the dialogue students showed in their discussions about what to do
and how the visual language mediate in their scientific discussions in order to solve the challenge.
Next we will present an example of the use of the “activity processes” visual icons as an aid to start
thinking in possible actions: brain storming.
We all agree we have to start defining the problem. But.. what next? Let’s see what actions we can
do [[student drags some process icons to the screen and all the members started a discussion
about what processes should follow]]
An example of the use of the “activity processes” visual icons as an aid to reflect about what they did and
plan next step is presented bellow:
Ok, let’s see, previous knowledge, and then we observed the data, explored the cartography link
and the water agency link, and then we researched for new information.
…But we don’t have enough I think now we have to obtain new data about the river: look at this
map [[open a link from the web resources]] it’s clickable! It shows the quantity of water of the
river at different points. How much water does it have in the different stages of the river? and in
Lleida? Look We can compare them.
An example of the use of the “activity processes” visual icons as an aid to organize and structure their
actions is presented next:
Ada: I would put all of this in one block: reflect and analyze. All the information we have in here …
Thus, all this information [[pointing at text written in one of the boxes]] is the information we got reading
on the web.
Alan: Yes
Ada: I will put the icons reflect and analyse, because we have already analysed it, haven’t we?
Alan: wait, wait, say it again and I will put the icons
Ada:I try to say what we are doing now?
Alan: Yes, and I agree [[she looks for an icon and drug to the computer screen]]
Ada: Brain storming [[this is the icon that Aln druged]] no, no, this later. We have done the analyses…
Observation of the planning process combined with feedback from students’ in-progress presentations
suggests that “activity stages” and “activity processes” visual icons promoted students to consider aspects of
the scientific research process that they would not thought otherwise. Therefore, the visual language included
in the planning tool enrich students’ scientific enquire processes.
143
ISBN: 978-989-8533-12-8 © 2012 IADIS
5.2 How does the Visual Language Stimulate Discussion and Reflection about
Scientific Processes?
We transcript and analyzed the dialogue of students of one group of students and in one class session. First,
we track in the transcription for words related with the visual language. During this session students
intensively used the words of the visual language in their discussion. In this line, students used 30 times
words related with the “activity stages” but they only put one icon of this category in their planning map.
Students used in their discussion words such as: conceptualize the challenge, methodology, predict the
results, hypothesis, and steps to follow.
Regarding to the visual language referred to “activity processes”, students also included intensively
during their group discussion words related to processes as: analyze, observe, brainstorm, explore, search for
new information, discuss.
From our point of view, this finding is very important because it might confirm that the visual language
had a positive impact on students’ dialogue and on the way students organize their science thinking.
In future research studies, we are intended to use of “text analyses software” as “Wordsmith tools” to
better analyse the use and the impact of visual language on the learning of scientific enquiry processes.
Additionally, a deeper analysis of students’ dialogue showed the presence of students’ reflection about the
most appropriate scientific processes to carry out in order to solve the science challenge. In this dialogue it
can be seen how Metafora visual language promoted and mediated the reflection about scientific process to
solve the task.
Ada: let’s see. Then, when we do that then?

Alan: so, in theory we are still here. We have not done anything, right? ((laughs))
Ada: yeh ... but from this, we should do an experimental design shouldn’t we? Or something.
Alan: if
Ada: This is experimental design, right? [[Looking for experimental design icon]]
Alan: wait, wait, and wait. First is the hypothesis
Ada: We put and if we do better steps to follow first, and observe second
Anna: if ... and reflect as well. Now we are reflecting, aren’t we?
Aln, if also
Ada: thinking...
Alan: here and to reflect put an arrow. So, after everything we've done we look in the mirror. Can I
do it? ... [[Ask for the photocopies of the icons]]
[[Ada recorded in the Freestyler put icons on the last day]]
5.3 Does the Visual Language Help Students To Develop Group Learning
Processes?
In collaborative learning situations, the process of shared meaning making is seen as just as important as the
actual outcome of the activity. In this respect, Mercer and Littleton (2007:25) argue that collaboration
involves “a co-ordinated joint commitment to a shared goal, reciprocity, mutuality and the continual
(re)negotiation of meaning.”
A key concept, related to this idea is the concept of ‘intersubjectivity’, which signifies the process of
developing communality in joint activity. Linell (1998:225) argues that, for collaborative projects to be
successful and truly collaborative, all parties must be ‘mutually other-oriented’. Additionaly, in the context of
Computer-Supported Collaborative Learning, Wegerif (2007:181) claimed that it is necessary to develop,
through social interaction, a “dialogic space”, which he sees as the social realm of the activity within which
people can think and act collectively, thus opening up a space between people in which creative thought and
reflection can occur.
In this section we wondered if the Metafora planning and reflection tool stimulated and mediated the
development of key collaborative processes.
The analyses of the session we transcript showed students shared meaning making, took reciprocal
perspective, students were mutually other-orientated and students created a dialogic space in which they
144
thought and acted collectively. Next we present an excerpt in which collaborative learning processes are
explicit.
The context of the next extract is: students are discussing the different concentration of nitrites and
phosphates in the water in different points of the river. They are analyzing different graphics from a web
resource.
Ada: that’s strange…However, I still do not understand why during the watering season there is
less [referring nitrites]. Maybe because they are more dissolved. I do not know.
Anna: I suppose, because of those about the fields, and how many times you can water the fields,
right?
Ada: yes
Anna: You have to water the fields every 15 days, ok? When you do not have to water is because
the humid is high.
Ada: then, during the watering season, there is less water because the plants absorb it?
Anna: yes. Because the land absorb it. They have that...
Ada: Likewise. So the land, during the watering season absorbs water and in the water is where
are the phosphates and nitrites, so is logical that there are less… and just when there is no
watering ... land does not absorb the water and then the water would pass without any difficulty
and go to the river again.
Alan: good explanation, different to my one… but yes, what you have said is also possible.
Anna: I know this because my uncle has a field, and I know that he waters every 15 days, and for 4
to 5 hours, they put water in the field till the whole field is watered
Alan: yeh… it can be, can be
Ada: yeh, then we can base on this.
Aln: Ok
Ada: with what you are saying DNLA. It is true.
Alan: so if there is no watering, they are not fixed in the land [referring to fertilize] and they go to
the river.
Ada: yes
In this extract students are making explicit their own ideas and arguments and provide reasons,
justifications, warrants and evidences to support one’s opinions. Ada started the dialogue explicitly
expressing a doubt. Anna tried to explain the issue, giving arguments and reasons. Students showed an
explicit effort to try to construct common knowledge which would enable them to make a conclusion. In
doing so, students re-elaborate their own and other’s ideas and reasons.
Students, in this extract, made the effort to provide different kind of arguments, for example, Alan gave
evidence that came from his own experience (his uncle). Besides, students made the effort to agree in the
conclusion.
6. CONCLUSIONS
This paper discusses the affordances of a new learning environment, supported by new technology that is
currently under development -Metafora. Learning how to learn together (L2L2) in science is a key complex
skill or competence for knowledge age work. The Metafora project aims at developing a better understanding
of this complex skill through specifying key features of learning together science processes that students need
to be aware of and able to work with, and by embodying these features in a visual language which forms the
main component of a planning and reflection tool.
We have reported a design-based research study in which the main objectives were to understand and
specify the Metafora’s potential affordances in promoting the learning and reflection about scientific enquire
processes and in supporting students’ development of L2L2 skills.
Findings suggest that the visual language we have developed can help raise students’ awareness of key
collaborative scientific enquiry processes. The Metafora visual language helped students to unpack and
reflect about the scientific processes to solve a complex science challenge. Additionally, the Metafora visual
language promoted students’ awareness about aspects and components of their collaborative learning
processes in science.
145
ISBN: 978-989-8533-12-8 © 2012 IADIS
The development of this visual language and its initial successful trials has potential pedagogy
significance in Science Education. In our study, the tool has shown itself to be of value to science teachers
who need to teach not only the content of science but also the process of scientific enquiry. Students of our
study reported that Metafora helped them to reflect about the nature of scientific methodology and about
scientific inquiry processes followed by the group. The Metafora planning tool allows the representation of a
shared inquiry process. This representation helped students to better understand the scientific methodology
and how to apply it in a specific context.
However, further research is needed to investigate the impact of using this tool on the ability of students
to learn together with others in new situations. This web-based support for groups learning to learn together
has to prove its significance in more disciplinary fields, across educational contexts and with larger studies.
Further research is already planned to explore the potential of the Metafora planning and reflection tool to
support distributed individuals learning together via the web.
ACKNOWLEDGMENTS
This study is funded by EU-R&D project “Metafora - Learning to Learn Together: A visual language for
social orchestration of educational activities” (EC/FP7, Grant agreement: 257872).
REFERENCES
Hakkarainen, K. (2010). "Learning Communities in the Classroom." International Handbook of Psychology in
Education: 177.
Johnson, L. F. & Adams, S., (2011). Challenge-Based Learning: The report from the Implementation project. Austin,
Texas: The New Media Consortium.
Kynigos C. (2007) Half-baked logo microworlds as boundary objects in integrated design. Informatics in education 6
(2),335 – 358. Institute of Mathematics and Informatics, Vilnius
Linell, P. (1998). Approaching dialogue: Talk, interaction and contexts in dialogical perspectives. Amsterdam: John
Benjamins Publishing Co.
Loll, F., Pinkwart, N., Scheuer, O., & McLaren, B.M.: Towards a flexible intelligent tutoring system for argumentation.
In I. Adeo, N. Chen, Kinshuk, D. Sampson, & L. Zaitseva (Eds.), Proceedings of the 9th IEEE International
Conference on Advanced Learning Technologies (ICALT 2009) (p. 647-648). Los Alamitos (CA). (2009).
Minocha, S. & Thomas, P. (2007). Collaborative learning in a Wiki environment: Experiences from a software
engineering course. New Review of Hypermedia and Multimedia, 13 (2), 187-209
Scheuer, O., McLaren, B.M., Loll, F., & Pinkwart, N.: An analysis and feedback infrastructure for argumentation
learning systems. In V. Dimitrova, R. Mizoguchi, B. Du Boulay, & A. Graesser (Eds.), Proceedings of the 14th
International Conference on Artificial Intelligence in Education (AIED-09), (pp. 629-631). IOS Press. (2009).
Shimoda, T.A., White, B.Y. Frederiksen J.R. (2002). Student goal orientation in learning inquiry skills with modifiable
software advisors. Science Education, 86 (2), 244 – 263.
Wang, F. & Hannafin, M. (2005). Design-Based research and Technology-Enhanced Learning Environments. Education,
Technology, Research and Development (ETR&D), 53 (4), 1042-1629.
Wegerif, R. (2007). Dialogic Education and Technology. New York: Springer.
Wegerif, R., Yang, Y., De Laat, M., Pifarre, M., Yiannoutsou, N., Moustaki, F. Smyrnaiou, Z., Daskolia, M., Mavrikis,
M., Geraniou, E. and Abdu, R. (2012). Developing a Planning and Reflection tool to Support Learning to Learn
Together (L2L2). Proceedings of IST-Africa 2012, Dar-es-Salam, Tanzania May 2012.
146
AGORAS: TOWARDS COLLABORATIVE GAME-BASED

LEARNING EXPERIENCES ON SURFACES
Alejandro Catala, Fernando Garcia-Sanjuan, Patricia Pons, Javier Jaen and Jose A. Mocholi
Grupo ISSI, Departamento de Sistemas Informáticos y Computación (DSIC), Universitat Politècnica de València
Camino de Vera s/n, 46022 Valencia, Spain
ABSTRACT
Children nowadays consume and manage lots of interactive digital software. This makes it more interesting and powerful
to use digital technologies and videogames supporting learning experiences. However, in general, current digital
proposals lack of in-situ social interaction supporting natural exchange and discussion of ideas in the course of creative
learning activities. Moreover, learning activities based on games should put the emphasis on the phase of creating
artifacts for the game ecosystem, not on just playing or consuming contents. With the aim of supporting learning
activities focused on “playing to create games”, this paper presents the implemented middleware to support this process
from a technical point of view. The system allows the creation on an interactive tabletop of both single and advanced
entities composed of shapes that can be enacted according to physics principles and rule-based behavior to create game
ecosystems for learning purposes.
KEYWORDS
Game, Learning, Tabletop. Collaboration, Entity, Middleware.
1. INTRODUCTION
Nowadays children and teenagers have access to lots of interactive digital media and software in many
different forms by means of several types of devices. This continuous contact with technologies contributes
to digital literacy and makes them advanced citizens in the digital age. As a consequence, these users will be
the base of skilled users in the future who will be able to deal with more complex and advanced software than
ever.
Digital media and software should be used as a help supporting learning in combination with traditional
materials rather than using the software as an exclusive mean for learning, as reported by McFarlane
(McFarlane, 2002) or as the pedagogical model elaborated in the work by Gros for formal learning settings
(Gros, 2007). In particular, these authors involved commercial videogames taking advantages of the
knowledge that students already have on technologies as digital natives. Videogames are useful in this
respect because they also maintain both motivation and engagement usually high (Michael, 2005), which is
important for learning activities. In addition, they explain that using commercial video-games instead of
software to conduct specific tasks is advantageous for several reasons: they are cheaper than designing and
ordering the development of specific digital serious games; and in spite of the low customization capabilities
of the commercial video-games, given that the pedagogical model can slightly change from one teacher to
another and from one issue to other, the wider range of commercial video-games available gives flexibility in
the purpose of enriching the learning setting. All this means that teachers demand platforms that permit
activity customization for the development of learning sessions rather than a video-game to support already
pre-established learning activities.
This vision is not essentially new, although videogames had not been applied in a generic way for
learning purposes until these studies in real classroom settings. Before the digital age, Clark Abt already
wrote about serious games as tools for learning and the way they should be used (Abt, 1970). Basically, he
considered that games are useful tools for learning when they are played and consumed. However his major
contribution is the idea that they are more powerful when teachers and tutors consider playing to create
games as the primary learning activity and involve students in the process. In his opinion it is therefore more
147
ISBN: 978-989-8533-12-8 © 2012 IADIS
rewarding from a learning point of view because this vision entails design tasks, making up rules, and
producing the materials and the logics behind the game to be created co-operatively rather than just
understanding pre-established game rules.
The perspectives illustrated above suggest that it will be more useful for teachers to have flexible digital
tools available at hand that may be integrated in their respective pedagogical models for formal instruction;
and that creative and social processes as the ones considered before the digital age along with higher
motivation provided by the use of technology are of special interest for learning. Thus, our future long-term
purpose is to provide a framework with these characteristics to be used in learning activities. Teachers would
be able to create incomplete game ecosystems and design several creative learning exercises around them.
These exercises would be addressed by the students in small groups, interacting and collaborating around the
table. The creation of artifacts to complete the ecosystems would support the kind of authorship proposed by
Abt, whereas the resulting game ecosystem could be enacted to test whether it works as expected, triggering
reflection and discussion processes for refining the created artifacts.
The main contribution of this paper is the middleware that has been implemented to support such a
framework from a technical point of view. This middleware allows the creation and enactment of virtual
ecosystems on interactive tabletops composed of 2D entities. An example about a vintage game ecosystem is
presented to illustrate and test the basic middleware functionality. On the implemented infrastructure several
learning activities can be prepared, ranging from storytelling performances to programming physics
simulation.
This paper is organized as follow. Section 2 describes some related work, focusing on those supporting
some kind of artifact creation by children or teenagers. Section 3 introduces the entity-based model to
support the construction of game ecosystems. Section 4 describes the middleware architecture implemented
to support the enactment of game ecosystems. Section 5 describes one of the sample ecosystems that has
been created to test the implemented middleware. Section 6 explains the associated rationale to use the
platform for supporting learning activities. Finally, this paper concludes by summarizing the contribution of
this paper.
2. RELATED WORK
Although there are a variety of proposals presenting software systems to create game ecosystems, in this
section we are interested in works that are more focused on the creation of some kind of entities for learning
activities. LogoBlocks is a graphical programming language to support programming for the LEGO
programmable brick (Begel, 1996). The language uses a drag&drop methaphor in a Windows-Icons-Mouse-
Pointers (WIMP) user interface. The brick is a small computer that can be embedded and used in LEGO
creations by reading from sensors and controlling engine activations. In this way, children and teenagers can
create ecosystems where robots are physical programmable entities.
Scratch is a graphical programming environment that allows children to program interactive stories,
games and animations composed of a set of sprite-based objects (Maloney, 2004). The programming
language to specify behavior is also based on a drag&drop metaphor of virtual blocks representing
instructions. The main screen of the tool shows the stage and the sprite representing the entities, allowing
program debugging, and testing new ideas increasingly and iteratively. The environment is a single-user
application based on WIMP interaction.
Another interesting work is Agentsheets (Repenning, 2000). It is a tool based on agents that allows users
to create simulations and interactive games. Users can create simulations of sprite-based agents in a 2D world
arranged in a rectangular array. The users are responsible for designing the visual aspect of agents by
drawing icons, so that these agents are actually sprite-based entities. Their behavior is based on event-based
rules which are edited following a visual approach to the rewriting rule paradigm.
Topobo is a 3D constructive assembly system that allows the creation of biomorphic forms like animals
and skeletons (Parkes, 2008). This is achieved by means of pieces embedded with kinetic memory. Topobo is
designed to be a user interface that encourages creativity, discovery and learning through active
experimentation with the system. It can help students to learn about several educational concepts on physics
such as balance, center of mass, coordination and relative motion.
148
Finally, another interesting work is ShadowStory (Lu, 2011). It is a storytelling system inspired in
Chinese traditional shadow puppetry. Children use a Tablet PC to create digital animated characters, and then
they are allowed to perform stories on a back-illuminated screen, controlling the characters with simple
movements by means of orientation handheld sensors. Thus, to some extent the activities considered in this
system would support the main aspect of the Abt’s ideas since children are allowed to create the entities and
later use them to perform a public story intended as a learning task.
As seen in the previously described related work, there are many different technologies used to support
the idea of creating some sort of games or interactive media. The difference with regards to our approach is
primarily the social component provided by tabletop technology. It supports more balanced interaction in
smalls groups and natural interaction with fingers and hands. Moreover, the combination of mechanisms for
simulating and enacting ecosystems has not been considered in any of the other approaches. Our work unites
the physics-based simulation along with logics behavior based on rules. It provides a higher level of
customization and possibilities for a wider range of activities.
3. PHYSICALLY-BASED ENTITY MODEL

Aiming at an environment in which the main tasks would focus on the creation of game ecosystems, it is
important to consider the factors that will determine the primary editing units of such ecosystems. The
ground technology being considered is an interactive tabletop. In contrast to other existing technologies, we
have chosen this one because we are interested in fostering collaboration and discussion during the task of
creating the game. This social aspect is relevant as it enhances the learning process by exchanging and
refining ideas between participants. Because this technology supports easily and effectively the sharing of
digital objects on the flat surface (Hornecker, 2008), the characteristics of the game ecosystems to be
elaborated should take advantage of this. Thus the game ecosystem should consider the composition of parts,
being carried out collaboratively, and the interaction techniques should be easy to perform following typical
gestures such as tapping, dragging fingers, etc. Specifically, we have considered a game ecosystem to be a
surface where 2D virtual entities can be located and simulated according to intelligible physics (i.e.
kinematics).
In addition, we have taken two design decisions with respect to the form of the entities that will determine
the capabilities of the model required and the range of alternatives for learning activities in the future. Firstly,
we would like to be able to include concepts such as the instantiation of entities. It would allow creating
instances at runtime under some conditions, reusing preexisting definitions and dealing with abstractions. It
would be what classes are for OO programming languages. These elements are useful to develop the
computational thinking skill (Wing, 2006), considered as a very important one for personal development.
Secondly, although entities could be single bitmaps or sprites, and this would be probably enough to build
most 2D game ecosystems, we are interested in allowing also the definition of entities composed of
architectural units in the same way that a skeleton is composed of bones. In this way, we could also support
the creation of more advanced behaviors and the more interesting learning scenarios based on storytelling
containing puppets.
To fulfill these goals we have defined a meta-model to cope with the visual representation and physical
properties of entities (see Figure 1). Basically, the stage or game ecosystem is composed of embodied
entities. An embodied entity-type (EmbodiedEntityType) is physically defined in terms of its internal
structure (StructureType) and the protocostume (ProtoCostume) covering such a structure. The structure can
be composed of a single component (StructuralComponent) or several components put together by joints
(StructuralJoints) to create more complex structures. These components are simply covered by a visual
representation by means of images (ProtoSkin). Figure 2 shows several examples of entities that can be built
using the meta-model. Essentially, the structure-type is like the internal skeleton that determines the physical
simulation whereas the proto-costume is like costumes or the dresses of the structures used for visual
representation purposes. From these elements, entities are simply instantiated embodied entity-types.
149
ISBN: 978-989-8533-12-8 © 2012 IADIS
EmbodiedEntityType
ProtoCostume ProtoSkin
+PhysicalResponse
* +Name +Rotation
+IsFingerSensitive *
+IsTagSensitive +ZOrder
+Rebound covers +Offset
1
StructuralComponent
1
+IsStatic
* +Rotation Texture
1 +Mass
1
StructureType +Position
+MomentOfInertia
+Name +Size TriangleStructuralComponent
+BaseFrictionCoef
1 0..1
RectangleStructuralComponent
structural_comp1
structural_comp2
EllipseStructuralComponent
*
StructuralJoint
*
* +Visible
PinStructuralJoint
ElasticPinStructuralJoint
RevoluteStructuralJoint SliderStructuralJoint
FixedRevoluteJoint
Figure 1. Meta-model for entity types.
Figure 2. Entity types examples.
4. ARCHITECTURE AND ENACTMENT MODEL

The designed architecture leading the implementation of the middleware to process and enact ecosystems
defined in terms of the meta-model described above is explained in this section. There are three layers in
which the middleware can be broken down (see Figure 3). The Model layer refers to the definition and
specification of the game concepts and their storage. This layer is responsible for managing the concepts
presented in the previous section concerning the stages, entities, rules, etc. available in the game ecosystem
being created, by giving support to update and retrieve all these basic elements.
The Controller layer holds the core functionality to orchestrate the simulation of a world stage. Basically
the simulator has to take a stage to be simulated from the ecosystem, and all the data from the Model layer.
The simulator has an event queue for the event occurrences produced during the simulation. Three types of
events are queued: those thrown by the actions when executed in this layer, those being consequence of the
physical simulation carried out by the underlying physics engine; and those related to the gestures or
interactions of the user on the surface. This queue is regularly consumed by the rule processor, which
determines which rule must be triggered, and eventually performs the execution of the action of the matched
rules. Actions can involve changes in logical entity properties (e.g. increment the variable “Hits” of a block
in a Breakout game) or in visual properties of the game (e.g. change of the visual representation or skin of an
150
entity). The Controller layer uses the services from the Model and View layer to address these two types of
run-time changes respectively. In this way, the simulator controls the evolution of the stage simulation by
consuming events and invoking services on the model or view as needed.
Finally, the View layer is responsible for visualizing the representation of entities under simulation. It
offers a core set of view services that allow us to include entities and change their visual properties from the
Controller layer.
The View layer is by far the most complex layer to implement. In order to perform the visualization of
physics components, this layer relies on the Farseer physics engine1 . It is an open-source physics engine
simulator that allows the simulation of shapes defined internally in terms of bodies, geometries and joints.
Since Farseer does not understand the concepts in which the visual structure of entities is defined (i.e.
StructuralComponents and StructuralJoints), the View layer has to translate the components of the entities in
our model into Farseer primitives. Moreover, it also has to maintain the correspondence between these
elements in Farseer and the model in order to be able to track which entities are eventually producing
physical events such as collisions. In this way, when two shapes collide, the middleware is able to determine
which entities these shapes belong to, and therefore it is able to throw a physics event occurrence associated
to the involved entities that will be queued in the Controller layer for further processing. All the components
of this middleware have been implemented in C# and using the Microsoft Surface SDK.
Figure 3. Architectural logical view of the implemented middleware.
5. MIDDLEWARE VALIDATION: A BREAKOUT GAME

The middleware allows the instantiation of any game ecosystem defined in terms of the model. Tests are
necessary to prove that the concepts and the definitions present in the meta-model along with the
corresponding implemented operationalization are functionally correct. To carry out this kind of validation,
vintage game ecosystems seem good candidates for the test because they are well understood and provide a
fair level of complexity. In particular, a basic Breakout game2 ecosystem is presented here.
In the original game, there are some layers of bricks filling the upper side of the screen, a paddle in the
lower side and bouncing ball. The user has to move left-right the paddle to prevent the ball reaching the
lower side of the screen, and he/she has to get points by breaking bricks when the ball hits them.
For the present use case, we have considered three extended features with respect to the original game:
bricks have a longer life, represented by an integer property “Hits”, which counts how many times the brick
1
Farseer Physics Engine in Codeplex: http://farseerphysics.codeplex.com/
2
Breakout Videogame: http://en.wikipedia.org/wiki/Breakout_%28video_game%29
151
ISBN: 978-989-8533-12-8 © 2012 IADIS
can be hit until destruction; there are two paddles whose scope is limited to a half of the stage intended for
two players; and two players are present, who get points up in their scoreboards when they break bricks after
hitting the ball with their own paddle. Figure 4 shows a picture of a user playing the game ecosystem being
enacted.
Figure 4. User playing the game ecosystem being enacted by the middleware.
The use of this game as a testing example is suitable for several reasons. Firstly, it is simple enough to
perform a meaningful proof. Secondly, it requires multiple instances of the same type (e.g. several bricks)
which could have different property values (e.g. one brick could require 2 hits in order to be destroyed
whereas others could require only 1 hit). Thirdly, the type of rules required to control the evolution of the
game are simple enough as to be intelligible and explained. Overall, it requires that the orchestration
performed by the Controller layer to work properly, combining the physical events (e.g. event Collision
between ball and brick) and logical events (e.g. the property Hits of a brick reaches 0, and as a result it will
have to be destroyed and removed from the on-going simulation).
The screen has been modeled as a stage in our middleware. The main entities to be simulated are the
bricks, the paddle and the ball. In addition, displays for the scoreboards are defined. All of them required
defining the corresponding structure-type and the protocostumes in the Model layer, which specify the
physics behavior and the visual representation of the entities respectively. Erro! A origem da referência
não foi encontrada. illustrates graphically these concepts from the model. Note that two protocostumes have
been defined for bricks since we would like to represent the bricks in a different way according to the value
of the Hits property (e.g. in red those whose Hits property value is set to 1 and green when it is set to 2). The
definition of types, structures and costumes provides the specification of the look and the behavior in terms
of physics. Moreover some rules have been defined in the game ecosystem, which represent the logics of
control governing the gameplay. Although it is not possible to show here all the required rules for space
limitation, Figure 6 shows two sample rules defined in the ecosystem. They illustrate the expressiveness
supported by the rule language implemented in the middleware. On the one hand, Figure 6(a) specifies a rule
that establishes to 1 the property LastPaddleHit of the Ball when the Paddle1 collides against the Ball. It will
be useful to know which scoreboard has to be incremented when the ball hits a brick. On the other hand,
Figure 6(b) specifies that the property Hits of the brick collided by the Ball has to be decreased when the
collision occurs.
EmbodiedEntityType StructureType ProtoCostume

BallType

BrickType

PaddleType

Figure 5. Breakout game modeling.
152
PRIORITY: 10 PRIORITY: 15
IF S: Paddle1 THROWS an EVENT E: Collision IF S: ANY BrickType THROWS an EVENT E: Collision
AND E.SlaveEntity = Ball AND S. Hits > 0 AND E.SlaveEntity=Ball
THEN WITH T: Ball THEN WITH T: ANY BrickType
PERFORM O: T.LastPaddleHit = 1 SO THAT E.Entity = T.Value
PERFORM O: T.Hits = T.Hits ‐1
(a) (b)
Figure 6. Sample rules.
6. ENHANCING LEARNING: PLAYING TO CREATE GAMES

The implemented middleware presented in this paper allows the creation of both simple and complex entities
enacting them in the world ecosystem according to physics principles. Since the system supports not only the
creation of game ecosystems but also the creation and customization of partial ecosystems to be completed,
this platform facilitates a range of activities that can be proposed to enhance learning, following the ideas
discussed by Abt and Gros in their respective works and leading to activities under the vision of “playing to
create games”. It provides enough flexibility for teachers to design sessions and decide which parts should be
already provided and which ones should be created or completed by pupils.
The activities can be oriented towards several core tasks. For example, the system can be used to create
characters for storytelling purposes so that the surface is the stage and anchorage points for entities and
gravity are defined so that entities resemble like flattened puppets to perform the story. Another possible
scenario can be focused on teaching essential physics by creating single entities as punctual masses aiming at
some specific goal and then exploring how the system evolves.
In addition, more complex and complete ecosystems can be devised by including reactive behavior
expressed by means of rules, as shown in the Breakout game example. Besides the specification of the visual
aspect, the rule behavior can be edited by means of a rule editing tool invoked from the world editor. The
implemented editor follows a set of drag& drop interaction techniques for some part of the rules combined
with data-flow expressiveness. Data-flows allow users to edit expressions collaboratively using their fingers
and some tangible objects like pucks for input on the surface without using any keyboard or mouse
peripherals. Each task to complete the definition of a game ecosystem is oriented as a collaborative learning
activity.
To demonstrate the flexibility and usefulness of the system for creativity learning tasks, the system has
been used in two experiments involving teenagers. In these experiments subjects faced problems such as
creating articulated entities (Catala, 2012a) and functional Rube-Goldberg machines (Catala, 2012b). The
activities were considered within a discussion-action-reflection loop so that students could create the
proposals collaboratively and interact between them. The tests showed that tabletop technology maintains
students motivated, helps in sharing digital objects more effectively and that the aforementioned loop
facilitates fairer co-operation interaction patterns, which are positive in terms of social skills development.
Figure 7 shows two subjects interacting to create a structure for their puppet-like entity.
Figure 7. Two users creating an entity.
153
ISBN: 978-989-8533-12-8 © 2012 IADIS
7. CONCLUSION
On the idea of providing a platform for learning activities under the vision of “playing to create games”, this
paper describes the implemented middleware to support such a learning framework from a technical point of
view. It relies on a meta-model for physically-based 2D entities that can be enacted according to physics
principles and rule-based behavior. The system uses an interactive tabletop as ground technology, which
provides several advantages over other existing approaches. It does not only keep motivation in high levels
but also encourages social participation and enables more natural interaction in collaborative tasks.
Our proposal focuses on the creation of several artifacts to create game ecosystems. It is flexible since it
provides a desirable level of customization for teachers, so that they can design the learning sessions as
needed. In this sense, the combination of two enactment principles (physics and rules) introduces even more
possibilities for designing a wider range of activities than those proposals that only use either programmable
simulations or storytelling live performances.
There are also some clear limitations. On the one hand, the use of the tabletop is not for masses but
intended for small groups, so that the discussion and interaction can be properly conducted from a
pedagogical viewpoint. On the other hand, the type of game ecosystems is based only on 2D performances
and simulations, leaving out more advanced interesting scenarios based on 3D concepts. Nevertheless, it is
still a powerful tool for teachers as it allows them to create partial ecosystems for many different learning
scenarios adapted to their needs as discussed in the paper.
ACKNOWLEDGEMENT
This work was funded by the Spanish Ministry of Education under project TSI2010-20488. Our thanks to
Polimedia for the support in computer hardware.
REFERENCES
Abt, C., 1970. Serious Games. Viking Press, New York, USA.
Begel, A., 1996. LogoBlocks: A Graphical Programming Language for Interacting with the World. MIT, Boston, USA.
Catala, A. et al, 2012a. Exploring Direct Communication and Manipulation on Interactive Surfaces to Foster Novelty in a
Creative Learning Environment. In International Journal of Computer Science Research and Application, Vol. 2, No.
1, pp. 15-24, ISSN 2012-9572.
Catala, A. et al, 2012b. Exploring tabletops as an effective tool to foster creativity traits. In Proceedings of the Sixth ACM
International Conference on Tangible, Embedded and Embodied Interaction (TEI '12), Kingston, Canada, pp. 143-
150.
Gros, B., 2007. The Design of Learning Environments Using Videogames in Formal Education. Proceedings of IEEE
International Workshop on Digital Game and Intelligent Toy Enhanced Learning, Jhongli, Taiwan, pp. 19-24.
Hornecker, E. et al, 2008. Collaboration and Interference: Awareness with Mice or Touch Input. Proceedings of the ACM
conference on Computer supported cooperative work, San Diego, USA, pp. 167-176.
Lu, F. et al, 2011. ShadowStory: Creative and Collaborative Digital Storytelling Inspired by Cultural Heritage.
Proceedings of SIGCHI conference on Human factors in computing systems, Vancouver, Canada, pp. 1919-1928.
McFarlane, A. et al, 2002. Report on the educational use of games. Teem: Teachers Evaluating Educational Multimedia,
Cambridge, UK.
Maloney, J. et al, 2004. Scratch: A Sneak Preview. Proceedings of the International Conference on Creating, Connecting
and Collaborating through Computing, Kyoto, Japan, pp. 104-109.
Michael, D. and Chen, S., 2005. Serious Games: Games that Educate, Train, and Inform. Course Technology PTR,
Mason, USA.
Parkes, A.J. et al, 2008. Topobo in the wild: longitudinal evaluations of educators appropriating a tangible interface.
Proceedings of SIGCHI conference on Human factors in computing systems, Florence, Italy, pp, 1129-1138.
Repenning A. et al, 2000. AgentSheets: End-User Programmable Simulations. In Journal of Artificial Societies and
Social Simulation, Vol. 3, No. 3.
Wing, J. M., 2006. Computational Thinking. In Communications of the ACM, Vol. 49, No. 3, pp. 33-35.
154
EXPLORATORY LEARNING THROUGH CRITICAL

INQUIRY: SURVEY OF CRITICAL INQUIRY PROGRAMS
AT MID-SIZED U.S. UNIVERSITIES
Stephanie M. Foote1, David S. Harrison2, C. Michael Ritchie3 and Andrew Dyer4

1
Associate Professor of Education, Department of First-Year Programs, Kennesaw State University, 770.423.6503
2
Professor and Chair in Global Business, University of South Carolina Aiken, 803.641.3376
3
Professor of Management, University of South Carolina Aiken, 803.641.3228
4
Associate Professor & Director of Critical Inquiry Program, University of South Carolina Aiken, 803.641.3443
ABSTRACT
Many alternatives exist for setting the delivery, content, direction, tone, and priorities for a Critical Inquiry/Thinking
general education program. Review of our university’s overall general education program indicated the need, and
overwhelming faculty approval, for a program to improve critical thinking skills, to specifically include exploratory
learning and improved cognitive abilities. As we moved ahead we formed a Critical Inquiry Committee for in depth, and
participative planning. One of the preliminary, investigative steps was to survey peer institution critical inquiry
programs. The results of the study are reported below, along with our steps in developing our first year, one credit,
general education critical inquiry program. Our paper includes details of our faculty development program and the results
of our pilot program implementation. Our full program is scheduled for implementation during fall 2012.
KEYWORDS
Critical inquiry/thinking, exploratory learning, analytic thinking, cognition, general education, curriculum
1. INTRODUCTION ~ IS THAT GOING TO BE ON THE TEST?

Faculty members are aware of the importance of critical inquiry (CI) and critical thinking CT), however,
these skills are seldom specifically addressed directly in courses. Further, critical thinking is generally
assumed to be a byproduct of secondary education, rather than an end in itself. While critical thinking skills,
exploratory learning methods, and cognitive abilities are considered essential to higher education success,
and prerequisite to satisfactory course progression, these skills are not often directly addressed either before
or during college. Even in the more objective and quantitative disciplines, such as math and science, the
inquiry, exploration, questioning, and decision processes (critical inquiry) are ordinarily secondary to
achieving the “correct” answers. (Sadly, with the recent, absolute prioritization of K-12 test scores, learning
to think may well be even more sidelined in favor of “answer-driven test outcomes.”) In higher education,
the trend continues. Students learn, all too frequently, what to do, rather than why they are doing it. We feed
the “is that going to be on the test mentality.” In spite of all the pedagogical discourse to the contrary,
students continue to learn their outcomes, learning them in the traditional, non-integrated “silos” of teaching
disciplines, seeking the test answers -- rather than the questions. Change is difficult.
Over the last decade, the American education process, especially higher education, has focused on content
areas of concentration that emphasize technical and memory based skills (Arumand and Roksa). The absence
of training on critical thinking, along with analytical ability, is ever present. Emphasis on test scores for both
undergraduate and graduate admissions are at an all time high. In fact, educators and institutions are
evaluated on test scores of both incoming and graduating students (Resmovitz). The idea of teaching critical
thinking has gained popularity in recent years (Mulnix). It was often believed that students were exposed to
critical thinking philosophies during their primary and secondary education programs.
Notwithstanding faculty members repeated complaints that students are not adequately prepared for
success in their courses, little attention is paid to the skill of thinking, Socratic dialogue, or exploratory
learning. Because we cannot readily change the st5udent preparedness, the most logical approach should be
155
ISBN: 978-989-8533-12-8 © 2012 IADIS
to consider change in our own pedagogical approaches. For example, Clabaugh, Forbes, and Clabaugh
suggest that students will demonstrate greater gains in critical thinking skills when the learning objectives for
specific courses are specifically developed toward building these skill; perhaps obvious, yes – but is it done?
It may be that small steps, specific activities and assignments directed solely at cognitive development can
combine to make significant progress to improve thinking and exploratory skills. More pervasive
alternatives, entire programs focused solely on critical inquiry, represent are another avenue to effect
improvement.
Our university, a small undergraduate institution in the Southeastern United States, determined that the
development of CI skills for entering freshmen should be the cornerstone of our Quality Enhancement Plan
(QEP), which was undertaken in preparation for a forthcoming accreditation review. We felt that a
successful critical inquiry course would enhance student thinking skills and their interest in inquiry and
learning. A welcome accompanying plus was that faculty involved in the CI program would naturally
advance their own CI skills, and perhaps more importantly, their ability to bring CI skill development to their
existing, traditional classes. Win, win… Creating a solid faculty development program to prepare
[eventually all] faculty to teach CI courses became the cornerstone of our CI implementation strategy. Better
trained CI faculty, across all disciplines would foster and cultivate continued critical inquiry and thinking
skills throughout the all levels of our curriculum.
There are many options to set the direction, content, tone, priorities, and activities for a viable CI
program. This can range from a dedicated CI course, sequential courses, discipline embedded CI segments,
integrated cross-discipline approaches, and step approaches. Following the CI model, with exploration as the
key foundational step, we felt learning about the types of CI programs in use would be valuable in designing
our program. This would guide us in discerning among programs to find the greatest likelihood of success.
In connection with the planning and implementation of our QEP, we decided to survey CI practices at peer
institutions, including the perceptions of administrators on successful CI efforts. The purpose of our study
was to: (1) determine the types of CI programs currently offered, (2) find out how CI is addressed within
various curricula, and (3) understand the sometimes conflicting views on which types of programs are likely
to have the greatest success. Our study gives a summary of CI practices at other institutions, presents our
experiences in creating our own CI program, including the pivotal CI faculty development program, and
concludes with lessons learned as we implemented our pilot program in 2011-12.
2. CRITICAL INQUIRY, SURVEY & COURSE IMPLEMENTATION

Formal and informal faculty/student interactions reinforce retention and academic performance; this is
especially important in the first year of college. Faculty teaching gateway courses (e.g., English, math, etc.)
if trained more systematically in CI methods hold a key position to make headway in critical thinking skills.
This should have synergistic advantages in furthering the CI agenda, at the same time improving delivery,
understanding, and retention of gateway course material. Getting faculty involved in the delivery of an early
CI course develops faculty in these areas, and carries on to additional opportunities for further student/faculty
interactions as students progress through college.
Derek Bok states that “Many investigators have found that critical thinking and learning in general can be
enhanced by giving students problems and having them teach each other by working together in groups.
Where these conditions exist, the great majority of studies show that students make much greater gains over
those achieved by classmates studying individually or competing with one another.” USCA student focus
group data, collected in spring 2010 by members of the QEP committee, revealed that students found classes
that involved discussion, hands on learning, or problem solving to be most memorable. Several studies
demonstrate the positive impact this type of course instruction can have, not only on first-year students, but
on faculty efficacy. USCA instructors have repeatedly reported being able to transfer active and engaging
teaching strategies from a first-year seminar to other courses. This “spill over,” applying CI teaching
strategies in an active learning environment, to other courses is one of the desired program outcomes.
2.1 Survey Results

We supplemented our CI pedagogy research by doing our own survey of CI practices at peer institutions. We
had strong connections with deans of southeast U.S. business schools, and decided to avail ourselves of that
156
resource, surveying business schools rather than a broader institutional approach. As general education is
typically a very significant portion of business school curricula, we felt our results would apply to our larger
range of interest. We conducted an online survey, from the USCA School of Business Dean to Business
School Deans of 80 institutions, which our Office of Institutional Effectiveness felt constituted a
representative comparison group. Although we wanted more information, we held the survey to 25 objective
and one open-ended question. We received 35 responses, 44%, which seemed a favorable response rate.
When we first began to consider the incorporation of a CI component to the curriculum, part of the
underlying impetus was widespread view among faculty that freshmen students were not adequately prepared
for their transition to the academic challenges ahead. To test this assumption among our peer institutions,
respondents were asked to indicate their level of agreement with the statement, “freshmen are not adequately
prepared for success in their courses.” The results are shown in Table 1.
Table 1. Freshman Preparedness
Survey Question: “Often educators hear that freshmen are not adequately
prepared for success in their courses. In your opinion, does this statement
apply to your entering first-year students?”
Answer Options Responses Response %
Strongly Disagree 0 0 %
Disagree 11 31 %
Agree 16 46 %
Strongly Agree 8 23 %
Total Responses: 35 < < > > Skipped Question: 0
Almost 70% of respondents agreed or strongly agreed with the statement, lending support to the notion
that freshmen students require additional training to enhance their preparedness for college. Beginning in fall
2011 we started a pilot CI program, a one-credit hour course, focused on themes and ideas presented in the
book The Last Town on Earth, (Thomas Mullen) our freshman first-year reading. Further support for our
approach is shown on the results in Table 2. Here almost 80% of respondents agreed or strongly agreed that
a one-credit hour critical thinking methods course would be useful in helping students achieve better
curricular success.
Table 2. Value of a One Credit Hour Critical Inquiry Course
Survey Question: “Do you think a short, one credit hour critical inquiry
methods course for the first-year students would help them achieve better
curricular success?”
Strongly Disagree 2 6 %
Disagree 5 15 %
Agree 20 58 %
Strongly Agree 7 21 %
Although the respondents indicated support for the inclusion of a one-credit hour freshman level CI
course, this method alone was not viewed as the single best approach to help students effectively learn
critical thinking skills. Instead, as shown in Table 3, respondents indicated the greatest amount of support for
a program that would incorporate critical thinking skills within the curriculum as a whole. For example, one
respondent likened the approach of teaching critical thinking to that of teaching writing and speaking (across
the curriculum), but acknowledged that this could make assessing student learning in this area more
challenging. Another respondent indicated that faculty on his/her campus wanted to embed critical thinking
into existing courses, but he/she wondered how faculty could integrate these efforts in a way that would
result in students developing these skills over time.
157
ISBN: 978-989-8533-12-8 © 2012 IADIS
Table 3. Critical Thinking Skill Development Alternatives

Survey Question: “In an ideal situation, how would students best learn
critical thinking skills?” Rank the following alternative options.
Better than Less than
Answer Options Best Acceptable Worst
acceptable acceptable
Specifically designed CT course 8 3 11 10 2
Specific areas within curriculum,
15 12 4 2 0
all four years
Specific areas integrated within
3 14 14 2 0
the major
Specific areas integrated within
4 8 10 9 2
general education
Through extra‐curricular activities 1 0 6 6 19
Respondents appeared to be supportive of a program designed to integrate critical thinking skills within
the curriculum, throughout all undergraduate four years, but in practice this did not appear to be taking place.
As shown in Table 4, only 20% of respondents indicated that critical thinking skills are actually integrated
within the curriculum for all four years in their respective institutions. Rather, over 42% of respondents
indicated that although their institutions do not have any specific CI segments in their curricula, the
development of critical thinking skills is inherently part of many courses.
Table 4. Critical Inquiry Curricular Approaches in Use
Survey Question: “Which of the following does your institution do to
develop critical thinking skills in your students?” (If more than one fits,
choose the most influential.)
Through a specific critical inquiry course 3 9 %
Specific segments integrated within the
7 20 %
curriculum, all four years
Specific segments integrated within the
2 6 %
major
Specific segments integrated within
3 9 %
general education
No specific program segments, but
15 42 %
inherently part of many courses
Through extra‐curricular programs,
0 0 %
projects and other activities
Other 1 3 %
No specific critical inquiry program 4 11 %
Part of this reluctance to specifically designate any single course as a critical thinking course may be
based on the perceived difficulty of actually measuring CI related outcomes and relying on these measures to
evaluate student performance. As shown in Table 5, over 45% of respondents do not fully believe that critical
thinking can be objectively assessed, or indicate skepticism about the ability of institutions to assess critical
thinking with traditional standardized multiple choice tests.
158
Table 5. Critical Thinking Assessment

Survey Question: “Do you believe critical thinking can be objectively assessed?”
No 3 9 %
Unsure 5 14 %
Yes, with standardized
8 23 %
multiple choice tests
Yes, somewhat agree 9 26 %
Yes, strongly agree 10 28 %
Nevertheless, an intrepid few have actually attempted to surmount the perceived hurdles and offer
students at least one critical thinking course. As shown in Figure 1, nine respondents (just over 25%) indicate
that their school or institution had a dedicated critical thinking course in their curriculum.
"Does Your institution has a critical thinking
course?"
YES
26% Unsure
YES
Unsure
8% NO NO
66%
Figure 1. Presence of Critical Thinking Course

While the focus of any CI course should be on critical inquiry skill development, as we have argued, an
important factor in the success of the program was tied to faculty development. Consistent with the
prevailing view on the preferred method of CI instruction (see Table 3), our goal is that critical inquiry skills
will ultimately be integrated and assimilated throughout all four years of university study. This goal will be
achieved in three ways: (1) as a natural outcome of the freshman CI course. If successful, CI skills will be
carried through, by the students, into other classes (and, indeed, their lives); (2) specific other courses, across
all majors and years, will be developed by interested faculty that specifically focus on furthering CI skills;
and (3) full faculty participation in teaching the freshman CI course. A natural outcome of their training and
development will result in furthering this process in courses they regularly teach. It is both interesting, and
appealing, to note that two of these three results are achieved somewhat indirectly. That is to say that as a
result of planned CI implementation, both students and faculty will be trained in the process (items 1 & 3),
with expected and desired extension, ideally, to all other classes.
The importance of faculty development is illustrated in Table 6. Although sample size for this question is
limited to only eight respondents, only one of those respondents strongly believed that faculty in their
institution who teach the critical thinking course receive adequate training and support. For the other
respondents, the subject of faculty development was noted as the “missing link.” One respondent discussed
how the lack of faculty development on his/her campus has resulted in professors “conveying their thoughts
not developing thoughtful students.” Further, he/she said, “Only a serious effort will change that model.”
159
ISBN: 978-989-8533-12-8 © 2012 IADIS
Table 6. Faculty Training & Support

Survey Question: “In your opinion, do faculty in your institution who teach
a critical thinking course receive adequate training and support?”
Strongly Disagree 1 12.5 %
Disagree 2 25 %
Agree 4 50 %
Strongly Agree 1 12.5 %
Total Responses: 8 < < > > Skipped Question: 27 (n/a)
2.2 Faculty Development

At USCA, faculty development plans centered on an intensive Critical Inquiry Workshop and a Faculty
Learning Community. After the workshop experience, faculty would be more knowledgeable about the
concepts and learning theory underlying critical inquiry instruction. The Faculty Learning Community would
be continuously available for consultations, and have planned full membership meetings to discuss successes
and failures, and plans for improvement. This would also provide faculty with access to teaching materials
and practical applications for developing critical thinking skills in students, which would be accumulated and
summarized within the learning community. Specific “Faculty Learning Outcomes” follow.
Faculty Learning Outcomes Æ During the workshop, the instructor will:
1. Define Critical Inquiry,
2. Create a sample course outline/timeline for leading the class,
3. Gain access to a variety of course materials (syllabi, assignments, grading rubrics, technology),
4. Develop strategies for including CI in other discipline-specific courses,
5. Create a plan for incorporating Peer Mentors as an integral part of the instructional strategy.
After the workshop, the instructor will be able to:
1. List the steps necessary for practicing critical inquiry,
2. Use a variety of activities for fostering critical thinking and collaborative learning in students
(e.g., discipline-specific strategies, group activities, and active-learning techniques),
3. Evaluate the effectiveness of the activities for fostering critical inquiry,
4. Integrate the First Year Reading Experience (FYRE) into the course structure,
5. Integrate information literacy into the course structure,
6. Encourage students to appreciate multiple perspectives.
Incentives to attract faculty to teach CI courses are varied, with economic incentive seemingly an
attractive option. In our survey, however, we did not find that direct payment was a widespread approach.
Only one institution of the eight respondents provided additional pay. The seven others factored the CI
course into the regular faculty teaching load. Other compensation options had zero responses. We decided
on partial course relief. Faculty teach two one hour sections, and receive credit for three hours of instruction.
We also paid for faculty attendance at a development seminar ($750), but not for teaching the course. We
also gave CI faculty priority scheduling for their CI course, and, importantly, priority by their unit heads in
scheduling their non-CI courses. We considered offering faculty the option to teach their two one-hour CI
courses concurrently, either at the beginning of the term or at the end of the term, giving them half a semester
of additional non-course time. Ultimately we decided the CI courses should run the entire semester, but we
are leaving that option open for future consideration.
Faculty workshops targeted the skills to help faculty members deliver course elements to achieve our
desired Student Learning Outcomes (SLO). The workshops provided [suggested] daily plans, resources,
activities, and teaching methods. Syllabus construction was addressed, including rubric options tied to the
SLO’s. Additional topics included using technology and social networking as part of the CI experience.
Regardless of the specific CI course direction, a constant emphasis was placed on how to integrate critical
inquiry into subsequent non-CI specific classes. One way to help promote the integration of critical inquiry
in the post-CI courses was to plan further workshops on group methods and group project design.
160
Another important element in creating and implementing a successful CI initiative is broad faculty
involvement. To that end, every effort was made to involve the entire USCA faculty in the CI courses.
Ideally, all full-time faculty, all disciplines, would eventually teach a CI course. Unit heads were encouraged
to promote the program and participation, with regular CI faculty rotations. As an added incentive, we
decided to avoid detailed course prescriptives, giving faculty the freedom to utilize creative and singular
approaches in their CI courses, using their own, individualized teaching philosophies and styles.
2.3 Pilot Program Implementation

We held a three-day CI workshop for an initial cohort of 34 CI faculty. A critical thinking and teaching
strategies expert led two days of activities and theory development, followed by a day of course material
preparation. Later a common course model was integrated into a sample syllabus and distributed to the
faculty. During the fall 2011 pilot the instructors for the CI course met three times as a group to discuss and
review their experiences and opinions concerning the course. At the end of the semester, faculty discussions
focused on positive aspects of the course, areas in need of review, change, or elimination. The faculty was
uniformly positive about the design and objectives of the course. Many felt they would like additional
experience with the delivery of certain critical thinking techniques, and stated they would incorporate some
critical inquiry techniques into their other university courses. To that end, mission accomplished!
Some teaching techniques worked very well. Activities such as two-sided debates, role playing, and
exercises where students are faced with moral decision-making were particularly popular. All of these
activities were more likely to be successful when students worked in small groups that had to reach
consensus on decisions. The faculty using student peer mentors as assistants in the course were enthusiastic
about the peer mentor contributions. Faculty also identified a number of teaching techniques that should be
discouraged because they prevent collaborative effort and prevent active participation by the entire class.
Among those techniques were traditional lectures and open-ended questions addressed to the entire class. A
sampling of 165 student reflective essays (a requirement of the course) revealed strong support for the class
overall. A large number of students felt the critical thinking exercises helped them gain greater appreciation
for different perspectives. A common comment was appreciation for the library literacy exercise because it
opened their eyes to new ways to conduct class research. They also commented that the in-depth discussions
on topics from the First Year Reading book were novel and enjoyable experiences and provided them with
new ways to consider the reading of a book. Overall, it appears that the content of the course was well
received by the students.
Our critical inquiry class highlighted student questioning, data gathering, and analytical evaluation. Many
of our students had never encountered a learning environment that asked for and embraced their opinion.
Ground rules for the class were clear: everyone has an opinion; an opinion grounded in research; all opinions
are respected. Creative approached were encouraged. For example the first class assignments was for groups
to interpret photographs, such as a man standing with a black case. The initial attempt at this exercise resulted
in comments such as “it’s just a guy waiting for a bus”. After each group gave their response, they were
asked to change groups and then re-evaluate the pictures. The next evaluations resulted in comments such as
“he is a spy and there are secret documents in the case”. The class then discussed the ideas of objectivity and
subjectivity and finished with a review of validity and truth. These are serious concepts for individuals that
have never been exposed to this type of inquiry. The class was also exceptional in that it gave a cohort group
the chance to explore together. Most of the class time was spent on discussion and lively debate, which arose
spontaneously from class room discussion. It took two or three weeks for the students to feel comfortable and
to trust themselves, fellow students and instructor with this new process. A key facet of the course was to
teach them that they are “allowed” to be creative and search for new and different ideas.
Faculty learning community sessions yielded valuable feedback. Instructors were positive about the
course, but recognized that critical thinking improvement were not likely to be observed in the short term.
The course was considered valuable in opening new learning activities and pedagogical styles for the
instructors. Most instructors commented that they would use the same techniques in other courses. For
example, one instructor learned more of the merits of course structure vs. open dialogue. On reflection he
felt his typical content-laden lectures were less productive than he thought. He planned for more of the
activity-based, group learning, with high outcome accountability for future classes. Others agreed and felt
this type of learning was well received by the students, with the added value of better student/teacher
interactions.
161
ISBN: 978-989-8533-12-8 © 2012 IADIS
3. CONCLUSION
Survey results indicated that there is widespread concern about the level of preparation exhibited by
incoming freshmen. Although most respondents believed that the integration of CI throughout all four years
of undergraduate education was optimal for CI development, there is also strong support for the incorporation
of a dedicated CI course in the first year curriculum. Most institutions surveyed did not have such a course.
Part of the reluctance to offer such a course may be related to the perceived difficulty in assessing critical
thinking skills using traditional classroom methods. There also appears to be inadequate training and support
for faculty assigned to CI courses. The proposed CI faculty development program described for USCA was
designed to alleviate some of these obstacles. Only through implementation and ongoing assessment can we
judge the quality and effectiveness of the CI program described. Our critical inquiry course will hopefully
set in motion a lifelong philosophy of student discovery and analysis. The philosophy must be incorporated
in the entirety of the educational process, with each new critical inquiry experience adding to the last.
After the pilot year of the CI program, instructor morale was very high and student feedback was positive.
Instructors were able to experiment with flexible teaching methodologies and approaches, and most
commented that their experiences would positively affect the other courses they teach. Student feedback
indicated their appreciation of the focus on critical thinking and the introduction to new learning skills.
Student also tended to favor the same teaching methodologies favored by the instructors. The effectiveness
of the CI course at USCA can best be judged when each cohort of CI students is assessed at graduation. We
will be use freshmen-senior assessment (ETS-PP testing) to those ends. Quantifiable answers, however, may
remain elusive, as isolating the singular effects of a freshman CI course four years later is surely confounded
by multiple factors. We firmly believe, however, that the introduction to critical inquiry as freshmen will
have lasting effects, especially on under-prepared students. By introducing freshmen to critical inquiry and
sound exploratory learning practices, and by interjecting critical inquiry methods throughout the curriculum,
we believe senior students will be more intellectually capable and inquisitive at graduation. We remain
convinced that the development of a critical inquiry program for freshmen is in the best interests of the better
undergraduate experience we all seek.
REFERENCES
Arum, R. And Roksa, J. 2011. Academically Adrift: Limited Learning on Campus. University of Chicago Press.
Bok, Derek. 2006. Our Underachieving Colleges: A Candid Look at How Much Students Learn and Why They Should Be
Learning More. Princeton University Press, Princeton, NJ, USA.
Braxton, James R. et al, 2000. Faculty Teaching Skills and Their Influence on the College Student Departure Process.
Journal of College Student Development, Vol. 41, No. 2, pp 215-227.
Claubaugh, Maurice G., Jr., et al, 1995. Bloom’s Cognitive Domains Theory: A Basis for Developing Higher Levels of
Critical Thinking in Reconstructing a Selling Course. Journal of Marketing Education, Vol. 17, No. 3, pp 25-34.
Conley, David T, 2007. Redefining College Readiness. Educational Policy Improvement Center, Eugene, Oregon, USA.
Donahue, Lee. 2004. Connections and Reflections: Creating a Positive Learning Environment for FIrst-Year Students,
Journal of The First-Year Experience, Vol. 16, No. 1, pp 77-100.
Dyer, Andrew et al. 2010. Quality Enhancement Plan: Think DEEP! Aiken, SC: University of South Carolina Aiken.
Fidler, Paul P., et al, 1999. Teaching the Freshman Seminar: Its Effectiveness in Promoting Faculty Development.
Journal of The First-Year Experience and Students in Transition, Vol. 11, No. 2,pp 59-74.
McClure, Amy I. et al, 2008. Transferring Teaching Skills: Faculty Development Effects From a First-Year Inquiry
Program. Journal of The First-Year Experience and Students in Transition, Vol. 20, No. 1, pp 31-52.
Mullen, Thomas. 2006. The Last Town on Earth. Random House, New York, NY, USA.
Mulnix, J.W. 2012. Thinking Critically about Critical Thinking. Educational Philosophy & Theory, 44(5), 464-479.
Pascarella, Earnest T. 1980. Student-Faculty Contact and College Outcomes. Review of Educational Research, Vol. 50,
No. 4, pp 545-595.
Reason, Robert D. et al, 2006. First Things First: Developing Academic Competence in the First Year of College.
Research in Higher Education, Vol. 47, No. 2 pp 149-175.
Resmovitz, J. 2012. Test Scores Often Misused in Policy Decisions. Huff Post, August 6.
Smith, Gerald F. 2003. Beyond Critical Thinking and Decision Making: Teaching Business Students How to Think.
Journal of Management Education, Vol. 27, No. 1, pp 24-51.
Terenzini, Patrick T. et al, 1995. Academic and Out-of-Class Influences on Students’ Intellectual Orientations. The
Review of Higher Education, Vol. 19, No. 1, pp 23-44.
162
A MOBILE LOCATION-BASED SITUATED LEARNING

FRAMEWORK FOR SUPPORTING CRITICAL THINKING
– A REQUIREMENTS ANALYSIS STUDY
Abeer Alnuaim, Praminda Caleb-Solly and Christine Perry

University of the West of England, Bristol, UK
ABSTRACT
This paper presents the requirements work carried out as part of developing an intervention to improve students’ critical
thinking skills using location-based mobile learning. The research emerged from seeking to identify ways of getting
Interaction Design students into real world environments, similar to those in which they will eventually be designing,
maximising their ability to identify opportunities for innovation.
The first stage in designing the system is to conduct a comprehensive requirements study to understand specific student
and staff needs in the envisaged scenario. As part of the requirements study we were interested in identifying weaknesses
in the current mode of teaching, as well as problems students currently experience in understanding key concepts. We
found that students did not stay adequately focused on the task and failed to identify appropriate technologies. We also
noted scope for further support with analysing their findings. This information is helping us to determine the type and
nature of location-based hints and formative feedback that the system can provide to aid students’ understanding of the
context they are in.
KEYWORDS
Location-based, mobile learning, critical thinking, requirements analysis
1. INTRODUCTION
Since the early days of mobile learning research, there has been a wide debate on its exact definition. Various
researchers have different interpretations. Many are eager to show that m-learning is not a reduced version of
e-learning (Belshaw, 2011). Vavoula et al. (2004) have defined it as “Any sort of learning that happens when
the learner is not at a fixed, predetermined location, or learning that happens when the learner takes
advantage of learning opportunities offered by mobile technologies”. Furthermore, Woodill (2011)
acknowledges that there is a shift in the perception of mobile learning, “Ten years ago, mobile learning was
about displaying e-learning on a small screen”. He argues that it allows learners to learn in an “anywhere,
anytime” manner and to access information when needed.
The strength of mobile learning lies in taking advantage of the rapidly evolving scope of mobile
technologies. Being able to sense the context and location of the learner has opened up a wide range of
possibilities for researchers to create more engaging, contextualised and personalised learning activities,
maximising the benefit of the learning experience. Personalisation is one of mobile learning’s strengths.
According to Kinshuk et al. (2009), personalisation could be acquired either by adapting to the learner’s
characteristics, learning styles, performance, and needs; or by adapting to the context in which the learning is
taking place.
This paper presents the requirements work carried out as part of developing an intervention to improve
students’ critical thinking skills using location-based mobile learning. The idea for the research emerged
from seeking to identify ways of getting Interaction Design students into real world environments, similar to
those in which they will eventually be designing, in order to enhance their ability to identify opportunities for
innovation. However, sending students out into real-world environments with a brief to be evaluative and
analytical, without the presence of a teacher, can lead to a superficial and frustrating experience, especially
for students with beginning levels of analysis and limited critical thinking skills. It is not always possible for
163
ISBN: 978-989-8533-12-8 © 2012 IADIS
teachers to accompany students, and moreover, rather than immediate input from teachers, prompts to
provoke the development of their own thinking might be more beneficial.
In order to design the system, the first stage is to conduct a comprehensive requirements study to
understand specific student and staff needs in the envisaged scenario. As part of this study we were interested
in identifying weaknesses in the current mode of teaching and problems students experience in understanding
key concepts. This information is helping us to determine the type and nature of location-based hints and
formative feedback that the system can provide to aid students’ understanding of the context they are in. It is
important to ensure that students don’t miss out on key areas that may help them with analysing the situation
properly. The hints can also give them the beginning of the threads to developing innovative ideas, thus
providing added value to the mobile-based learning system.
The next section of the paper draws on the literature to explain context-aware and location-based
learning, and the relevance of situated learning and critical thinking to this study, summarising the related
work in the area. This is followed by an outline of the research methodology detailing the requirements
gathering process and insights gained and explaining how these have been incorporated into the initial
prototype designs of the application.
2. LITERATURE REVIEW
2.1 Context-Aware and Location-Based Mobile Learning

Context-aware computing is a rapidly growing research area. It aims to promote a flowing interaction
between humans and technology (Barkhuus and Dey, 2003), collecting information from the surroundings of
the user to provide an understanding of what is currently happening (Naismith et al., 2004).
Abowd et al. (1999) defines a context-aware system as follows: “A system is context-aware if it uses
context to provide relevant information and/or services to the user, where relevancy depends on the user’s
task”. Context, according to Brown et al.’s (2010), is “the formal or informal setting in which a situation
occurs; it can include many aspects or dimensions, such as location, time (year/month/day), personal and
social activity, resources, and goals and task structures of groups and individuals”.
Barkhuus and Dey (2003) define three levels of context-aware applications depending on the interactivity
with the user:
1. Personalisation: the user determines the way the application behaves in a particular situation.
2. Active context-aware: the application changes the content independently, based on the sensor data.
3. Passive context-aware: the application presents the changed context, sensor data, to the user and
lets him/her take control of decisions about how the application behaves.
Much research has shown the significance of context-awareness in education (Yau and Joy, 2009; Fisher
et al., 2009; Ghiani et al., 2009; Bhaskar and Govindarajulu, 2010; Chiou et al., 2010). Fisher et al. (2009)
argue that the use of mobile devices such as tablet PCs in education can enhance the teaching experience of
lecturers, as well as the quality of the learning experience of students. Shih et al.’s (2010) research indicates
that using mobile learning helped lower the cognitive load of students with low achievement rates.
A large area of context-aware mobile learning research has been focussing on museums and tours in
providing information based on the person’s location (Reynolds et al., 2010; Chiou et al., 2010; Yatani et al.,
2004; Costabile et al., 2008; Park et al., 2007; Hsu and Liao, 2011). According to Reynolds et al. (2010)
many students appreciated the contextual information offered by the mobile device which encouraged them
to ask more questions. This enhanced their knowledge about the objects in the museum.
From a pedagogical perspective, context-aware and location-based mobile learning is clearly related to
the situated learning theory: it is important to gain an understanding of this in order to learn how to optimise
the development and implementation of context-aware and location-based applications.
2.2 Situated Learning

Lave and Wenger’s (1991) situated leaning paradigm, states that the situation in which learning occurs has a
great effect on learners. They argue that learning must not be abstract and out of context. It is situated, and
takes place within the context, activity, and culture in which it occurs as a “legitimate peripheral
164
participation” process. Lave and Wenger (1991) emphasise social communication and interaction as being
significant parts of situated learning. Learning should be presented in an authentic setting supporting
knowledge exchange between learners (Naismith et al., 2004).
Defining the key characteristics of situated learning can differ between disciplines and technologies
(Yusoff et al., 2010). When designing situated learning using mixed reality technology, Yusoff et al. (2010)
outline three main elements: authentic context, authentic activity/task, and users’ collaboration. Lunce
(2006), in designing situated learning using simulation, suggests four criteria for situated learning: a specific
context that impacts learning must be defined; peer-based interactions and collaboration between students
must take place; knowledge is tacit; and tools must be used to accomplish real-time objectives.
Herrington et al. (2000) propose the following elements for situated online-learning using multimedia:
“authentic contexts and activities, access to expert performances and the modelling of processes, multiple
roles and perspectives, collaborative construction of knowledge, coaching and scaffolding, reflection to
enable abstractions to be formed, articulation to enable tacit knowledge to be made explicit, and integrated
authentic assessment”.
In summary there is agreement that although technologies differ, for a successful learning experience,
situated learning has to take place in an authentic setting, with authentic contexts and activities. Therefore it
is vital that this research is aligned to, and integrated into real teaching and learning scenarios to ensure
validity. Additionally, facilitating collaboration between learners can be an important enhancement of the
learning experience.
2.3 Critical Thinking

As this research proposes to encourage and develop students’ critical thinking and analysis, it is important to
define what this means. There are several relevant definitions of critical thinking, some as early as Dewey
(1933). However, for the purposes of this study one definition has been identified, that of Scriven and Paul
(1987), who defined it as "the intellectually disciplined process of actively and skillfully conceptualizing,
applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation,
experience, reflection, reasoning, or communication, as a guide to belief and action". Their definition shows
a clear relation to Bloom’s taxonomy, as it relates critical thinking to the three higher levels of the taxonomy
(analysis, synthesis, and evaluation) (Duron et al., 2006). This definition emphasizes the multifaceted nature
of critical thinking, expressed through a number of activities. These activities correspond to the assessed
work carried by students in this study explained in section 3.1
3. THE SITUATED LEARNING PROBLEM DOMAIN

This intervention aims to resolve issues faced by students when learning in a real-world situation. The initial
situated learning activity is being developed for a level 2 Human-Computer Interaction (HCI) module in the
Department of Computer Science and Creative Technologies at the University of the West of England
(UWE), Bristol. As part of their work for this module, students are required to evaluate and carry out a
context-based analysis as part of a requirements gathering process for a computer-based system. The
requirements gathering process is to explore opportunities for a technological intervention, and also to ensure
that the solution developed will suit the particular situation/users; so the emphasis is on gaining a really deep
understanding of this: the people involved, their activities and the context. The student designer needs to
consider the question: ‘what are the constraints and barriers within the situation that need to be addressed?’
To facilitate this, the PACT (People, Activities, Context, and Technology) framework is used to prompt
students to consider specific categories in their analysis. The elements of the framework are described by
Benyon et al. (2010):
1. People: they differ physically, psychologically, and in terms of their knowledge of
technology.
2. Activities: they differ in terms of temporal aspects (response time, frequency of the
activity, time pressure and peaks), cooperation, complexity, and safety-criticality.
3. Contexts: the different environments in which the activities take place encompass the
organisational and social context and the physical environment.
165
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. Technologies: these should reflect the specific issues identified in considering the previous
elements. Features include input, output, communication, and content.
The students’ brief is to go into specific environments relevant to their design task and to collect data
regarding the first three elements of the PACT framework, using mainly observation, but also formal and
informal interviews, questionnaires and focus groups. They then need to analyse critically the data collated in
order to identify possible constraints and solutions.
As stated earlier, this research is seeking to investigate a mobile location-based system to support the
students’ activity. The next section details our findings from the requirements gathering work carried out.
3.1 Research Methodology

Requirement gathering is a significant part of any user-centred design (Lazar et al., 2010); it aims to establish
a deep understanding of the situation, to refine the user requirements and identify the functional and
non-functional requirements of an application. In order to improve validity, a range of approaches was used,
enabling us to triangulate the findings as part of the analysis. The approaches include the analysis of the
students’ submitted assignments, observations of teaching and interviews with the lecturers. Figure 1
illustrates this approach.
Figure 1. Requirements approach
3.1.1 Analysing Students’ Submitted Assignments

Undergraduate students in their second year of the Web Design course in the Department of Computer
Science and Creative Technologies at the University of West of England were required to submit a portfolio
of small assignments. 47 out of 48 students submitted the part of their portfolio considered here. The work of
these students was anonymised prior to analyses. Each student’s work was separately scrutinised to identify
his/her weaknesses and any good practice. It is crucial to know how common a particular issue is among the
students to gain an understanding of whether that issue needs to be considered when designing and
developing the application. The analysis was verified by checking its correspondence with written feedback
from the lectures on each aspect of their work.
To anonymise the students, each was given a number from 1 to 47. The table below shows the issues
identified and occurrences, identified by the student number.
Table 1. Issues and occurrences
Issue Issues Occurrences Total number
ID of students
A No clear links of the issues discussed in P, A, and C to 2,3,4,7,8,11,12,14,17,20,2 17
Technologies. 2,24,26,27,29,34,45
B Some issues were not related to the right element of PACT 2,3,4,7,8,9,11,13,17,26,47 11
C No real consideration of the human factors. 3,4,10,13,22,23,28,29,35, 10
44
D Issues were general and not mainly context related 2,4,13,22,23,30 6
E No links of the issues discussed in P, A, and C with 4,10,23,32,33,36,41,44 8
Technologies.
166
F Gave the issue with the solution rather than putting the solution 11,17,24,26,27,37,41,45,4 10
under Technologies and linking it to P,A or C. 6,47
G Need more thoughtful consideration of the context 12,29,32,41,44, 5
H Need to address issues found under each PACT element. 13,28,29 3
I Technology issues could be expanded 16,17,35 3
J Linking should be more explicit 18,19,21,30,46,47 6
K Need to find solutions to current problems not eliminate ideas 28,19 2
because of that problem.
L Need to think about and address issue from observations, 28 1
experience.
M Need to identify issues under each P, A, C, and then see what T 31,32,34,35 4
can allow for the proposed self-checkout not for the cafeteria.
N Need to consider human factors in more depth. 33,34,35,41 4
In the table above we can see that 36% of the students had difficulties linking the characteristics of the
people, activities, and context identified to technologies. In other words they should have identified the
technologies that would serve the characteristics of the people carrying certain activities in that particular
context. Moreover, 25% of them had issues with understanding the PACT framework itself. However, it
should be noted that the lecturers had not put a great deal of emphasis on this, as mentioned below in the
section, Observations of Teaching. It is clearly important to consider the people who will be using the
technology; nevertheless, 21% of students did not give this much attention. The chart below shows each issue
with the corresponding percentage of students to whom this applies.
Figure 2. Percentages of the occurrences of issues

As future work we hope to organise a focus group to elicit further information about the difficulties
encountered by students from a pragmatic point of view, and the functionalities that they would wish to see
in the application to help them overcome these difficulties. We also hope to cross-reference this information
with the information obtained from the students’ coursework analysis presented above.
3.1.2 Interviews with Lecturers
Our aim was to explore the following issues with the lecturers teaching the HCI module:
1. Their current practice of teaching students, especially concerning the PACT framework’
2. Their current approach to explaining assignment to students,
3. The students’ current practice in completing the assignment, the difficulties they encounter, and the
reasons behind these difficulties from the lecturers’ point of view, and
4. What they hope this intervention will achieve.
A series of unstructured interviews with two lecturers teaching the HCI module was carried out. This is a
significant part in the requirements gathering as it highlights clearly the functionality of the application that
needs to be considered.
Current practice of teaching
In the Human-Computer Interaction module, students learn about how people undertake activities in
context using technologies. They apply the PACT (People, Activities, Context, and Technology) framework
to analyse situations in order for them to design interactive systems. The lecturers explain the PACT
framework in detail to students giving them specific examples to clarify the concept. These include scenarios
167
ISBN: 978-989-8533-12-8 © 2012 IADIS
such as (a) a female student using her smart phone to send a text message whilst on a moving bus, when she
is seated, when she is standing holding on to a bag and an overhead strap for balance, and when the bus is
extremely crowded, and (b) and elderly woman setting her burglar alarm which is located in a dimly lit
passageway, with situations where the elderly woman has different age-related conditions. These example
scenarios are formulated to support the students in understanding the elements of the framework.
Photographs are shown to provide students with a realistic view of the physical environment. However, the
weakness is that the students are not able to immerse themselves in the actual environment to get a tangible
understanding of the constraints, and therefore fail to develop empathy for the users.
Explaining the assignment to students
As part of one of their assignment activities, students are required to conduct a requirements study for the
design of a new technology. In the past this has included the design of a university information kiosk and a
digital guide for a music festival. This year students were asked to consider the design of a self-service
checkout for use in a cafeteria. Using the ‘PACT’ framework, students were required to analyse the factors
that they would need to take into account in designing such a system. They were required to gather data for
the analysis via observations of the OneZone cafeteria (Main University Cafeteria at UWE) at various times,
consider their own experiences, as well as conduct short interviews with at least 3 stakeholders.
Students were then required to present their findings as a mind-map/spider diagram, ensuring that there
were clear links between the People, Activities and Context elements and the Technologies considered. They
needed to explain in separate paragraphs and in relation to each element of the PACT framework, why the
points that they had noted were of significance.
This was explained to the students in class and described on their coursework assignment specification
alongside the marking criteria.
The students’ current practice of the assignment from the lecturers’ point of view
It is crucial to investigate the lecturers’ understanding of the students’ current practice of the assignment,
the difficulties they encounter, and the reasons behind it from their point of view. Lecturers assume that
students get distracted by the environment and sometimes forget the main purpose of their assignment. From
their experience, students miss out key details when carrying out their analysis, leading to a disconnected
analysis, especially between the elements of PACT. Moreover, students tend to forget that ‘people undertake
activities in context using technology’; as a result they fail to consider the implications of what they have
identified for each of the elements, People, Activities, and Context, in relation to the Technology. They thus
miss the purpose of their assignment, to analyse the situation and consider technologies that reflects peoples’
needs when carrying out certain activities in a particular context. In some cases, students fail fully to engage
with, or appreciate the relevance of going to the location at all, and complete the activity in a rushed manner
with little or no reflection.
The lecturers’ view of the intervention
The lecturers want this mobile application to assist students when carrying out their analysis. They want it
to provide students with prompts when they are at the location. These prompts should address the students’
weaknesses already identified by the lecturers and also from the analysis of the previous students’
assignments, discussed earlier in the paper. The lecturers suggest that the students should be able to capture
images using the application, take notes, and track their own progress.
3.1.3 Observations of Teaching
In addition to the interviews with lecturers, observation of teaching was conducted. This gave us a better
understanding of the current practice. Attending HCI lectures was a valuable part of the research, giving an
insight into how students engage with the lectures and what question they might raise about the PACT
framework and the assignment. Observing the collaboration forum on Blackboard was also useful, revealing
students’ queries and concerns and the feedback given by the lecturer. Students expressed concern was about
the elements of PACT and the relationship between the different elements. This raised significant questions
that needed resolving. To what extent is it crucial to encourage students to use the PACT elements correctly?
Is it a tool for bringing to light many factors or do we value it as a categorisation tool? It was important get
back to the lecturers to discuss those two issues. It was agreed that we should remind students of the PACT
elements without putting undue emphasis on categorisation.
168
3.2 Findings
This section will explain the insights gained so far. It also explains how we are translating these into design
features.
Students lose focus on the purpose of tasks when away from classroom. They may get distracted by their
surroundings and miss out key elements. So a key feature of this mobile application could be to assist
students of the purpose of their learning and support their progression through the activities in a personalised
manner.
When students reach a pre-specified location, the application should display a detailed map identifying
the various sub-locations and containing either text and/or images. These hints could be designed to aid them
in widening their perspectives, in developing their own ideas and in critical evaluation. The text notes could
vary from simple instructions and prompts, to questions, and in some cases to links that will open a quiz
webpage; the particular content would depend on the specific aspect that the lecturer would want the students
to focus on.
It is important to encourage students to think of issues beyond their own realm of experiences and
perspectives. Providing students with functionality to share comments, ideas and perhaps stories if desired,
may enable them to benefit from their peers’ knowledge and different perspectives. Adding a collaborative
learning aspect to the activity, students will be able to post their comments for their lecturers and fellow
students.
Students have varying levels of ability when it comes to design thinking, and work at different rates. A
mobile application provides opportunity for personalised learning, such as paced progression, checklists to
give a sense of achievement and motivation, and structured disclosure, based on the students’ level of
interaction with the application.
Some students have been found to struggle in analysing their findings and specifically in using their
findings to develop new ideas. Prompting them with probing questions that challenge their assumptions or
get them to explore other methods of requirements gathering, beyond observation could help them identify
innovative opportunities. This approach could also address the problem of their failing to identify appropriate
technologies for the specific characteristics that they have identified earlier.
4. CONCLUSION AND FUTURE WORK

Research in mobile situated learning in higher education is still in its beginnings. Our research has potential
to add to understanding of how mobile applications can assist students learning in-situ. This paper outlines
the background to the research, discussing some key concepts. It focuses on the data obtained from the
interviews conducted with the lecturers of the HCI module, the observations of teaching, and analysis of the
students’ submitted assignments; this data has confirmed the difficulties that students encounter whilst
carrying out their coursework, helping us to establish the functional and non-functional requirements to be
considered when designing and developing the mobile application. We are continuing with the research,
adopting a user-centred, iterative approach to the design. We are currently working on the first prototypes of
our application and will evaluate these, applying such usability criteria , as how easy the features of the
application are to understand, its learnability, the effectiveness of feedback and the ease of interaction.
REFERENCES
Abowd, G., et al. (1999). Towards a better understanding of context and context-awareness. HUC '99 Proceedings of the
1st International Symposium on Handheld and Ubiquitous Computing, Karlsruhe, Germany. 304-307.
Barkhuus, L., & and Dey, A. (2003). Is context-aware computing taking control away from the user? three levels of
interactivity examined. Proceedings of UbiComp 2003, Seattle, Washington. 150-156.
Benyon, D. 2010, Designing Interactive Systems. 2nd edn, Addison-Wesley., Reading.
Bhaskar, N., & and Govindarajulu, P. (2010). Advanced and effective learning in context aware and adaptive mobile
learning scenarios. International Journal of Interactive Mobile Technologies, 4(1), 9-13.
Brown, E., et al. (2010). Location based and contextual mobile learning in deliverable contribution small scale study,
STELLAR.
169
ISBN: 978-989-8533-12-8 © 2012 IADIS
Chen, C. -., & & Hsu, S. -. (2008). Personalized intelligent mobile learning system for supporting effective english
learning. Educational Technology & Society, 11(3), 153-180.
Chiou, C., et al. (2010). An adaptive navigation support system for conducting context-aware ubiquitous learning in
museums. Computers & Education, 55(2), 834-845.
Costabile, M., et al. (2008). Explore! possibilities and challenges of mobile learning. Proceeding of the 26th Annual CHI
Conference on Human Factors in Computing Systems CHI 08, Florence, Italy. 145-154.
Dewey, J. 1933, How We Think, DC Heath and Co, Boston, MA.
Duron, R., et al. (2006). Critical thinking framework for any discipline. International Journal of Teaching and Learning
in Higher Education, 17(2), 160-166.
Fisher, L., et al. (2007). Using situated learning theory to explore tablet pc use in academic work: Perceptions of staff and
students. IADIS International Journal on Computer Science and Information Systems, 2(2), 99-109.
Ghiani, G., et al. (2009). UbiCicero: A location-aware, multi-device museum guide. Interacting with Computers, 21, 288-
303.
Guerreroa, L., et al. (2010). A mobile learning tool for improving grammar skills. Procedia - Social and Behavioral
Sciences, 2(2), 1735-1739.
Hsu, H., & Liao, H. (2011). A mobile RFID-based tour system with instant microblogging. Journal of Computer and
System Sciences, 77(4), 720-727.
Hwang, G., & Chang, H. (2011). A formative assessment-based mobile learning approach to improving the learning
attitudes and achievements of students. Computers & Education, 56(4), 1023-1031.
Hwang, G., et al. (2010). Development of a ubiquitous learning platform based on a real-time help-seeking mechanism.
British Journal of Educational Technology, 42(6), 992-1002.
Hwang, G., at al. (2009). A context-aware ubiquitous learning environment for conducting complex science experiments.
Computers & Education, 53(2), 402-413.
Kelley, J.F., (1984), An iterative design methodology for user-friendly natural language office information applications.
ACM Transactions on Office Information Systems, 2:1, pp. 26–41.
Kinshuk, Chang, M., Graf, S. & & Yang, G. 2009, "Adaptivity and Personalization in Mobile Learning.", In 2009 AERA
Annual MeetingSan Diego, CA, USA., April 13-17.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University
Press.
Lazar, J., et al. (2010). Research methods in human-computer interaction. West Sussex, UK: Wiley & sons LTD.
Lunce, M. L. (2006). Simulations: Bringing the benefits of situated learning to the traditional classroom. Journal of
Applied Educational Technology, 3(1), 37-45.
Naismith, L., et al. (2004). Literature review in mobile technologies and learning No. 11). Bristol: NESTA FutureLab.
Park, D., et al. (2007). A context-aware smart tourist guide application for an old palace. Proceedings of the 2007
International Conference on Convergence Information Technology (ICCIT '07), 89-95.
Reynolds, R., et al. (2010). Web-based museum trails on PDAs for university-level design students: Design and
evaluation. Computers & Education, 55(3), 994-1003.
Rogers, Y., Sharp, H. & Preece, J. 2011, Interaction Design: Beyond human-computer ineraction, 3rd edn, Wiley & sons
LTD., West Sussex, UK.
Ryu, H., & and Parsons, D. (2008). A learner-centred design of a location-aware learning reminder. Int. J. Mobile
Learning and Organisation, 2(2), 187-200.
Scriven, M. & and Paul, R.W. 1987, "Critical Thinking as Defined by the National Council for Excellence in Critical
Thinking", 8th Annual International Conference on Critical Thinking and Education Reform, Summer 1987.
Shih, J. L., et al.. (2010). An inquiry-based mobile learning approach to enhancing social science learning effectiveness.
Educational Technology & Society, 13(4), 50-62.
Vavoula, G. N., et al. (2004). Producing guidelines for learning, teaching and tutoring in a mobile environment. 2nd
IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE), Taoyuan, Taiwan. 173-
176.
Yatani, K., et al. (2004). Musex: A system for supporting children's collaborative learning in a museum with PDAs.
Paper presented at the Wireless and Mobile Technologies in Education, 2004. Proceedings. the 2nd IEEE
International Workshop, 109-113.
Yau, J. & Joy, M. (2008) A Self-Regulated Learning Approach: A Mobile Context-aware and Adaptive Learning
Schedule (mCALS) Tool. International Journal of Interactive Mobile Technologies, vol. 2, no. 3, pp. 52-57.
Yau, J. & Joy, M. (2009) A Mobile and Context-aware Learning Schedule Framework from a pedagogical perspective -
an interview study. IADIS International Conference Mobile Learning, pp. 189-196, Barcelona, Spain.
170
ALIGNING LEARNER PREFERENCES FOR

INFORMATION SEEKING, INFORMATION SHARING
AND MOBILE TECHNOLOGIES
Leila A. Mills1, Gerald Knezek1 and Ferial Khaddage2

1
The University of North Texas, USA
2
Higher Colleges of Technology, UAE
ABSTRACT
This paper reports on the development of a new information communications technology (ICT) learning preference
survey, its cross-validation with attitudes towards mobile learning, and new perspectives on information seeking,
information sharing, and mobile access derived from the relationships uncovered. The Information and
Communications Technology Learning (ICTL) survey instrument is designed to measure learners’ preferences and
information behavior for learning, knowledge acquisition, and sharing in the Web 2.0 technology pervasive
environments of the twenty-first century. The instrument development process includes analysis for reliability and
validity of instrument scales. The multi-step refinement process revealed two constructs with, respectively, very good
and respectable measurement properties: Information Sharing (alpha = .83) and Information Seeking (alpha = .71).
Cross-validation with a newly established mobile learning instrument confirms that both the Information Sharing and
Information Seeking scales of ICTL have significant (p < .001) alignment with mobile learning attitudes. Implications
of findings are discussed.
KEYWORDS
ICT learning, information seeking behavior, mobile learning.
1. INTRODUCTION
Collaboration fostered by a doctoral level psychometrics course and scholarly interest in the role of mobile
technologies in learning prompted the authors to develop and validate an instrument to measure learners’
dispositions toward new learning and knowledge acquisition activities that are made available by information
and communications technology (ICT). The authors also sought to examine relationships between the ICT
survey and an established mobile learning survey.
The two surveys employed in this research study were:
1) The Information and Communications Technology Learning (ICTL) survey, and
2) The Mobile Learning Scale (MLS).
The development of the Information and Communications Technology Learning survey is described in
detail in subsequent sections of this paper. The Mobile Learning Scale v1.0 also plays a key role in this study
and therefore its development history and psychometric properties are briefly discussed. MLS v1.0 was
originally created from the key points developed for a paper on mobile learning prospects for informal
learning in higher education (Khaddage & Knezek, 2011). Many of these key points initially emerged during
working group discussions of the 2011 International Summit on ICT in Education (UNESCO, Paris, 2011)
(Knezek, Lai, Khaddage, & Baker, 2011). Explorations of the relationships among the emerging factors
uncovered the relationships that are reflected in the title of this paper and are also addressed in details in the
discussion. These findings illustrate that cross-validation techniques can contribute to the richness of
understanding as gauged by the underlying constructs found in the cross-validation of instruments with
established reliability.
171
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. CONCEPTUAL RATIONALE
Measuring student technology attitudes related to learning and information seeking in technology pervasive
learning environments of the twenty-first century is of particular interest to educators and learning
technologists who seek to integrate ICT for effective teaching and learning. Also of great interest are
learners’ attitudes towards learning with mobile technologies. The role of mobile technology in education is
yet to be clearly defined, and therefore it is reasonable that many current papers present convincing
arguments about ways that that mobile technologies and applications can bring many benefits to the learning
environment. For example, Khaddage and Latterman (2012) point out that mobile technologies and
application offer benefits such as portability, simplicity, and availability. A multi-year study conducted by
Kolko and colleagues at the University of Washington reports on the adoption of ICT in Central Asia stating
that “mobile phone usage is outpacing the rate of Internet adoption, that access to the Internet is primarily
through public access sites”. Also reported is a trend towards the combination and conflation of technology
related information seeking and communication usage patterns (Kolko, Rose & Johnson, 2007).
The seemingly unlimited new media contexts and information access options in formal and informal
settings are often regarded as venues of invaluable opportunity for teaching and learning in the twenty-first
century (Arnone, Small, Chauncey & McKenna, 2011). There are those who warn that education that does
not utilize these new media-based technologies will not adequately develop student potential, leaving
students with a deficit in digital literacies that may contribute to student disengagement from classroom
activities (Judson, 2010). Felt (2010) identifies new media literacy (NML) skills as a set of cultural
competencies, social skills, and ICT tool skills that are essential for learning and mastery in the twenty-first
century. New media literacy skills are being associated with content-based learning in the framework of
Jenkins and colleagues (Jenkins, Clinton, Purushotma, Robison, Weigel, 2006), and are thought to support
student thinking and learning.
With information communications technology (ICT) now commonplace in all facets of our daily lives,
including teaching and learning (Christensen & Knezek, 2006), it is important to understand student
information seeking behavior in the Web 2.0 framework for formal and informal learning. Kuhlthau's
Information Search Process (ISP) model, devised in the 1980’s and revised in the 1990’s, depicts six stages
of student information activity: initiation, selection, exploration, formulation, collection, and presentation
(Kuhlthau, 1991). These six stages reflect the primary tasks to be accomplished when students go beyond
social communications technology exchanges and engage in complex information behaviors that can be
useful for directing learning activities. Emphasis should be placed on supporting information seeking tasks
which are associated with independent learning and knowledge construction (George, Bright, Hurlbert,
Linke, St. Clair & Stein, 2006). Established for an understanding of student information behavior in support
of guided instructional, diagnostic, and intervention programs, Kuhlthau’s ISP model has been applied
extensively for decades. An extensive review of literature and inquiry project conducted in 2007 among
(n=574) school students indicated that Kuhlthau’s information seeking model for affective, cognitive, and
physical dimensions of information behavior continues to be usefulness for explaining student information
seeking behavior and knowledge acquisition in the digital, technology pervasive information environment of
Web 2.0 (Kuhlthau, Heinstrom, & Todd, 2008). Findings confirm that while some stages in the search
process may be intensified in new instantaneous information environments “the Information Search Process
seems to be an over-arching process regardless of search venue, or print or digital format” (Kuhlthau, et al,
2008).
Mobile learning is a relatively new phenomenon with its theoretical basis still under development
(Kearney, Schuck, Burden, Aubusson, 2012). With the rapid growth of mobile devices throughout the world
(World Bank, 2012), the need has emerged for studies of affordances and barriers that might enhance or
constrain the adoption of mobile learning in higher education. This approach is consistent with the principles
of diffusion of innovations (Rogers, 2003) that have served as a useful model for studying the introduction of
new information technologies in the past, and which have been suggested by Chueng and Hew (2009) as an
appropriate framework for examining the adoption of mobile devices in teaching and learning.
While mobile learning is recognized as the desired outcome for application of mobile technology to
distance learning, the widespread availability of mobile technology and the association of mobile learning
with new forms of communications related activities for engaged learning spotlights mobile learning for a
broad range of formal and informal learning activities in the twenty-first century (Andrews and Tynan,
172
2012). Cox (2012) contends that research approaches investigating innovative ways of teaching and learning
with ICT in the future should address technology-enhanced learning “outside formal educational settings” (p.
2) as well as the opportunities presented by “the uptake of thin client technologies” (p. 5) that are mobile and
personalized. Lai (2011) devoted a section of his work on digital technology and the culture of teaching and
learning in higher education to “… how digital technologies may provide a more active and flexible learning
experience by adopting a participatory pedagogical approach and by blending formal learning with informal
learning” (p. 1263).
3. DEVELOPMENT OF THE INFORMATION AND COMMUNICATIONS

TECHNOLOGY LEARNING (ICTL) SURVEY
The Information and Communications Technology Learning (ICTL) survey was designed and validated to
help address questions related to how students prefer to utilize ICT for information seeking, information
sharing, and knowledge acquisition. Instrument development included analysis for internal consistency
reliability, principal components exploratory factor analysis, multidimensional scaling, and higher order
factor analysis. Items were initially gathered by the first author in an effort to better understand learner
preferences and the role that ICT plays in learning today. A review of literature did not reveal validated
instruments for measurement of student learning preference for use of ICT for learning activities and
knowledge acquisition within the ubiquitous communications landscape of the twenty-first century (Mills and
Knezek, 2012). Items for a 15-item prototype instrument, the Information Communications and Technology
Learning (ICTL) survey version 1.0, are displayed in Figure 1. Survey questions are Likert-type, rated on a 5
point scale with response choices from 1= strongly disagree to 5 = strongly agree. Note that item 9 is
negatively worded requiring that response ratings be reversed before summing (or averaging) with ratings
from other items to form a Likert-type scale. The reverse coding for item 9 results in answer choices having
increasing value along the response scale 1-5.
Information and Communications Technology Learning (ICTL) survey

1. I would like to be a participating member of an online community.
2. I use Internet technology to explore topics of interest.
3. I like to share interests and reflections online.
4. I like to enroll in classes to continue my education.
5. I use Internet communications and other technology tools for self-expression.
6. I learn many things by interacting with other Internet users.
7. I like to take classes from good professors.
8. I use Internet communications technology tools when I want to learn about something new.
9. I learn best in a traditional classroom setting. (R)
10. Internet technology helps me be successful in my college classes.
11. More classroom learning should include interactive communication technology experiences.
12. The things I need to know are taught by instructors in the classroom.
13. I learn more when I regulate my own learning experience and seek information on things that I want to
learn about.
14. I use Internet communications technology to keep current on topics related to my field of expertise.
15. I post information that might be of interest to other people.
Figure 1. Information and Communications Technology Learning (ICTL) survey items. Note: ICTL v1.0 2011 by L.
Mills & G. Knezek.
4. DATA COLLECTION
Survey subjects were volunteer participant college students enrolled in one of two institutions of higher
education in Texas (USA), a junior college with a two-year curriculum, and a four-year state university.
Sixty-two (62) respondents completed the ICTL survey online during the fall semester of 2011. The survey
participants were 89% women (n=55) and 11% men (n=7) spanning 18 to 59 years of age.
173
ISBN: 978-989-8533-12-8 © 2012 IADIS
5. INSTRUMENT REFINEMENT
5.1 Internal Consistency Reliability and Construct Validity

Reliability analysis was used to assess the internal consistency of the instrument scales according to
Cronbach’s Alpha index. As shown in Table 1, Cronbach’s Alpha for all 15 items of the ICTL instrument is
considered “respectable”, (alpha = .77) according to reliability guidelines by DeVellis (1991).
Exploratory factor analysis, principal components analysis (PCA) with varimax rotation, was conducted
in order to identify scales/factors that are orthogonally aligned (Mertler & Vannatta, 2005). PCA produced
four factors with eigenvalues greater than one. Scree plot analysis (see Figure 2) indicated a two or possibly
four factor solution in light of Stevens’ (1992) suggestion that constructs in the sharp decent of the graph,
before the first point of leveling, be retained. The four-factor solution was selected because the items within
each factor were judged to have greater content validity. These four factors (see Table 1) were tentatively
named Online Reflection, Internet Exploration, ICT Research, and Classroom Learning, with alphas of 0.88,
0.70, 0.43, and 0.54, respectively, when subjected to post hoc internal consistency reliability analysis.
Measurement scales produced for these factors were found to have internal consistency reliabilities ranging
from very good for Online Reflection to unacceptable for Classroom Learning, according to guidelines by
DeVellis (1991).
Table 1. Cronbach’s Alpha for ICTL. Internal consistency reliabilities for ICTL scales
Scale No. of Item Cronbach’s Alpha DeVellis
Items numbers Guidelines
ICTL- Total scale 15 1-15 0.77 Respectable
Online Reflection 6 3, 5, 11, 6, 0.88 Very Good

15, 1
Internet
Exploration 4 7, 2, 13, 4 0.70 Respectable
ICT Research 3 8, 10, 14 0.43 Unacceptable
Classroom Learning 2 12, 9 0.54 Unacceptable
Figure 2. Scree plot for exploratory factor analysis of ICTL items

Higher order factor analysis was conducted to explore explanatory constructs at a higher level common
among the four factor scales identified by PCA: Online Reflection, Internet Exploration, ICT Research,
Classroom Learning (see Table 2). Scale scores produced for each of the factors were used as input variable
values for the higher order procedure. A Scree plot for higher order factor analysis (Figure 3) reveals two
factors in the sharp decent of the line. Table two shows that these two components explain 70% of the total
variance.
174
Table 2. Higher order factor analysis loadings for four ICTL scales. Two components account for 70% of variance
___________________________________________________________________
Figure 3. Scree plot for higher order factor analysis

Internal consistency reliability was computed for these two higher-order construct scales. The first scale,
Information Seeking was found to have alpha = .71 (respectable according to DeVellis (1991)). Cronbach’s
Alpha index for the second scale, Information Sharing was .83 (very good per DeVellis’ guidelines (1991)).
The items forming each higher-order scale are listed in Table 3.
5.2 Higher Order Factor Analysis and Multidimensional Scaling

Higher order factor analysis and multidimensional scaling indicated two possible factors which, upon
examination by researchers, proved to have content (face) validity. One factor consisted of items that were
related to ICT use for reflection, communication, and sharing. The second factor grouped items related to
ICT for seeking information related to areas of interest, expertise, or study. Multidimensional scaling was
conducted to further examine the underlying factors for the ICTL survey. The ALSCAL Euclidian distance
model with maximum of two dimensions was generated to examine distances and proximities for items in
relation to one another. Two main output clusters are visible on the Euclidean distance model (Figure 4). This
two-scale alignment confirms results of higher order factor analysis. Instrument items in quadrant II & III,
with item #8, which is located near the Y axis were examined as the first of two scales, and all other items in
quadrants I & IV as the second of two scales which were identified by use of higher order factor analysis and
multidimensional scaling. This two-factor solution was accepted and resulted in the Information Seeking, and
Information Sharing scales of the ICTL.
175
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 4. Euclidean distance model. Multidimensional scaling plot of ICTL items
Table 3. Items for higher order scales. Two scales emerged from higher order factor analysis
ICTL Scale # Items Item Alpha Rating (DeVellis)
Numbers Cronbach’s
ICTL- Total scale 15 1-15 0.77 Respectable
subscales:
7 2,4,7,8,10,13, 0.71 Respectable
Information Seeking 14
Information Sharing 8 1,3,5,6,9r,11, 0.83 Very Good

12,15
Note: Item #9 was reverse coded.
5.3 Criterion-related Validity

Criterion related validity was examined for the three scales of ICTL by correlation analysis with the Mobile
Learning Scale, a 7-item, unidimensional, Likert-type survey instrument developed by Khaddage and Knezek
(2011) for used with higher education students. The Mobile Learning Scale version 1.0 was designed to
assess acceptance of mobile learning, including perceptions of mobile learning devices and tools (Apps) for
informal learning, and feelings about using theories and models to incorporate mobile learning into higher
education (Knezek & Khaddage, 2011). Knezek and Khaddage (2011) found the internal consistency
reliability for the seven-item instrument to be very good (alpha = 0.85) (DeVellis, 1991) among 81
undergraduate and graduate university students completing the survey in a large Midwestern university in the
USA during August and September of 2011. A list of the items on the Mobile Learning Scale version 1.0 is
provided in Figure 5. Correlation analysis revealed concurrent validity between ICTL, Information Seeking,
Information Sharing and the Mobile Learning Scale. Significant correlations (p < .01) were found between
Mobile Learning (ML) total scale score and all three ICTL measures: a) ML with ICTL_Total r = .37 (p <
.0005), b) ML with ICTL_Information Seeking r =.35 (p < .0005), and c) ML with ICTL_Information
Sharing r =.37 (p < .0005). This magnitude of Pearson Product Moment Correlation would be considered a
moderate effect size according to guidelines by Cohen (1988). This confirms that the ICTL survey scales
have demonstrable criterion related validity in the form of alignment with an established Mobile Learning
Scale. This research finding contributes new knowledge to the emerging set of models and theories
developing in the field of mobile learning. In particular, the implication is that both information seeking and
information sharing are important aspects to be considered in mobile learning environments
176
Instructions: Select one level of agreement for each statement to indicate how you feel.
SD = Strongly Disagree, D = Disagree, U = Undecided, A = Agree, SA = Strongly Agree
MOBILE LEARNING SCALE SD D U A SA
The rapid development of Mobile Learning devices and tools
1.
(Apps) has empowered informal learning.
Mobile Apps could be integrated seamlessly to support informal
2.
learning.
Mobile Apps could bring enormous opportunities into universities
3.
to further empower informal learning
Student acceptance of Mobile Learning in higher education would
4.
be high.
Recent developments in Mobile Learning are leading to the
5.
exploration of new methods/models at universities
Theoretical models and methods can assist in informing the design
6.
for mobile learning Apps
The integration of mobile applications, mobile social networking
7. platforms and other mobile technologies has become pervasive in
teaching and learning.
Figure 5. Mobile Learning Scale (MLS) Version 1.0 items. Note: MLS v1.0 8/2011 by F. Khaddage & G. Knezek
Additional associations were found between other attributes of interest and one or more of the scales used
in this study. Females tended to rate higher than males on mobile learning attitudes (p = .012) but there were
no significant differences by gender on information seeking or information sharing preferences. Age was
negatively correlated with information seeking ratings (p = .008) and information sharing (p = .001), but no
significant correlation was found between age and mobile learning attitude (NS). Mobile learning attitudes
were positively correlated (p = .002) with self-reported creative tendencies as measured by a 13 item Likert
Scale. These findings add context to the associations reported in the previous paragraph, and raise additional
questions to be addressed in future research.
6. DISCUSSION
In recent decades the role of formal education as the disseminator of significant learning experiences has
diminished (Aittola, 1999). Information communications technologies of the twenty-first century support a
variety of learning, sharing and knowledge acquisition options which can take place in many places
facilitated by internet-based communications tools (Knezek, Lai, Khaddage & Baker, 2011), thereby altering
the role of formal education in learning. The information seeking, sharing and mobile access dimensions of
learning which are measured by validated instruments identified in this paper are well fitted to assist teachers
and learning technologists to design instruction that will support learning interaction with new options in
mobile learning technologies and ICT.
As stated by Naismith, Lonsdale, Vavoula, and Sharples, (2004, p. 36) the current challenge is for
educators and technology developers to find a way to ensure that new learning methods are effective – highly
situated, personal and collaborative for the long term, and accepted by students. Research for formal and
informal learning in new technology pervasive environments should include data on student preferences that
can help identify students’ attitudes, level of acceptance, opinions and expectations regarding the integration
of new technologies such as mobile learning into teaching and learning. This approach is what Andrews and
Tynan (2012) have referred to as “investigating the human voice” (p. 565) to meet the unique needs of
today’s learner. Hence, by creating and validating instruments to measure attitudes toward adoption and use
of new technology options, such as the ones presented in this paper, which can more precisely and efficiently
assess reliability and validity and make continuous refinements, we can allow for informed understanding of
ICT learning and information seeking preferences of students. Additionally, the findings of this study imply
that the ICTL and Mobile Learning Scale are useful for measurement in three areas of focus for the study of
students’ needs: information seeking preferences and behaviors, information sharing preferences and
177
ISBN: 978-989-8533-12-8 © 2012 IADIS
behaviors, and perceptions of teaching and learning with mobile devices.

A review of the literature revealed that existing instruments for measurement of technology in teaching
and learning have frequently focused on self-reported measures that assess student and teacher attitudes and
proficiency with information and communication technology for technology integration (Knezek &
Christensen, 2000). Instruments have also been validated to measure success with teaching and learning in
Internet enriched learning environments (Hong, Ridzuan, & Kuek, 2003). A number of instruments measure
different facets of learning. For example, Litzinger, Wise, Lee, & Bjorklund (2003), developed an instrument
to measure aspects of self-directed learning. The authors contend that there is a need for the ICT and mobile
learning scales to measure learner preference for ICT and mobile technologies for information behavior such
as seeking and sharing.
7. SUMMARY AND CONCLUSIONS

Data collected online from college student volunteers for a learning preference study was used to refine the
Information Communication Technology Learning (ICTL) survey. Construct validity was examined in a
multi-step process. Reliability was analyzed for all 15 items of this survey and found to be respectable (alpha
= .77) according to established guidelines (DeVellis, 1991). Four possible measurement factors were
identified by PCA exploratory factor analysis: two of these were considered unacceptable with low internal
consistency reliabilities according to established guidelines. However, higher order factor analysis and
multidimensional scaling identified a two higher order constructs with very good and respectable
measurement properties: Information Sharing (alpha = .83) and Information Seeking (alpha = .71). Scales
produced from these two factors were found to be significantly correlated (p < .0005) with student mobile
learning (MLS) attitudes, providing cross-validation evidence for ICTL scales. The correlations between
ICTL scales and MLS were at the r = .35 level or higher, indicating that sharing and seeking information both
are perceived as functions to be performed with mobile technologies. These findings support the authors’
conclusion that the Information and Communications Technology Learning (ICTL) instrument is worthy of
further use, either alone or as part of a battery of instruments including others such as the Mobile Learning
Scale.
This study also contributes new knowledge to the emerging set of models and theories developing in the
field of mobile learning. In particular, the implication is that both information seeking and sharing behaviors
are important points of consideration in mobile learning environments. Further research is needed to
determine if the instruments used for this research can assist educators and researchers in better
understanding student preferences for ICT integration in order to improve support for information behavior
targeting formal and informal learning. Findings from this study, such as the positive correlations identified
between mobile learning attitudes and creative tendencies, and the confirmed female preference for mobile
learning while no gender differences regarding information seeking or sharing were found, suggest that the
ICTL and MLS instruments possess criterion related characteristics that may be useful in addressing
additional research questions on students’ attitudes towards mobile devices, ICT integration, and information
seeking behaviors. These areas will be pursued in future studies.
REFERENCES
Aittola, T., (1999). Possibilities of Informal Learning Inside and Outside the School. Retrieved:
http://eric.ed.gov/PDFS/ED469193.pdf.
Andrews, T. & Tynan, B., (2012). Distance learners: connected, mobile and resourceful individuals. Australasian Journal
of Educational Technology, Vol. 28, No. 4,pp 565-579.
Arnone, M.P., et al., (2011). Curiosity, Interest and Engagement in Technology-Pervasive Learning Environments: a New
Research Agenda. Educational Technology Research and Development. Vol. 59 No. 2, pp181-198.
Cox, M. J., (2012). Formal to informal learning with IT: research challenges and issues for e-learning. Journal of
Computer Assisted Learning DOI: 10.1111/j.1365-2729.2012.00483.x.
Cheung, W. S., & Hew, K. F., (2009). A review of research methodologies used in studies on mobile handheld devices in
K-12 and higher education settings. Australasian Journal of Educational Technology, Vol. 25, No. 2,pp 153-183.
178
Cohen, J.,( 1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Earlbaum Associates,
Hillsdale, NJ.
Christensen, R., & Knezek, G., (2006). Construct Validity for the Teachers’ Attitudes Toward Computers Questionnaire,
Journal of Computing in Teacher Education, 2006, Vol. 23, No. 2, pp 55-61.
DeVellis, R.F.,( 1991). Scale development. Sage Publications, Newbury Park, NJ.
Felt, L., (2010). Making Education (Double) Count. Boosting Student Learning via Social and Emotional Learning and
New Media Literacy Skills. Elearn Magazine. Retrieved: http://elearnmag.acm.org/archive.cfm?aid=1839556
George, et al., (2006). Scholarly use of information: graduate students' information seeking behaviour. Information
Research, Vol. 11, No. 4, paper 272. Retrieved: http://informationr.net/ir/11-4/paper272.html.
Hong, K., et al, (2003). Students' attitudes toward the use of the Internet for learning: A study at a university in Malaysia.
Educational Technology & Society, Vol. 6, No. 2,pp 45-49. Retrieved: http://www.ifets.info/journals/6_2/5.html
Jenkins, H., et al, 2006. Confronting the challenges of participatory culture: Media education for the 21st Century. The
MacArthur Foundation, Chicago, IL.
Judson, E. (2010). Improving technology literacy: does it open doors to traditional content? Educational Technology
Research and Development, Vol. 58, No. 3,pp 271-284.
Kearney, M., et al, (2012). Viewing mobile learning from a pedagogical perspective. Research in Learning Technology,
Vol. 20, No. 1063519, pp 1-17.
Khaddage, F., & Knezek, G., (2011), Device Independent Mobile Applications for Teaching and Learning: Challenges,
Barriers and Limitations. Proceedings of Association for the Advancement of Computing in Education: Global
Learn. Asia Pacific, pp 1-7.
Khaddage, F.,Lattenman C., (2012) The Future of Mobile Apps for Teaching and Learning Book Chapter in the
“Handbook of Mobile Learning – definitions Publisher: Routledge (Imprint of Taylor and Francis Group), USA (in
press).
Knezek, G. & Christensen, R., (2000). Impact of new information technology on teachers and students. Education and
Information Technologies, Vol. 7, No. 4,pp 369-376.
Knezek, G., & Khaddage, F. (2011). A preliminary analysis of the Mobile Learning Scale. Paper presented to the World
Educational Research Association (WERA) Annual Conference. Kaohsiung, Taiwan, December 17, 2011.
Knezek, G., et al. (2011). Student technology experiences in formal and informal learning. Preceedings of EduSummIT.
Paris, France. Retrieved: http://edusummit.nl/res2011/calltoaction2011/briefpapers2011.
Kolko, B., et al., (2007). Communication as information-seeking: the case for mobile social software for developing
regions. Paper presented at the 16th international conference on the World Wide Web (WWW), Banff, Alberta,
Canada, pp 863 – 872.
Kuhlthau, C, (1991). Inside the search process: information seeking from the user's perspective. Journal of the American
Society for Information Science, Vol 42, No. 5,pp 361-371.
Kuhlthau, C., et al, (2008). The ‘information search process’ revisited: Is the model still useful? Information Research,
Vol. 13, No. 4.paper 355. Retrieved from http://informationr.net/ir/13-4/paper355.html.
Lai, K. W. (2011). Digital technology and the culture of teaching and learning in higher education. Australasian Journal
of Educational Technology, Vol. 27, Special Issue No. 8,pp 1263-1275.
Litzinger, T., et al. (2003). Assessing Readiness for Self-Directed Learning. Proceedings, 2003 ASEE Conference and
Exposition American Society for Engineering Education. Retrieved from
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.2900&rep=rep1&type=pdf
Mertler, C., & Vannatta, R. (2005). Advanced and Multivariate Statistical Methods (Third. Edition). Pyrczak Publishing,
Los Angeles, CA.
Mills, L. & Knezek, G. (2012). Measuring Learning Preferences within the Integrated Communications Learning
Landscape. In P. Resta (Ed.), Proceedings of Society for Information Technology & Teacher Education International
Conference 2012 . Chesapeake, VA,pp 1994-1999.
Naismith, L., et al. (2004). NESTA Futurelab Report 11: Literature Review in Mobile Technologies and Learning.
NESTA Futurelab. Bristol, UK.
Stevens, J., 1996. Applied multivariate statistics for the social sciences. Lawrence Erlbaum Associates, Hillsdale, NJ. pp
366.
Rogers, E. M. (2003). Diffusion of innovations, 5th ed. The Free Press, New York.
World Bank (2012). Maximizing Mobile. Retrieved
http://siteresources.worldbank.org/EXTINFORMATIONANDCOMMUNICATIONANDTECHNOLOGIES/Resourc
es/IC4D-2012-Report.pdf
179
ISBN: 978-989-8533-12-8 © 2012 IADIS
INFLUENCE OF STUDENTS’ LEARNING STYLES ON THE

EFFECTIVENESS OF INSTRUCTIONAL INTERVENTIONS
Thomas Lehmann* and Dirk Ifenthaler#

*
University of Freiburg
#
University of Oklahoma
ABSTRACT
This research contributes to answer the question whether learning/cognitive styles of students serve as a justified starting
point for creating target-group appropriate instruction. The study was realized in a self-regulated problem-based learning
environment. Data of 56 participants on their individual learning styles, their acquired problem solution and their increase
in declarative knowledge were collected. Results indicate that a consideration of learning styles to design a matching
instruction or learning environment does not transfer into higher quality problem solutions or an enhanced increase in
declarative knowledge.
KEYWORDS
Learning style, cognitive style, preflection, prompting, mental models, self-regulated learning
1. INTRODUCTION
To be an effective self-regulated learner, students actively have to influence and adjust all learning processes
of the cognitive, metacognitive, and motivational dimension. Such a multilayered regulation of learning
enables improvements in procedures of the cognitive, metacognitive, and motivational component of learning
(Schiefele and Pekrun, 1996, Bransford et al., 2000). Zimmerman’s (2008, 2000, 1998) and Pintrich’s (2000)
models of self-regulation share those key-components and orchestrate their sub-processes in a
cyclic-reflective understanding to specific phases of self-regulated learning.
Learning styles are regarded as one of the crucial factors to be taken into account when designing
instruction and learning environments: “Instruction designed to address a broad spectrum of learning styles
has consistently proved to be more effective than traditional instruction, which focuses on a narrow range of
styles” (Felder and Brent, 2005, p. 59). However, the scientific understanding of the concept of learning style
is rather inconsistent. Learning-style models range from holistic approaches (e.g., the Dunn and Dunn
learning-style model; Dunn et al., 2009) to specific models that focus on certain dimensions of learning, i.e.,
information gathering, processing, and retrieval (e.g., the Felder-Silverman learning-style model; Felder and
Silverman, 1988).
2. THEORETICAL BACKGROUND
2.1 Learning Styles in Theory and Educational Practice

The field of educational psychology comprises numerous learning style models, all aiming at an
improvement of learning. Learning style assessments could contribute to the learner’s intrapersonal
knowledge as a part of his/her declarative meta-knowledge. Accordingly, intrapersonal knowledge is defined
as knowledge about the own thinking, memory, and corresponding tendencies (Brown, 1984). Therefore, by
knowing about specific preferences a learner unifies, he/she at least fulfills a basic prerequisite to self-
regulate his/her learning process and/or environment in a way that suits his/her individual learning
preferences.
180
However, assessing learning styles could not only contribute to the learners’ meta-knowledge and
subsequently create the basis for improvement of his/her control mechanisms in a self-regulated learning
environment. Even before learners participate in a (self-regulated) learning environment, learning styles
could be an important factor. As aforementioned, several ID-models advise to assess the actual learning
styles of the target-group as a central aspect of the learners’ characteristics (e.g., Morrison et al., 2011,
Rothwell and Kazanas, 2008). Analogous to self-regulation mechanisms of learners that can occur in the
phases of self-regulated learning, instructional designers could consider the predominant learning styles of
their target-group when designing instruction. Theoretically this has to result in an optimization of the
designed instruction by improving its suitability for the target group/learner.
The relevance of learning styles for instructional designers as well as the field of self-regulated learning
and according research, thus, seems to be unquestionable. However, a vast number of different
understandings of learning style definitions, theoretical positions, models, and measures can be identified.
This leads to an increasing lack of clarity what a learning style is and how it facilitates instruction and (self-
regulated) learning processes. Cassidy (2004) states: “As a consequence of the quantity of research, the
disciplines and domains in which the research is conducted, and the varied aims of the research, the topic has
become fragmented and disparate” (p. 419). To deal with learning styles, first, a basic understanding for the
concept is obligatory and to be presented in order of not to contribute to further dilution of the concept.
Hawk and Shah (2007) consider learning styles not as a state. The authors mention that learning styles
and learning style models all share the basic assumptions that “students learn in different ways”, the concept
of “learning style is a component of the wider concept of personality”, and “learning style falls into the
categories of dispositional traits and characteristic adaptations where there are differences across individual
humans but there are groupings of humans who have common or similar learning style characteristics”
(Hawk and Shah, 2007, p. 2).
Referring to Hawk and Shah’s (2007) characterizations as a basic premise, an interchangeable use of the
terms “learning style” and “cognitive style” seems to be possible. Some models comprise solely cognitive
aspects of gathering, processing and retrieving information (e.g., the Felder-Silverman model), whereas other
models include constructs dependent on the age-appropriate assessments administered, e.g., environmental
stimuli (sound, light, temperature, seating, etc.) that feature no direct relation to cognitive processes (Dunn et
al., 2009). Riding and Cheema (1991) further elaborate on the not well defined usage of learning style and
cognitive style: “Whilst cognitive style is a bipolar dimension, learning style entails many elements and are
usually not ‘either-or’ extremes. One either has or does not have the element in one’s style, similarly, the
absence of one element does not necessarily imply the presence of the opposite element” (p. 194).
2.1.1 Empirical Findings on Learning Styles
Although, terminological shortcomings in learning/cognitive style research have been made explicit for
several decades and the benefit of taking such styles into account when designing instruction seems to be
obvious – at least from a theoretical perspective – scientists still argue about its practical efficacy. Regarding
various empirical studies on learning styles, results still draw an unclear picture about the “true” relevance of
the concept.
An in-depth literature review revealed numerous contradicting results of research on learning styles. The
following investigations and reviews all support the benefit that arises from considering learning styles when
designing instruction (e.g., Dunn et al., 2009, Schmeck, 1988, Hayes and Allison, 1993, Hayes and Allison,
1996, Riding and Grimley, 1999, Bajraktarevic et al., 2003). The scope of the instructions observed in these
studies provides a wide range from traditional training situations to hypermedia learning environments. In
contrast, other studies and reviews disagree with the aforementioned (e.g., Kavale and Forness, 1987, Snider,
1992, Cook et al., 2007, Gilbert and Swainer, 2008, Martin, 2010). Analogous to the contradicting studies,
the investigated learning environments and instructional settings also range from traditional classroom to
multimedia learning environments. Thus, as already mentioned, the picture about the “true” relevance of
learning styles remains unclear. Positive effects of taking learning styles into account when designing
instruction are still questionably.
2.1.2 Index of Learning Styles (ILS)
The Index of Learning Styles (ILS; Felder and Silverman, 1988) was “designed to capture the most important
learning style differences among engineering students and provide a good basis for engineering instructors to
181
ISBN: 978-989-8533-12-8 © 2012 IADIS
formulate a teaching approach that addresses the learning needs of all students” (Felder and Spurlin, 2005, p.
103). The model shares commonalities with other popular learning style approaches, e.g., the Learning Style
Inventory (LSI; Kolb, 1984) or the Myers-Briggs Type Indicator (MBTI; Lawrence, 1994).
With regard to the theoretical assumptions made above, the ILS is explicitly said not to include ‘either-or
categories’ of its bipolar dimensions. All scales are to be understood as continua, which means that a
student’s cognitive preference to learning on a given ILS scale may be either fairly well balanced,
moderately, or strongly distinctive for one or the other pole of the scale. The four bipolar ILS dimensions can
be described as follows (Felder and Silverman, 1988):
• Active – Reflective. Active learners tend to gather and understand information best if they engage
with it actively and try things out, e.g., by debating, bringing something to application, or via
teaching-back. Reflective learners prefer to think about new things respectively information for
themselves first. The motto of active learners is “Let us try and see how it works”, whilst reflective
learners pursue the principle “Let me first think carefully about it”.
• Sensual – Intuitive. Sensing learners tend to do well in learning facts and to follow established
approaches and procedures when solving problems. They are more goal-oriented, carefully and
patiently, but avoid complex issues or surprises. Intuitive learners on the other hand prefer to
explore different possibilities and relationships and innovative approaches. The can better grasp new
concepts, work usually faster and more innovative and have less difficulty with abstract concepts
and mathematical expressions. However, they tend to avoid rote learning, repetition, routines and
fixed schemes.
• Visual – Verbal. Visual learners remember more of what they have seen, e.g. pictures, diagrams,
flow charts, films, demonstrations, etc. Instead, verbal learners prefer linguistically based learning
that is written and spoken information or declarations. However, it should be noted at this point, that
regardless of the deviation of a person, the combination of visual and verbal information is most
conductive.
• Sequential – Global. Sequential learners tend to understand better by learning in logical linear
steps, where each step is the logical consequence of the previous step. In contrast, global learners
rather tend to make big steps and gather different material and information quasi-random and
without the recognition of contexts and relationships, but suddenly, they understand the whole
context.
2.2 Designing Model-Based Learning Environments

2.2.1 Mental Models as a Basis for Analyzing Problem Solutions
A mental model is an idiosyncratic representation of a fact or a thing, of ideas or more generally an ideational
framework about something interesting in the world. Mental models, as types of representations, rely on
language and use symbolic pieces and processes of knowledge to construct a heuristic for a situation. The
theory of mental models follows the constructivist assumption that human knowledge is always fragmentary
and thus regarded as dynamic, because it is constantly expanded and/or modified (Johnson-Laird, 1983,
Pirnay-Dummer et al., 2012). Hence, mental models function as internal models of the outside world and
produce subjective plausibility (Ifenthaler and Seel, 2011).
With regard to problem solving processes, studies show a strong correlation between effective mental
modeling and the solution of complex problems (Jacobson, 2000). According to Dörner (1976) problems are
characterized by means of three components: 1.) an unsatisfying actual state, 2.) a desired target state, and 3.)
a barrier which currently inhibits the mental transformation of the actual to the target state. The bigger this
barrier is the more complex the problem is regarded. The definition of problem solving processes clarifies the
importance of mental models for successful problem solving. Funke (2003) states that problem-solving
thinking is characterized by filling the gaps within an action plan that cannot be accomplished routinely. For
that purpose a mental model is constructed which bridges the way from the initial to the goal state (Ifenthaler
and Seel, 2011).
Ifenthaler (2010, 2008) describes the externalization of internal cognitive structures as a deliberate
communication process of mental models. Whenever the observation of knowledge representation is
essential, externalization is necessary, because direct access is not possible. Accordingly, methods to measure
182
knowledge representations result from the conscious communication of mental models, e.g., via thinking
aloud, text writing, constructing graphs, knowledge or concept maps (Ifenthaler, 2010). This results in the
differentiation between internal knowledge representations and external, so called re-representations
(Ifenthaler, 2010). The re-representations are communicated on the basis of the mental representations and
adequate sign and symbol systems (Shute et al., 2009, Seel, 1999). Accordingly, the solution to a
phenomenon in question which is represented by an individual mental model, and consecutively the re-
representation of this model as a result of a communication process allows the investigation of individual
problem solutions (Ifenthaler, 2010).
2.2.2 Preflection – a Pendant to Reflection
Expert self-regulated learners are said to be effective in actively influencing and adjusting all learning
processes of the three key-components of learning: cognition, metacognition, and motivation (Schiefele and
Pekrun, 1996, Bransford et al., 2000). Metacognitive processes are attributed to play a superior role, because
metacognition can serve as a tool to regulate the further dimensions of learning and consequently optimize
their specific aspects with regard to the goals set (Leutner and Leopold, 2003).
Zimmerman’s model of self-regulation (2008, 2000, 1998) suggests a cyclic-reflective understanding of
self-regulation. Three phases are identified analogous to an input-output-system, where each result of an
action sequence affects the following. At the beginning of a learning cycle stands the forethought phase.
Pintrich (2000) adds “planning and activation” to the label of that initial stage of each self-regulated learning
cycle. In general, it can be characterized as preactional phase, followed by the actional and post-actional
phase. Consequently, the learning phase of action is enclosed by a preparatory and a post-processing phase.
In terms of metacognition, the differentiation between prospective and retrospective monitoring (Nelson and
Narens, 1994) can be applied to the surrounding phases, represented by two psychological constructs:
Reflection is primarily characterized to refer to something experienced (Pintrich, 2000), thus suiting the
postactional phase, whereas preflection is relating to future events and based on prospective thoughts, thus
suiting the preactional phase. The latter, hence, is to be understood as the pendant to reflection. In regard to
what Dewey (1933) described when introducing the concept of reflection – humans learn more from
reflection on their own experience than from the experience itself – preflection can be characterized as
reflection’s inverted counterpart: a tool for optimizing subsequent experience, respectively problem-solving
and learning processes. Accordingly, preflection involves the activation of central learning relevant structures
(e.g., content knowledge, strategies, values, interest, etc.) that are prospectively regarded to be beneficial for
future performance.
2.2.3 Prompts as an Instructional Intervention in Self-Regulated Learning Environments
External instructional interventions have to be used at the right time and the right extent, so that the self-
regulatory learning process is not disturbed, but rather supported. Instructional designers need to develop
learning environments implementing an appropriate level of external- and self-control which poses not an
easy challenge. Therefore, research has gained big interest in investigations of demand-oriented support and
assistance interventions (scaffolds). One specific form of scaffolds are formed by instructional cues, so called
prompts. Within the context of self-regulated learning they are regarded as effective, because they serve as a
“short-time intervention” (Bannert, 2009) and only represent a minimal external control mechanism.
Davis (2003) and Ifenthaler (2012) differentiate between two forms of prompts: generic and directed.
Generic prompts follow the principle “stop and think”. A generic prompt encourages learners to interrupt the
current learning or problem solving processes for a moment and reflect. Here, the object of reflection is left
completely open. There are no specific issues highlighted or instructed by the prompt.
Directed prompts, on the other hand, follow the principle “stop and think about ...”. Davis (2003) states
that the more specific directed prompts should be more effective than general, unspecific prompts, if they
were adequately implemented in the learning or problem-solving environment. It is assumed here that
individuals who do not have the necessary knowledge and skills required for the desired deliberations –
reflective or preflective – need an additional instruction along with the prompt. Directed prompts include
such an instruction, for example, in form of to complete sentences. Ifenthaler (2012) alternatively found
generic prompts to be more efficient compared to directed prompts, because they leave a certain amount of
autonomy for self-regulative acting. Still, Ifenthaler (2012) agrees on Davis’ (2003) argument that directed
prompts may be more efficient for learners that do not already have a specific set prior knowledge and skills.
183
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.3 Research Questions and Hypotheses

The contrasting views within the community of educational psychology as well as the contradictory empirical
results on the effects of taking learning styles into consideration when designing instruction led us to the
present study. The objective is to investigate the influence of students’ learning styles on the effectiveness of
instructional interventions (metacognitive prompts within a problem-based learning environment). Regarding
the theoretical foundation of learning styles, we assume that reflective students as well as students with a
propensity for verbal learning develop higher quality problem solutions and perform better in retrieving
declarative knowledge than the active type of students and students with a preference for visual learning. The
reasons for these assumption are that (1) active learners tend to profit from actively doing something, e.g.,
discussing, applying information, teaching back, or trying it out; reflective learners gain much more from
introspective processing and thoroughly thinking something through before trying it out (Kolb, 1984) and (2)
the fact that the learning environments’ metacognitive prompts plus the problem solving itself happen
basically verbal. Thus, the following hypotheses are addressed: The more reflective and verbal the learners
are, the higher the quality of their developed problem solutions (Hypothesis 1). The more reflective and
verbal the learners are, the better they perform in retrieving declarative knowledge (Hypothesis 2).
3. METHOD
3.1 Participants and Design

A total of N = 56 undergraduate students (42 female and 14 male) from a German university took part in the
experiment. Their average age was 22.73 years (SD = 3.83). They were all enrolled in an introductory course
on research methods and were asked to participate in an experiment to meet the course requirements.
Participants were randomly assigned to two experimental groups (GP = generic prompt, n1 = 28; DP =
directed prompt, n2 = 28). Participants in the GP group received a general prompt to conduct a phase of
preflection on the subsequent problem-solving task. Participants in the DP group received instructional to-
complete-sentences in order to prompt preflective thoughts.
3.2 Materials
3.2.1 Problem Scenario and Prompts
The prompts were embedded in a problem-based self-regulated learning environment where participants were
asked to help their mother who was moaning about dorsalgia and concerned about having a herniated disk. A
partially illustrated article on the spine’s and spinal cord’s anatomy and functionalities as well as the reflex
circuit was used as learning content.
Generic and a directed prompts were developed accordingly. The generic prompt included a general
suggestion for a preactional phase of forethoughts, planning, and activation on the subsequent problem
solving process and according self-regulation mechanisms (please use the following ten minutes to optimize
your upcoming problem solving process through well-thought-out planning and preparation). In contrast, the
directed prompt was designed more specific. In total it contained eleven to-complete-sentences in order to
induce the specific processes of prospective considerations and preactional self-regulation (e.g., activation of
prior knowledge: About the substantive subject area of the problem I already know that…; goal setting: To
achieve the main objective I’ll set the following sub goals…).
3.2.2 Domain Specific Knowledge Test
The knowledge test included eleven multiple-choice questions with four possible solutions each (1 correct, 3
incorrect). Two versions (pre- and post-test) of the domain-specific knowledge test were administered (in
which the eleven multiple-choice questions appeared in a different order). It took about eight minutes to
complete the test.
184
3.2.3 ILS
A German translation of the Index of Learning Styles (ILS; Felder and Spurlin, 2005) was used to assess the
students’ individual learning styles. Each of the four ILS dimensions consists of eleven items. Each item
possesses two response possibilities (a or b) that either correspond to one extreme of a dimension’s
continuum (e.g., active vs. reflective). Felder and Spurlin (2005) report correlation coefficients for the ILS-
scales of three test-retest studies with values between .72 and .87 at a four weeks interval, .60 and .78 at
seven months, and .51 and .68 at eight months.
Regarding the validity of the ILS, it is to report that both exploratory factor analysis with an eight-factor
solution, as well as a feedback analysis of the sample with approval levels of 80 percent for the Sequential-
Global scale and over 90 percent for the economies of scale Active-Reflective, Sensing-Intuitive, Visual-
Verbal seem to prove the validity of the ILS (Litzinger et al., 2007).
3.3 Procedure
First the participants completed the domain-specific knowledge test, a demographic data and the ILS
questionnaire. Then they were introduced to the problem-scenario and the groups received their specific
prompts to preflect. The learning material was handed out 10 minutes afterwards. Participants worked for 30
minutes on the solution of the problem. After a total of 40 minutes for solving the problem, including the
induced phase of preflection, participants were asked to represent their solution of the problem as a written
text. Finally, the post-version of the domain-specific knowledge test was completed.
3.4 Analysis
3.4.1 HIMATT
HIMATT (Highly integrated Model Assessment Technology and Tools; Pirnay-Dummer et al., 2010) allows
an automated analysis of the text-based problem solutions. A detailed description of the seven measures of
HIMATT, which include four structural and three semantic measures, is provided by Ifenthaler (2010). High
reliability and validity measures have been reported for HIMATT (Pirnay-Dummer et al., 2010).
3.4.2 Reference Model
In order to quantify the qualities of the participants’ problem solutions an expert solution was used as a
reference model for our statistical analysis. The text-based reference model was worked out within the same
problem-based learning environment by an orthopedic specialist and apprenticed physiotherapist.
4. RESULTS
Table 1 describes the distribution of learning style tendencies of all four ILS-scales for the 56 participants of
our experiment. The distribution is in line with reports from other studies that used the German translation
(Derntl and Graf, 2009) as well as the English version of the ILS (Felder and Spurlin, 2005).
Table 1. Distribution of learning style preferences
active reflective sensing intuitive visual verbal sequential global
36 20 39 17 47 9 28 28
64.3 % 35.7 % 69.6 % 30.4 % 83.9 % 16.1 % 50 % 50 %
On the declarative knowledge test, participants could score a maximum of eleven correct answers. They
scored an average of M = 3.23 correct answers (SD = 1.53) in the pretest and an average of M = 6.16 correct
answers (SD = 1.68) in the posttest. The increase in correct answers (not normally distributed) was
significant, Z = -6.049, p < .001, r = -.808 (strong effect).
For testing our hypotheses, we computed an ILS main score for each participant using the according
numeric relatives of both hypotheses-relevant ILS-scales, active-reflective and visual-verbal. Then,
185
ISBN: 978-989-8533-12-8 © 2012 IADIS
experimental groups were segmented into three sub-groups via percentiles (GP: active-visual n1a = 9, non-
distinct n1b = 4, reflective-verbal n1c = 16; DP: active-visual n2a = 17, non-distinct n2b = 0, reflective-verbal
n2c = 11). To check hypothesis 1, the qualities of problem solutions were investigated by independent t-tests
respectively Mann-Whitney-tests, if HIMATT measures were not normally distributed.
For the GP group t-tests revealed no significant effect for GRM, t(23) = .424, p = .675; STM, t(23) =
1.344, p = .192; GAM, t(23) = 1.463, p = .157; PPM, t(23) = -.729, p = .473. Additionally, the Mann-
Whitney-tests showed no significant effect for SFM, U = 60.5, p = .514; CCM, U = 63.0, p = .610; BPM, U =
63.0, p = .610. For the DP group t-tests revealed no significant effect for SFM, t(26) = -.391, p = .699; GAM,
t(26) = -1.046, p = .305; CCM, t(26) = -.445, p = .660; PPM, t(26) = -.100, p = .921; BPM, t(26) = -.365, p =
.718. Additionally, the Mann-Whitney-tests showed no significant effect for GRM, U = 68.0, p = .225; STM,
U = 64.5, p = .172. Accordingly, no significant differences of the quality of problem solutions between
reflective-verbal and active-visual learners could be identified in either experimental group. Therefore,
hypothesis 1 is rejected.
Secondly, we assumed that the increase in declarative knowledge for subjects with a preference for
reflective and verbal learning is superior compared to others. For the GP group (knowledge increase not
normally distributed) the Mann-Whitney-test revealed no significant differences between reflective-verbal
and active-visual learners, U = 51.5, p = .238. For the DP group (knowledge increase normally distributed) t-
test revealed no significant differences between reflective-verbal and active-visual learners, t(26) = -1.083, p
= .289. Accordingly, we reject hypothesis 2.
5. DISCUSSION
A consideration of the orientation of the applied HIMATT measures (structural vs. semantic) shows that in
average the subject performed ways better on the structural than on the semantic level of the posed problem.
This suggests that, although a minimum of necessary complexity could be obtained, the subjects were not
able to use sufficiently correct concepts as well as to create semantically correct relationships within their
solution to the problem. For higher quality results they should have increased their usage of correct concepts
and propositions.
Contrary to our hypotheses, the more reflective and verbal study groups did not differ significantly from
the more active and visual subjects, neither in the quality of their developed problem solutions nor in their
increase in declarative knowledge. Accordingly, we conclude that the matching of our instructional scaffolds
respectively both the suiting generic and directed prompts did not work beneficially for their target group.
Therefore, either the prompts or the dimensions of the Felder-Silverman learning style model have to be
questioned in terms of validity.
Regarding the first two critical aspects, we have to mention that our experimental design with its strict
time schedule could be one reason for the fact that the consideration of learning styles did not transfer into
higher quality problem solutions and an increase in declarative knowledge. But at the same time, one could
assume that a matching respectively mismatching between the type of the developed self-regulated problem-
solving environment as well as the embedded instructional support (generic/directed prompts) on the one
hand, and the present learning styles (either more distinctive reflective-verbal or active-visual) on the other
hand, still would result in differences regarding our dependent variables.
6. CONCLUSION
Given that generic and directed prompts have been found to be an important instructional method for
self-regulated learning in problem-solving environments (Ifenthaler, 2012, Davis, 2003, Wirth, 2009), we put
in question if learners really benefit from taking learning/cognitive styles into account when designing
instruction. Based on our empirical finding, one would have to answer »No« to that question.
186
REFERENCES
BAJRAKTAREVIC, N., HALL, W. & FULLICK, P. 2003. Incorporating learning styles in hypermedia environment:
Empirical evaluation. Proceedings of the 14th Conference on Hypertext and Hypermedia, Nottingham, UK. 41-52.
BANNERT, M. 2009. Promoting self-regulated learning through prompts. Zeitschrift für Pädagogische Psychologie, 23,
139-145.
BRANSFORD, J. D., BROWN, A. L. & COCKING, R. R. (eds.) 2000. How people learn: Brain, mind, experience, and
school, Washington, DC: National Academy Press.
BROWN, A. L. 1984. Metakognition, Handlungskontrolle, Selbststeuerung und andere noch geheimnisvollere
Mechanismen. In: WEINERT, F. E. & KLUWE, R. H. (eds.) Metakognition, Motivation und Lernen. Stuttgart:
Kohlhammer.
CASSIDY, S. 2004. Learning Styles: An overview of theories, models, and measures. Educational Psychology, 24, 419-
444.
COOK, D. A., GELULA, M. H., DUPRAS, D. M. & SCHWARTZ, A. 2007. Instructional methods and cognitive and
learning styles in web-based learning: Report of two randomised trials. Medical Education, 41, 897-905.
DAVIS, E. 2003. Prompting middle school science students for productive reflection: Generic and directed prompts.
Journal of the Learning Sciences, 12, 91-142.
DERNTL, M. & GRAF, S. 2009. Impact of Learning Styles on Student Blogging Behavior. 9th IEEE International
Conference on Advanced Learning Technologies. Riga, Latvia.
DEWEY, J. 1933. How we think, New York, Dover.
DÖRNER, D. 1976. Problemlösen als Informationsverarbeitung, Stuttgart, Kohlhammer.
DUNN, R., HONIGSFELD, A., DOOLAN, L. S., BODSTROM, L., RUSSO, K., SCHIERING, M. S., SUH, B. &
TENEDERO, H. 2009. Impact of learning-style instructional strategies on students‘ achievement and attitudes:
Perceptions of educators in diverse institutions. The Clearing House: A Journal of Educational Strategies, Issues and
Ideas, 82, 135-140.
FELDER, R. M. & BRENT, R. 2005. Understanding student differences. Journal of Engineering Education, 94, 57-72.
FELDER, R. M. & SILVERMAN, L. K. 1988. Learning and teaching styles in engineering education. Journal of
Engineering Education, 56, 674-681.
FELDER, R. M. & SPURLIN, J. 2005. Applications, reliability and validity of the Index of Learning Styles.
International Journal of Engineering Education, 21, 103-112.
FUNKE, J. 2003. Problemlösendes Denken, Stuttgart, Kohlhammer.
GILBERT, J. E. & SWAINER, C. A. 2008. Learning styles: How do they fluctuate? Institute for Learning Styles
Research Journal, 1, 29-40.
HAWK, T. F. & SHAH, A. J. 2007. Using learning style instruments to enhance student learning. Decision Sciences
Journal of Innovative Education, 5, 1-19.
HAYES, J. & ALLISON, C. W. 1993. Matching learning style and instructional strategy: an application of the person-
environment interaction paradigm. Perceptual and Motor Skills, 76, 63-79.
HAYES, J. & ALLISON, C. W. 1996. The implications of learning styles for training and development: a discussion of
the matching hypothesis. British Journal of Management, 7, 63-73.
IFENTHALER, D. 2008. Practical solutions for the diagnosis of progressing mental models. In: IFENTHALER, D.,
PIRNAY-DUMMER, P. & SPECTOR, J. M. (eds.) Understanding models for learning and instruction. Essays in
honor of Norbert M. Seel. New York: Springer.
IFENTHALER, D. 2010. Scope of graphical indices in educational diagnostics. In: IFENTHALER, D., PIRNAY-
DUMMER, P. & SEEL, N. M. (eds.) Computer-based diagnostics and systematic analysis of knowledge. New York:
Springer.
IFENTHALER, D. 2012. Determining the effectiveness of prompts for self-regulated learning in problem-solving
scenarios. Journal of Educational Technology & Society, 15, 38-52.
IFENTHALER, D. & SEEL, N. M. 2011. A longitudinal perspective on inductive reasoning tasks. Illuminating the
probability of change. Learning and Instruction, 21, 538-549.
JACOBSON, M. J. 2000. Problem solving about complex systems: Differences between experts and novices. In:
FISHMANN, B. & O’CONNOR-DIVELBISS, S. O. (eds.) Fourth International Conference of the Learning Sciences
Mahwah, NJ: Lawrence Erlbaum Associates.
JOHNSON-LAIRD, P. N. 1983. Mental models. Towards a cognitive science of language, inference, and consciousness,
Cambridge, UK, Cambridge University Press.
187
ISBN: 978-989-8533-12-8 © 2012 IADIS
KAVALE, K. A. & FORNESS, S. R. 1987. Substance over style: Assessing the efficacy of modality testing and teaching.
Exceptional Children, 54, 228-239.
KOLB, D. 1984. Experiential learning: Experience as the source of learning and development, Englewood Cliffs, NJ,
Prentice Hall.
LAWRENCE, G. 1994. People types and tiger stripes, Gainesville, FL, Center for Applications of Psychological Type.
LEUTNER, D. & LEOPOLD, C. 2003. Selbstreguliertes Lernen als Selbstregulation von Lernstrategien. Ein
Trainingsexperiment mit Berufstätigen zum Lernen von Sachtexten. Unterrichtswissenschaft, 31, 38-56.
LITZINGER, T. A., LEE, S. H., WISE, J. C. & FELDER, R. M. 2007. A psychometric study of the Index of Learning
Styles. Journal of Engineering Education, 96, 309-319.
MARTIN, S. 2010. Teachers using learning styles: Torn between research and accountability. Teaching and Teacher
Education, 26, 1583-1591.
MORRISON, G., ROSS, S., KEMP, J. & KALMAN, H. 2011. Designing effective instruction, Hoboken, NJ, John Wiley
& Sons, Inc.
NELSON, T. O. & NARENS, L. 1994. Why investigate metacognition? In: METCALFE, J. & SHIMAMURA, A. P.
(eds.) Metacognition: Knowing about knowing. Cambridge, MA: Bradford.
PINTRICH, P. R. 2000. The role of goal orientation in self-regulated learning. In: BOEKAERTS, M., PINTRICH, P. R.
& ZEIDNER, M. (eds.) Handbook of self-regulated learning. San Diego, CA: Academic Press.
PIRNAY-DUMMER, P., IFENTHALER, D. & SEEL, N. M. 2012. Designing model-based learning environments to
support mental models for learning. In: JONASSEN, D. H. & LAND, S. (eds.) Theoretical foundations of learning
environments. 2 ed. New York: Routledge.
PIRNAY-DUMMER, P., IFENTHALER, D. & SPECTOR, J. M. 2010. Highly integrated model assessment technology
and tools. Educational Technology Research and Development, 58, 3-18.
RIDING, R. & CHEEMA, I. 1991. Cognitive Styles—an overview and integration. Educational Psychology, 11, 193-
215.
RIDING, R. & GRIMLEY, M. 1999. Cognitive style, gender and learning from multi-media materials in 11-year-old
children. British Journal of Educational Technology, 30, 43-56.
ROTHWELL, W. J. & KAZANAS, H. C. 2008. Mastering the instructional design process. A systematic approach, 4th
ed. San Francisco, CA, Pfeiffer.
SCHIEFELE, U. & PEKRUN, R. 1996. Psychologische Modelle des selbstgesteuerten und fremdgesteuerten Lernens. In:
WEINERT, F. E. (ed.) Enzyklopädie der Psychologie - Psychologie des Lernens und der Instruktion. Pädagogische
Psychologie. Göttingen: Hogrefe.
SCHMECK, R. R. 1988. Strategies and styles of learning: An integration of varied perspectives. In: SCHMECK, R. R.
(ed.) Learning strategies and learning styles. London: Plenum Press.
SEEL, N. M. 1999. Educational semiotics: School learning reconsidered. Journal of Structural Learning and Intelligent
Systems, 14, 11-28.
SHUTE, V. J., JEONG, A. C., SPECTOR, J. M., SEEL, N. M. & JOHNSON, T. E. 2009. Model-based methods for
assessment, learning, and instruction: Innovative educational technology at Florida State University. In: OREY, M.
(ed.) Educational media and technology yearbook. New York: Springer.
SNIDER, V. E. 1992. Learning styles and learning to read: A critique. Remedial and Special Education, 13, 6-18.
WIRTH, J. 2009. Prompting self-regulated learning through prompts. Zeitschrift für Pädagogische Psychologie, 23, 91-
94.
ZIMMERMAN, B. J. 1998. Developing self-fulfilling cycles of academic regulation: An analysis of exemplary
instructional models. In: SCHUNK, D. & ZIMMERMAN, B. J. (eds.) Self-regulated learning: From teaching to self-
reflective practice. New York: Guilford Press.
ZIMMERMAN, B. J. 2000. Attaining self-regulation: A social cognitive perspective. In: BOEKAERTS, M., PINTRICH,
P. R. & ZEIDNER, M. (eds.) Handbook of self-regulation. San Diego, CA: Academic Press.
ZIMMERMAN, B. J. 2008. Investigating self-regulation and motivation: Historical background, methodological
developments, and future prospects. American Educational Research Journal, 45, 166-183.
188
ASSESSING STUDENT LEARNING ONLINE:

OVERCOMING RELIABILITY ISSUES
Stephen D. Arnold
The University of Arizona South
Tucson, Arizona USA
ABSTRACT
Assessing students in online university courses poses challenges to the reliability factor of the measures being utilized.
Some programs have the latitude to incorporate proctored assessments, but this is not always practical in asynchronously
structured courses reaching out across a broad geographic region. This paper explores digital audio and video outputs as
viable student pontification alternatives.
KEYWORDS
Digital, video, online, assessment, multimedia, reliability.
1. INTRODUCTION
Assessment is the process of collecting, synthesizing, and interpreting information in order to make a
decision (Airasian & Russell, 2007). Value in any instructional system comes from assessment: what is
assessed in a course or a program is generally associated with value; what is valued becomes the focus of
activity (Swan et al., 2007). Effective assessment typically includes ongoing “formative assessment”
checkpoints, and end of term “summative assessment.” Instructors signal what knowledge skills and
behaviors they believe are most important by assessing them, while students quickly respond by focusing
their learning accordingly (Swan et al., 2007). The end-of-course assessment method, and more specifically
the requirements that underlie this assessment mode, make a difference (Struyven et al., 2006). In an online,
asynchronous course, whereby the students and instructor do not meet, obtaining reliable assessment
measures becomes more difficult than in a traditional face-to-face class. If exam proctoring is not an option
other innovative means of measuring student learning outcome must be examined. This paper explores the
creation of digital audio and video as a means for students to demonstrate learning outcomes in online
university courses when exam proctoring is not possible. Although it is possible as an instructor to elicit
online quizzes, papers, and projects from students it is important to collect several pieces of information
about the performance being assessed to increase reliability (Airasian & Russell, 2007). Without the exam
proctor possibility, determining who is (and how many are) involved in the submission of the common
assessment items is problematic. As a means to strengthen assessment reliability, and foster students’ creative
engagement, the use of digital audio and video pontifications in online courses at a large USA university will
be examined and discussed as a viable means to foster more reliable assessment outcomes for students and
instructors.
2. ASSESSMENT
In any course measuring the growth students have made towards the objectives is critical to determine the
effectiveness of instruction. As instructors we have to be certain that our efforts are resulting in optimum
outcomes for students. In higher education written exams are often the assessment means of choice due to
large numbers of students and limited instructor time. Final course grades and exams are the most common
189
ISBN: 978-989-8533-12-8 © 2012 IADIS
measures of learning outcomes for seniors across Majors (NSSE, 2010). Scheduled test events tend to
increase students’ study time efficiency (McKenzie, 1979).
2.1 Types
There are many categorizations of assessment. First it is possible to make a distinction between assessment
and test. The first being the process of determining the learning gains, and the second being the instrument or
measurement tool for gathering the data. Some use the terms evaluation and assessment in a similar manner.
It is important to consider the time-based snapshots of learning which are addressed by on-going incremental
measurements (formative), and culminating end of term measurement of the entire term (summative). When
looking at the instrument for gathering measures of learning itself there are two achievement test output
categories to consider: written (conceptual, selected response), and performance (applied, task-oriented).
Constructed response tests (short answer, essay), on the other hand, blur the boundaries between written and
performance. They are written, and sometimes performance. “Like stories, reports, or show-your-work
problems, essays and extended-response test items are important forms of performance assessments (Airasian
& Russell, 2007). Educators’ perspective on what constitutes performance varies, but the existence of three
common characteristics prevails in identifying assessment as performance: (a) multiple evaluative criteria;
(b) pre-specified quality standards; and (c) judgmental appraisal (Popham, 2011). Bloom’s taxonomy of
instructional objectives correlates closely to performance when students are expected to recall and apply
information in an actual task. In some cases schools are requiring student exhibitions, culminating projects,
experiments, solving of realistic math problems, and various other demonstrations of competence (Slavin,
2012).
2.2 Online Options

In online courses the assessment options are impacted by the student population demographics. Although
students take online courses frequently at the campus of which they are a resident due to schedule restraints,
remote students pose the greatest assessment challenge. Meeting with students online in videoconferencing is
one way instructors assess students informally through interactive dialogue. Oftentimes students indicate
dissatisfaction when instructors of online courses offer them synchronously at scheduled times due to their
time/place-bound circumstances. If a student is beyond a reasonable commuting distance or has set hours of
employment it is difficult to attend any scheduled class whether face-to-face or online. In the asynchronous
delivery scenario the time can be more forgiving, but the options for assessment are more limited. It is
possible to develop selected response (multiple-choice, true/false) and constructed response tests in course
management systems, although reliability is an issue.
2.3 Reliability
When circumstances allow having a test proctored for online students will increase its reliability. Reliability
in this instance refers to consistency over time. If an instructor were able to administer a test to the same
student at different times ideally the results would be the same. When a test is administered, aspects related to
the test construction itself, the student, graders, and various circumstances surrounding its administration
could cause the results to be inconsistent (Slavin, 2012). One major factor that can affect the reliability of a
non-proctored online exam is its equivalence to a take-home or open book test. In a face to face or proctored
scenario the test-taker is being monitored. With the take-home scenario used in standard face-to-face classes
it is possible to have supplemental in-person exams for context. Furthermore, the instructor has constant
in-person engagement with the students which can provide an opportunity for oral engagement on the subject
matter. In the fully online situation, however, it is difficult to gauge who or how many are working on the
same exam. Given such test reliability is compromised. In one study (Watson & Sottile, 2010) students
reported equitable rates of cheating in face-to-face classes as compared to online classes, but 5.2% more had
someone else give them answers during an online class quiz or test. It is possible students perceive this as
acceptable collaboration, not cheating.
190
3. THE COURSE
The events of this study transpired in a semester-based Technology for Educators course. It is organized in
accordance with the Quality Matters guidelines for online instruction, and by particular themes: (a) Getting
started; (b) The Internet; (c) Computer-based software; (d) Audio; (e) Interactive hypermedia; and (f)
Multimedia.
3.1 Participants
In the spirit of action research “to address an actual problem in an educational setting….practical issues that
will have immediate benefits for education” (Creswell, 2008) an online Technology for Educators class was
analyzed in progress to understand assessment options in the online class scenario better. The course is
required for all pre-service teachers in the Elementary Education program of a large southern USA
university. One online section (N=21) was analyzed in-depth by the researcher within the usual scope of its
delivery. The students are comprised of non-traditional older students, primarily female, and 24% Hispanic.
The Elementary Education Program is a transfer program; therefore students join the program in their third
year of a bachelor’s degree, and with a foundational development in elementary education.
3.2 Procedures
As the basis of the content in this paper, the researcher analyzed methods for students to demonstrate
competency of learning objective while increasing the reliability of assessment. The analysis is based upon
the formative and summative project outcomes of students in the undergraduate Technology for Educators
courses in a teacher education program. For the final exam students are given a choice of either writing a
paper, or developing a comprehensive video. If they choose the video they may create it as either an
individual or group project (self-selected groups). There are also other key assignments throughout the
semester that have students expound upon their growth in the course content through various other digital
outputs that incorporate text, static images, audio, video, or a combination. Group projects have the means to
provide increased student understanding of content and instructor related advantages including multiple
perspectives and pooled efforts (Young & Henquinet, 2000). From an instructor’s standpoint, presentations
provide an alternative means for students to demonstrate their competency vested in a culminating course
project (Arnold, 2010). As a means to capture the “presentation” component of a face-to-face class in the
online course medium the final project had the presentation element at its core. The process of presenting acts
as reinforcement for learning that will oftentimes motivate presenters towards adequate preparation and
information grounding (Arnold, 2010). Students are able to demonstrate meaningful, multidimensional tasks
via this authentic assessment (Montgomery, 2002). The course at the heart of this study, Technology for
Educators, was broken down into five modules, each with three weeks devoted to a certain theme. Given that
the course is primarily for pre-service teachers the focus was on pedagogy and using technology to support
the standards-based subjects in the classroom. Theoretically and practically, teaching requires substantive
merging of content, pedagogy, and technology knowledge (Roblyer & Doering, 2010). Each of the modules
had an overlying technology-based theme with multiple technologies addressed, substantial readings,
academic content standards tie-in, pedagogical foundation, and emphasis on integration.
3.3 Technology Integration

Over time computer programs and online applications wax and wane. Many withstand the test of time, but
undergo significant repackaging. Given the continual transitioning of computer technology the focus of the
Technology for Educator courses discussed in this paper has been on helping students become more
confident in exploring technology while focusing on “why” we need and choose technologies over “what”
and “how.” For instance the first module’s theme is “online.” In this theme students explore the topics digital
information integrity, information literacy and multiple issues pertaining to online information exchange:
legalities, safety, equity, societal, and reliability. During each module students use and create comprehensive
projects with multiple cloud and computer based technologies while exploring an instructional
191
ISBN: 978-989-8533-12-8 © 2012 IADIS
delivery/e-learning concept such as podcasting. Their culminating project for this module, as an example,
would result in a precise public service announcement embedded in a Wiki page and uploaded to an RSS
feed site (iTunes U), which demonstrates students’ growth (or lack of) from the multiple readings (textbook
& journal articles), instructor created multimedia presentations, and digital technology explorations. At each
step of the way students are reminded to critically analyze through a K-12 teacher’s lens.
3.4 Input and Output

It is possible to look at the process of developing one’s content knowledge as “Input” and demonstrating
what one knows as “output.” More commonly this is referred to as learning and assessment. Hunter (2004)
equated the terms with input of information into the students’ cognitive learning processes, and output of
information in a mastery of the learning objective sense so that proper assessment may occur. Output is also
associated with the active process of learning whereby the process of output draws heavily upon the content
knowledge students experienced through the input process reinforcing the learning (Arnold & Moshchenko,
2009). When students are given the opportunity to produce a tangible product or demonstrate something to an
audience, their willingness to put forth quality increases (McTighe, 1996). Given the complexities of reliable
assessment in the online environment, and that the students of the Technology for Educators course were
taught the skills to create podcasts and video-based digital stories alternate assessments termed Technology
Integration Pontifications were developed to analyze student growth in the course subject-matter.
3.5 Performance Pontification

Digital video editing is well suited for providing authentic, meaningful reflective experiences for teachers
(Calandra et al., 2009). If it is more pointed in its output with specific criteria then it becomes a viable
assessment tool. When constructed by the students who are being assessed, and as participants in the video,
the instructor will be able to analyze the video for key levels of pontification pertaining to the course and
assessment objectives as the following pontification assignment summary illustrates.
In this final project you will create a fun digital story on a subject of interest that would be covered
in an elementary classroom and that supports a content area. It may be anything from science (rocket
propulsion for instance) to a social message (wash your hands frequently to reduce germs), or any
other subject you would expect students of your favorite grade-level to learn (look in the content
standards for a grade-level and subject of choice to identify a specific Performance Objective). In
the spirit of the “reality” TV mash-ups (i.e., “Survivor” for instance where the program shows the
tribes in action then cuts away to an individual sharing his/her perspective on that action in an
interview scenario) I would like you to intersperse yourself into the video as the teacher giving your
perspective regarding the use of technology in the learning/teaching process & with your chosen
subject, while teaching the viewer about a chosen topic (i.e., the earth’s rotation/tilt & seasons).
Have fun with this and get kids involved if possible! You may do this as a group project with other
students. (S. Arnold, personal communication, April 23, 2012)
In order to discern specific concepts critical to the outcomes of the course further detailed criteria were
included. Some pertained to the technology skills whereas others were targeting educational technology and
integration with the elementary school subject-matter concepts. The following are abbreviated samples of
measurable objectives included in the project:
• Refer to and include specific Educational Technology supportive content from at least 6 journal
article or textbook sources that were assigned during the term
• Devote about 1/3 of your video to talking about integrating technology into the classroom, and the
remainder to teaching about a specific topic in a grade-level and subject of choice. Be sure to
combine them so it does not appear like two separate videos.
• Include at least 3 separate motion video clips of yourself talking about integrating technology into
teaching & learning.
• Include at least 2 separate audio clips of yourself talking about images, 3rd party motion video clips,
technology integration, or explaining visual examples of the subject matter.
• Connect with and identify multiple standards: Information literacy, NETS*T, state academic content
standards, and state educational technology standards for students.
192
• Make the presence of each group member equitable and evident throughout the video
• Demonstrates competency with the following technologies/technology processes: Movie Maker,
Audacity, online file conversion, iTunes, YouTube, ID Tag Editor, synchronized and overlapping
soundtrack and narrations.
• Effective integration of still images, motion video, text slides, overlays, soundtracks, and narrations.
• Important elements of a presentation: Introduction, body, and conclusion
On a smaller scale, and in a similar manner students were given a reading response assignment whereby
they had to create an audio narrated hypermedia presentation (PowerPoint) in which they identified key
points made in the readings as text on the slides and discussed them in audio format. Having met with each
student individually at the beginning of the semester in a videoconferencing site, and requiring students to
post audio introductions in their e-portfolios the instructor was familiar with students’ voices. Given such it
resulted in more personable assessment than written papers. A couple of drawbacks, especially pertaining to
use in the online environment, include devising a systematically reliable video evaluation means, and the
technological requirements for developing a video that represents one’s content development. Validity is also
important to this type of assessment; does the video allow an instructor to measure the conceptual knowledge
that needs to be measured? Identifying the expected outcomes wasn’t really a problem, but some students
opted to read from scripts which can leave the evaluator wondering if the presenter is engaged or simply
reading information that is not internalized.
3.6 Evaluating Audio & Video

From the grading perspective there is still a disparity in reliability from one instructor to another. Multiple
teachers grading the same essay paper will assign grades ranging from A to F with some teachers making few
to no comments or marks on the papers, but instead just producing a grade (Brimi, 2011). As noted above
there is a convergence of technology use skills, technology integration with subject-matter propensity, and
any given number of subtopics pertaining to educational technology covered in the course that must be
weighed when evaluating a video produced by students of a Technology for Educators course. Time is a
critical element in the analysis, but quality is the most decisive in determining if students are pontificating
about the concepts covered in the course. In the scope of this analysis most students were able to expound
upon their chosen topic (a science lesson on volcanoes for instance), but most commonly underdeveloped
their connection of the lesson topic to their use of technology to demonstrate it, or other examples of
technology that would further support the teaching of the lesson. The next confounding factor that tended to
affect students results, whether audio or multimedia, was the technology medium being used for the output.
3.7 Technological Factors

Early in the course the technology skillsets were more limiting to the quality of course concept-infused
outputs than later in the term. Given such the course was structured with less complex technological
components in the beginning; create a Wiki site via a Web content management system for instance.
Week-by-week new technologies are introduced. During the first three-week module students are asked to
record and embed a simple wave file into the Wiki, convert a Wav to Mp3 via a “free” online media
converter, and create a simple Google presentation to embed. During subsequent modules as students’
efficacy climbs they are directed toward more complex technological developments such as multi-tracked
audio and video outputs using programs that balance user friendliness, effectiveness, and relatively free
availability. These are characteristics that are likely to encourage pre-service teachers to continue using
technologies adequately when they transition to in-service status; a time that has many reeling from the steep
learning and time commitment curve common during the first two years. Toward the end of the term students
in the Technology for Educators course are pushing some of the low-end technologies to their limit, which
inevitably impacts their output and perception of technology.
193
ISBN: 978-989-8533-12-8 © 2012 IADIS
3.8 Student Media Preference

When students were asked if they preferred demonstrating content they’ve learned in the course through
audio or video output (reading responses, interactive hypermedia, videos, etc.) over writing a paper on the
same 81% strongly agreed, and 19% agreed. Given the number of technical glitches that were communicated
during the term it is curious that no students indicated preference for writing a pontification paper over
creating the video. Perhaps as Roblyer and Doering (2010) point out technology can improve student
motivation, attitude, and interest in learning. In an end of course improvement evaluation the instructor
administers to students a rank-order question indicated that students prefer audio and video enriched
technologies (see Fig. 1). The question only analyzed the larger project outputs without sub-analysis of the
smaller technologies that most often fed into the larger projects. In the ranks for each project/media type
students identified Podcasting/Audacity (88) as their optimum output medium with E-Learning/PPT (81) and
Video Pontification/Movie Maker (77) close behind in that order. In discussions between the instructor and
students most seemed more enthusiastic about the outcome of their E-Learning and video activities, but the
researcher believes the higher rate of technology glitches and higher complexity level of the assignments
associated with their development led to lower ranking than the Podcast. Large file sizes, program freeze-ups,
conversion to an iTunesU compatible mp4 file format, and student self-consciousness about presenting in the
video were common concerns voiced by students during the latter part of the term devoted to the multimedia
projects.
Figure 1. Class rank-order preference of major projects
3.9 Student TPACK Perception

Students (N=21) were given an additional questionnaire, the Technological Pedagogical Content Knowledge
Alignment Perception Scale (TPACKAPS) at the beginning of the semester, and again at the end. Each
student was asked to rate various components of the course (readings, discussions, papers, and media) on a
scale from 1 to 10 for technology, pedagogy, and content knowledge emphasis (1 = none; 10 = primarily).
Paired-samples t tests were conducted to determine whether students’ perception of the course components
varied upon completion of the course. The results indicated that the pre and post means (Table 1) varied
significantly at the p < .01 level in students’ perceptions of videos and podcasts for technology, pedagogy,
and content knowledge. The positive correlation indicates that students perceive more technology focus in
the beginning, but they perceive more pedagogy and content knowledge in the media projects. When asked if
they felt that media output represented and demonstrated their level of learning in the course with regards to
194
technology, pedagogy, and content knowledge 76% strongly agreed, 14% agreed, and 10% neither agreed nor
disagreed.
Table 1. Student TPACK perception means
Video Podcast
Pre Post Pre Post
Technology 8.67 7.81 8.86 7.62
Pedagogy 3.95 7.91 3.95 8.24
Content Knowledge 4.86 8.48 5.14 8.10
4. EDUCATIONAL SIGNIFICANCE
The results of this study indicate that students prefer multimedia over other types of technology, view digital
video and audio as TPACK rich media capable of demonstrating their competencies. Given that the course
utilized in this study is heavily infused with large doses of pedagogy and content knowledge instruction in
addition to the technology literacy skill development a balanced TPACK approach is modeled for the
students. Furthermore, students are challenged to create outputs that equitably merge each TPACK
component.
Aside from an instructional and learning tool video has been around for many years as a formative
assessment, feedback, and planning tool. Common uses in this realm have included recording oneself giving
a speech or presenting a student teaching lesson, real-time and post-game sports analysis, diagnosis of
medical conditions or behaviors, pre and posttest analysis of research subjects, and anything that requires a
comparative stop-action, archival capability. From an instructor’s point of view the digital audio and video
output can offer a creative and visual dimension not offered in print. From the student perspective whereby
they are interjecting an audible or visual presence in the media it is typically more customary as a self-
assessment tool. Video tools are not uncommon as a means to teach content to others, even with the self in
the visual mix. Through the use of audio and video students are able to solidify their learning due to the
increased cognitive processing needed to develop quality output. In addition students will encounter added
motivation due to prospect of having a novel means to demonstrate their competency. As a demonstration of
what one has learned based upon engagement in a course, however, digital audio and video in the online class
environment is an under-utilized and viable output. It warrants further analysis as an output tool not only in
Educational Technology, but in less technology-focused disciplines as well.
REFERENCES
Airasian, P.W., & Russell, M.K. (2007). Classroom assessment: Concepts and application (6th ed.). New York, NY:
McGraw-Hill.
Arnold, S.D. (2010). Student e-cooperative: Computer mediated group projects. In the International Technology,
Education, and Development Conference Proceedings CD (ISBN: 978-84-613-5538-9). Valencia, Spain: INTED.
Arnold, S.D., & Moshchenko, M. (2009). Technology input versus input and output: Does it result in learning differences
among elementary school students? In C.D. Maddux (Ed.), Research Highlights in Technology and Teacher
Education 2009 (pp. 1-9). Chesapeake, VA: Society for Information Technology & Teacher Education.
Brimi, H.M. (2011). Reliability of grading high school work in English. Practical Assessment, Research & Evaluation,
16(17), pp. 1-12.
Calandra, B., Brantley-Dias, L., Lee, J.K., & Fox, D.L. (2009). Using video editing to cultivate novice teachers’ practice.
Journal of Research on Technology in Education, 42 (1), pp. 73-94.
Creswell, J.W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research
(3rd ed.). Upper Saddle River, NJ: Pearson Education, Inc.
Hunter, R. (2004). Madeline Hunter’s mastery teaching: Increasing instructional effectiveness in elementary and
secondary schools. Thousand Oaks, CA: Corwin Press.
McKenzie, G.R. (1979). Effects of questions and test-like events on achievement and on-task behavior in a classroom
concept learning presentation. Journal of Educational Research, 72(6), pp. 348-351.
195
ISBN: 978-989-8533-12-8 © 2012 IADIS
McTighe, J. (1996). What happens between assessments? Educational Leadership, 54 (4), pp. 6-12.
Montgomery, K. (2002). Authentic tasks and rubrics: Going beyond traditional assessments in college teaching. College
Teaching, 50 (1), pp. 34-39.
National Survey of Student Engagement [NSSE]. (2010). Major differences: Examining student engagement by field of
study—annual results 2010. Bloomington, IN: Indiana University Center for Postsecondary Research.
Popham, J.W. (2011). Classroom assessment (6th ed.). Boston, MA: Pearson Education, Inc.
Roblyer, M.D., & Doering, A.H. (2010). Integrating educational technology into teaching (5th ed.). Boston, MA:
Pearson Education, Inc.
Slavin, R.E. (2012). Educational psychology: Theory and practice (10th ed.). Boston, MA: Pearson Education, Inc.
Struyven, K., Dochy, F., Janssens, S., Schelfhout, W., & Gielen, S. (2006). The Overall Effects of End-of-Course
Assessment on Student Performance: A Comparison Between Multiple Choice Testing, Peer Assessment, Case-
Based Assessment and Portfolio Assessment. Studies in Educational Evaluation, 32, pp. 202-222.
Swan, K., Schenker, J., Arnold, S. & Kuo, C-L. (2007). Shaping online discussion: Assessment matters. E-mentor, 1 (18),
at: http://www.e-mentor.edu.pl/_xml/wydania/18/390.pdf
Watson, G., & Sottile, J. (2010). Cheating in the digital age: Do students cheat more in online courses? Online Journal of
Distance Administration, 13 (1). Retrieved from http://www.westga.edu/~distance/ojdla/spring131/watson131.html
Young, C.B., & Henquinet, J.A. (2000). A conceptual framework for designing group projects. Journal of Education for
Business, 76 (1), pp. 56-60.
196
U-ALS: A UBIQUITOUS LEARNING ENVIRONMENT
Sandra Dutra Piovesan1, Liliana Maria Passerino1 and Roseclea Duarte Medina2
1
Universidade Federal do Rio Grande do Sul
2
Universidade Federal de Santa Maria
ABSTRACT
The diffusion of the use of the learning virtual environments presents a great potential for the development of an
application which meet the necessities in the education area. In view of the importance of a more dynamic application
and that can adapt itself continuously to the students’ necessities, the “U-ALS” (Ubiquitous Adapted Learning System)
was proposed and developed. This system was built based on the learning virtual environment Moodle and on the Module
Mle-Moodle, available in a Cloud Computing infrastructure and it has as a main purpose the adaption to the student’s
computational context, glimpsing technical characteristics like the environment adequacy to the user’s speed connection.
The obtained results show the feasibility of working with systems that are sensitive to the context, bringing
improvements in the students’ access to the materials and tools.
KEYWORDS
Ubiquitous Learning Virtual Environment, U-Learning, U-ALS, Adapted Environments, Cloud Computing.
1. INTRODUCTION
The use of the Virtual Learning Environments (VLEs) in the academic environment has become very
common. The use of these environments with the support and help in the teaching and learning processes
make the VLEs an important pedagogical resource (RIBAS, 2009). In this case, making the environment
adequate to the student, considering its individual characteristics is a necessity today. In this field the
ubiquitous learning (U-learning) is inserted, which enables the access to the educational resources with total
mobility and adaptation of the system to the students’ computational context. In this context, this paper seeks
for proposing a new application for these environments where the students, besides having the facilities of
mobility and adaptation of the device through the Mle-Moodle, they will have an environment which will
adjust itself to its computational context, making the materials adequate to the different speed connection,
available in a Computational Cloud infrastructure which guarantees availability and access through different
devices.
The Cloud Computing emerges as a technology to improve and make the use of computational resources
more efficient, through characteristics like availability, elasticity and the adaptability of services, where the
user will be able to access his programs remotely and he can have a larger capacity of storage and processing
available, without the necessity of owning more expensive equipments, because his data will be available in
the cloud. In this computation model, the technology resources are allocated on a scalable way, this way
these services are offered to external clients through technologies via Internet (CEARLEY, 2010).
According to David Cearley, the clients don’t need to know how it works; they will simply be able to use
the offered services. Thus, Cloud Computing is a model in which the computation (processing, storage and
software) is somewhere in the net and it’s accessed on a remote way (CEARLEY, 2010).
The increasing of size, complexity, the number of users in the internet problem, as well as technological
diversity that the same use to access the net brings the necessity of treating each user on a specialized and
individual way, identifying the profile and developing applications in virtual environments able to adapt
themselves.
The existent VLEs own static platforms, enabling only the selection of the contents and the tolls that will
be used by the teachers, without considering the students’ individual characteristics that will use the
environment.
197
ISBN: 978-989-8533-12-8 © 2012 IADIS
The trend to the VLEs’ evolution is to evolve to aggregation tools which seem to be able to help to
materialize the PLE (Personal Learning Environment) concept (OLIVER AND LIBER, 2011). The central
idea of PLE is to give the student or, more generically, to the learner, a personal space where it adds the
contents that are useful, produced by others, but also the ones that it produces itself, and that this way
provides according to its interest. In other words, instead of using a designed space similarly to a group of
students, as it happens in the traditional VLEs, it creates its own space as well it keeps it under control. This
evolution has been referenced as the transition from a “one for all” system to “one for me” systems and it
permits to highlight the control question, being the learner able to determine his own choices and action
course (HARMELEN, 2006).
From the researches done through a speed measurer integrated to the VLE Moodle and to the Module
Mle-Moodle of the X School (XX), we noticed through the register of accesses of the environment for
desktops and of the accesses in the environment for mobile devices, the several ways that the students can
access the environment, through mobile devices or desktop computers and the great range of connection
speed, showing that the students’ computational context bring a problem in the usage of the VLEs, because
with lower speeds of connection problems of access to bigger files arise, because it takes them a long time to
be loaded, transferred or open and to use tools that need a larger broadband internet connection like the chats.
This work makes the VLE Moodle dynamic, with characteristics of a software u-learning through the
creation of a module that can identify the student’s computational context when he enters into the course. The
module identifies the student’s connection speed and adapts the contents and the tools, that with the module
Mle-Moodle brings to the environment the characteristics of availability, mobility and sensibility to the
context. With this work, the VLE Moodle and the Module Mle-Moodle became not static and didn’t serve as
simple repository of information where the teachers put their subjects and they can just select the tools that
they will use to become an environment that identifies user’s variables that influence in the way that the
teachers access the environment and in its usage. The Science Computer usage in the Clouds brings the
advantage that it’s not necessary to keep the hardware infrastructure, enabling the increase of the storage
capacity, memory and processing because of the increase in the number of students, dispensing the
maintenance of the TI staff to keep, update and reconfigure the environment according to the necessities. In
the education area the Science Computer technology usage in the Clouds justifies itself by the fact that the
informations can be accessed from anywhere, through the Internet, the elasticity of available resources and
the wide access through personal computers, smart-phones or PDAs.
For better comprehension of the research and the results found, this article is structured in six chapters,
being the second chapter the presentation of some concepts referent to the cloud computing; in the third
chapter are presented definitions about mobile VLEs; in the fourth chapter are presented some correlated
works; in the fifth chapter is presented the methodology and the development of the Moodle U-ALS; in the
sixth chapter are presented the results and discussions.
2. CLOUD COMPUTING
Cloud Computing is a new trend of technology that intends to be global and to provide services to people that
go from the final user that hosts personal documents in the Internet to companies that outsource all the TI
infrastructure (Technology of Information) (BUYYA el al, 2009). The Cloud Computing is justified by the
fact that as the software resources as the hardware ones, get obsolete in a short period of time, making the
computing platforms usage of third ones the ideal solution to the infrastructure problems. To use their
systems the user doesn’t need high computing resources, being of software or hardware, decreasing the
machine purchase cost (MACHADO and MOREIRA, 2010).
The infrastructure of the Cloud Computing environment is usually formed by several physical machines
connected by a net. Each machine has the same configuration of software, but they can range the hardware
capacity related to the processing, storage and memory power. Inside each physical machine there are a
variable number of virtual machines or nodes in execution, according to the physical machine (SOROR et al,
2010).
Nowadays there are several technologies that permit the creation of a new infrastructure of cloud
computing, among them: Amazon Web Service, Google App Engine, OpenNebula, Nimbus e a plataforma
Eucalyptus, chosen for this work.
198
2.1 Eucalyptus
It’s an open code infrastructure that permits the creation of a compatible infrastructure for the users can try
the Cloud Computing (LIU, LIANG and BROOKS, 2007). The architecture of the eucalyptus is simple,
flexible and modular and it also has a hierarchical conception that shows the common resources of the
environment.
An installation in Eucalyptus cloud can add and manage resources from only one cluster or several ones.
The cluster is a group of machines connected to the same LAN. It’s possible to have one or several instances
of a NC (Node Contrller) in a cluster, and every one of them manages the instantiation and the termination of
virtual instances (IBM, 2011).
According to Machado (2010), the Eucalyptus has as a purpose to help the research and the development
of technologies for Cloud Computing, with simple implementation using tools for administration and help to
the users’ and system management, with the ability of setting several clusters.
3. VIRTUAL ENVIRONMENT AND MOBILE LEARNING

In the last decades, we have been experienced the appearance of a connected mobile society, with a variety of
information resources, technologies and ways of communication available. Before the advancing that the
Mobile Computing is achieving, is the fact that, the usage of mobile devices is reaching all kinds of users and
usages, among them the usage in VLEs (MOZZAQUATRO, 2010).
The Moodle is a software of orientation Open Source (free), its development objects the management of
learning and collaborative work in virtual environment permitting the creation and the management of online
courses, work groups and learning communities. In Brazil, the Moodle was approved by MEC as platform for
Distance Education, which will be able to be adopted by any institutions that want to apply this kind of
teaching. The Moodle presents several resources, like discussion forum, diaries, glossaries, tasks, chats,
questionnaires that can be selected by the teacher/manager in a way to create a more flexible learning
environment, that attend to the pedagogical goals and to the students’ necessities, providing a module to the
access via mobile device too (FRANCO, 2010).
The possibility of personalizing the environment where the student interacts in the teaching search,
helping all the ones involved in the teaching/learning process to get their goals, it makes the VLE dynamic.
An important factor during the student’s accompaniment process is to be aware about the context where the
student is acting, enabling the adaptation of teaching strategies according to the specific student’s reality,
enabling an adaptive support, where each student will have their characteristics referent to the environment
and to the presented contents. This adaptation of the environment is characteristic of the Sensitive
Environments to the Contexts that are presented.
3.1 Environments Sensitive to the Context

Sensibility to the Context refers to all that happens around the user, and that influences the way as he
interacts with the environment and with the other people. To the ubiquitous computing, sensibility to the
context is similar, where computing systems can perceive the context and interact according to him
(PERNAS et al, 2009).
Sensibility to the context refers to the capacity of a class of systems to the usage of the contextual
information to offer of better services to the user, in a flexible and manageable way (DEY et al, 1999). A
system that is sensitive to the context is able to extract, interpret and use the contextual informations and to
adapt its functionality to the current context of usage, to provide services to a person particularly, place, time,
event, etc (BYUN and CHEVERST, 2004).
Each student accesses the VLE under determined conditions. The student can be at home or in another
environment, he can use a personal computer or a mobile device, he can have knowledge acquired from other
sources or having no knowledge about the content, he can have done part of the activities or done none of
them (PERNAS et al, 2009). So, each student has his situation characterized and too individual, so to
determine these variables that can interfere directly in the student’s teaching is a very important task.
199
ISBN: 978-989-8533-12-8 © 2012 IADIS
3.2 U-Learning Environments

To Chiu (2008), u-learning is a system aware of the context and that can fell the student’s informations and
the informations around the students in the real world, and then can offer personalized services. So, the
students can learn the knowledge, abilities and problems and be able to solve while interacts with the real
world through authentic scenarios.
To Weiser (1993), the computers should be embedded on a implicit way to the user’s environment, with
human-computer interaction on a non-intrusive way, without the artifacts usage imposition like keywords and
remote-controls, being this one closer to the way that the human-been gesticulate, talk or write to
communicate.
U-learning environment, in counterpoint to the traditional VLEs that usually are static and own contents,
structures and static presentations, they object to provide an environment with functionalities that permit the
adaptation of this environment to the specific situation lived by the user in every interval of time, thus, a very
innovative propose.
In the educational area, the focus of this work, many virtual environments of learning are not more than a
static repository of content, with the same materials, structures and presentation to all the students
(GASPARINI et al, 2009).
4. WORKS CO-REPORTS
Since 1994 we notice the initiatives of software development with context treatment, we can mention: the
Shopping Assistant (1944), the XeroxParc (1996), the Cyber guide (1996), the Conference Assistant (1999),
the Lookout (1999) and the Campus Aware (2002).
The SEDECA objectified to identify and to adapt the VLE MLE Moodle to the student’s cognitive style
through a system created to diagnosticate learning styles. The analysis of these categories permitted to define
indicators that enabled the adaptation of the learning virtual environment MLE Moodle to these different
cognitive styles to the adaptive hypermedia. The materials and proposed activities in the adapted
environment were presented according to the four cognitive styles that highlighted the most in the research:
Holist, Serious, Divergent and Reflexive (MOZZAQUATRO, 2010).
The Adapt Web (Adaptive Teaching-Learning Environment on the Web) is an AS (Adaptive System) of
EAD based on the web, open source. The Adapt Web has the purpose of adapting the content, the
presentation and the navigation according to the user’s profile. Its adaptation is supported by the creation of a
student’s flexible model (user’s model), where, for each student are saved personal informations like his
background, knowledge, preferences, navigational historic and technological resources (PALAZZO et al,
2008). The educational contents are organized by a hierarchical structure of concepts, establishing criterions
of pre-requirement. The structure is defined during the authorship level and it is stored in the XML format
(Extensible Markup Language). These XML documents pass by an adaptation before being presented to the
students. The adaptation occurs as in the content, as on the interface and in the navigation (PERNAS et al,
2009).
5. METHODOLOGY AND DEVELOPMENT OF THE U-ALS MOODLE

Talking about the nature of the research, this paper presents an applied research, because seeks for the
solution of a specific problem and about the technical procedures used, the research is classified as
experimental research, because it selects variables that are able to influence the studied goal (SEVERINO,
2000).
The general goal of the U-ALS Moodle development was to provide, in a Cloud Computing
infrastructure, an adaptive environment as to desktop computers as to mobile devices providing an
environment adequate to the student’s computing context, through the adaptation of the content and of the
available interface, according to the speed connection.
The proposed architecture presents an organization in two distinct layers, the Interface Layer and the
Communication Layer, more the integration with the DBMS (Data Bank Manager System). On the Interface
200
Layer, where the student accesses the content to do the activities, the content will be already available
according to the each student’s speed connection. On this layer the contents and the tools are accessed. On
the Communication Layer, the architecture proposes an Adaptation Agent that has the function of treating the
executed informations on the Interface Layer by the students, providing the adequate material.
The treated informations are the speed connection identification, which is stored in the XML file
(Extensible Markup Language). From this the Adaptation Agent modifies the environment for the student to
have an adequate navigation. The adaptation of the environment is done in the provided materials and also in
the tolls that are provided according to the student’s net context, being the variable analyzed by the Agent
and the access speed Variable to the VLE. The Agent is going to communicate with the Data Bank Manager
System (DBMS) and to adequate the environment to each student.
Despite the DBMS saves the inserted informations, the stored variables to the adequacy of the
environment to the student’s profile, considering its speed as IP, connection speed and student’s access date
in the VLE are stored in an XML file, which is saved inside the Moodledata, placed in the Cloud provider
created, and these data are useful for posterior statistic analysis. The student’s speed identification will
indicate what contents and what tools will be provided in his access to the environment.
The Agent that does the content adaptation watches the environment while the student is logged-in, at
each interval of 60 seconds showing the speed on the screen and saving the data. If the student has his speed
reduced, the environment adapts itself to this speed and if the speed increases the Agent also provides
materials and tools to this connection.
The material to be used by the students is stored in distinct categories, one for low speeds and other for
normal and high speeds, making the environment totally personalized, and that the student will only visualize
the ideal files for his connection speed, as well as the adequate tools to his navigation.
The cloud infrastructure provides to the development of the U-ALS, is based on nine computers, with the
following format: a gateway that permits the access to the environment through the net; a provider with the
cloud controller, cluster controller and storage controller functions and; seven machines serving the node
controller, which ones provide the cloud processing. All the machines used in this environment, as node
controllers, own the technology of native visualization in hardware, being six computers powered with Dual
Core processors and one with Quad Core Processor, totalizing the maximum of sixteen cores in the
computing cloud.
The Eucalyptus framework choice was because it is open source and to enable the implementation of
scalable and efficient clouds. Through the virtualization it is possible that several instances of the operational
system be executed simultaneously in only one computer, being possible to control the usage of the CPU,
memory and storage and even to permit that the operational system migrates from a machine to another. In
this paper, the KVM (Kemel-based Virtual Machine) was used to the virtualization. The KVM is native in
the Ubuntu Operational System, being possible to load Windows and Linux clients being used to raise the
instances in the Eucalyptus framework.
The Cloud Computing usage brings the elasticity of resources advantage to VLE, because if the number
of users or the necessity of more space to store more content, it is not necessary the purchase of more
hardware resources like new providers with more space or more power of processing, the necessary resources
are automatically allocated by the cloud, and the reallocation of resources can be done in real time, without
the necessity of stopping in the system.
The computing cloud implantation model is private, and the service model is a kind of SaaS that has as a
purpose to make easier and more accessible the resources providing to the students.
The U-ALS works in a transparent way to the user, when the student enters into the environment, this one
does not present any apparent change, having characteristics of U-learning software. When the student clicks
on the course the system will check its connection speed and it provides the materials that are adequate to
that connection usage. The U-ALS also provides the tools that are showed.
The adapted environment stores the students’ connection speeds and the adapted materials that were
accessed, making the students’ profile study possible, with the possibility of indicating improvements of the
VLE conditions to attend the students’ necessities, and thus influencing in the final quality of the course.
201
ISBN: 978-989-8533-12-8 © 2012 IADIS
6. RESULTS AND DISCUSSIONS

The validation of the U-ALS Moodle was done at X (XX) School, in the subjects about Wireless Technology
and Computer Architecture, where were collected data that will be presented in this paper. The Figure 1
presents the U-ALS Moodle screen showing the users’ connection speed and the adapted materials, in the
Wireless Technology Subject.
Figure 1. U-ALS Moodle with Adapted Materials

The graphic of the Speed Range, of the Figure 2 in the horizontal position, “X” axis, presents the five
lanes of speed divided to present the students’ range of speed who accessed the environment and the vertical
axis, “Y” axis, presents the number of students that accessed the environment, making this way the
connection among the number of students with the access speed. We can notice that most of the students still
have connection speed considered low comparing to the available speeds, until 15 mega via broadband
connection (OI, 2011).
Figure 2. Graphic of Speed Range

The graphic shows the importance of the creation and availability of environments sensitive to the context
in the computers net area to an improvement in the quality of the students’ access to the materials. The static
VLE, without identifying the student’s speed variables indicate that about 70 accesses of the 200 selected
accesses in the research would show problems, in other words, 36% of the students would have problems
watching videos or opening heavier files. If adapted materials were not provided these students would end up
without the contents or partially presented and non-stimulated to access the environment again.
The Figure 3 presents the Graphic with the number of students that accessed adapted contents in the
period. The Graphic shows that the selected sample, 64% of the students that accessed the environment did
not need that the materials were adapted, but 36% of the accesses presented a computing context that needed
that the tools and the materials were adequate to a better AVA usage. From the 200 selected accesses, 72 of
them were with connections with until 500 Kbps as we found in the Graphic of the Picture 2, of the Speed
Range, with this, 36% of the students had access to the differentiated materials, showing this way the
necessity of each time that the VLEs treat each student in a personalized and individual way.
202
Figure 3. Graphic showing the number of students that accessed adapted materials.
One of the problems of the usage of a standard virtual environment (unique for all the students) and that
is about different user’s net contexts is in the fact that many limitations in the access occur, decreasing the
environment usage by the students that have speed limitations. This can reflect in the resistance or even in the
abandonment of using the environment, if a course in the distance modality, can also influence in the course
evasion.
It is necessary that all the VLE structure help in an effective way on its usage, being necessary that
alternative ways of access be available, like the use of mobile equipments and also that the environment have
guarantee of availability, what in this U-ALS Moodle is granted through the created cloud.
From the finding that the same user can access the environment in more than one way, using different
equipments like computers, laptops, smart-phones or tablets and from different places, the student can access
from home or from school, it becomes extremely necessary the usage of technologies like Cloud Computing,
because this guarantees that the environment will be always available when the student chooses to do his
works. We also found that despite most of the accesses occur in the morning and in the afternoon; some
students accessed the environment during the night, increasing thus the necessity that the environment be
always available.
The elasticity of resources is also another advantage found from Computing Technology in Cloud,
because the teachers have, thus, the possibility of creating new courses and register new students without
worrying about the physical part of the structure where the VLE is installed. The facility of adding resources
is another advantage found, because the teachers don’t need a technical staff to maintain the system working,
not having investments with the TI staff and with the infrastructure.
This new scenery shows the contributions of the U-learning environment to the course development as on
the presence modality as on distance, making necessary the implementation of systems that make the access
to the easy and stimulant to the student. It is necessary that all the structure of these environments be
adequate to the student and work in a way that provides diversity and facility of access, that’s why the
necessity that the virtual environments be improved and adapted arises more and more to their users’
individual characteristics.
REFERENCES
Byun,H.,E.; Cheverst, K.; 2004.“Utilizing context history to provide dynamic adaptations”, Applied Artificial
Intelligence, Vol. 18, No. 6, pp. 533–548.
Buyya, R., Yeo, C. S., Venugopal, S., Broberg, J., And Brandic, I. 2009. Cloud computing and emerging it platforms:
Vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst., 25(6):599–616.
Cearley, David W. Cloud Computing - Key Initiative. 2010. Disponível em:
<http://www.gartner.com/it/initiatives/pdf/KeyInitiativeOverview_CloudComputing.pdf>, Acessado em Outubro
2011.
Chiu, P.; Kuo, Y.; Huang, Y.; Chen, T.; 2008. A Meaningful Learning based u-Learning Evaluation Model. In: Eighth
IEEE International Conference on Advanced Learning Technologies.
Dey A. K.; Futakawa M.; Salber D.; Abowd G. D; 1999. The Conference Assistant: Combining Context-Awareness with
Wearable Computing. 3rd International Symposium on Wearable Computers (ISWC '99) – IEEE Computer Society
Press. San Francisco, CA. October. Pages 21-28.
203
ISBN: 978-989-8533-12-8 © 2012 IADIS
Franco, C, P., 2010. “A Plataforma Moodle como Alternativa para uma Educação Flexível”. In: Revista EducaOnline,
Volume 4 – Número 1.
Gasparini, I., Pimenta, M., Amaral, M.A., Oliveira, J. P. M.; 2004. “Navegação e apresentação adaptativos em um
ambiente de EAD na Web”. In: Webmedia & LA-Web 2004 Joint Conference - 10th Brazilian Symposium on
Multimedia and the Web, Ribeirão Preto, v. 2.
Harmelen, M., V.; 2006. “Personal Learning Environments.” IN: ICAT 2006: Sixth Internatinal Conference on Advanced
Learning Technologies, IEEE.
Liu, S., Liang, Y., And Brooks, M; 2007. Eucalyptus: a web service-enabled einfrastructure. In CASCON ’07:
Proceedings of the 2007 conference of the center for advanced studies on Collaborative research, pages 1–11, New
York, NY, USA. ACM.
Machado, J.C.; Moreira, L., O.; SOUSA,F.,R.,C.; 2010. Computação em Nuvem: Conceitos, Tecnologias, Aplicações e
Desafios. In: ERCEMAPI.
Mozzaquatro, P. M. Modelagem de um Framework para Adaptação de Ambientes Virtuais de Aprendizagem Móveis aos
Diferentes Estilos Cognitivos, no estado do Rio Grande do Sul: Santa Maria/UFSM, 2010. Dissertação de Mestrado.
Oliver, B.; Liber, O.; 2011. “Lifelong Learning: The Meed for Portable Personal Learning Environments and Supporting
Interoperability Standarts.” Disponível eletronicamente em: http://wiki.cetis.ac.uk/iploads/6/67/Oliverandliber.doc.
Acesso em 19 de dezembro de 2011.
Palazzo, José M. De Oliveira; Lima, J. V.; Gasparini, I., Pimenta, M. S., Brunetto, M. A. C., Proença Jr, M., Faggion,
R.2008. “Adaptive Multimedia Content Delivery in AdaptWeb”. XIII Taller Internacional de Software Educativo
TISE, 2008, Santiago. Nuevas Ideas en Informática Educativa. v. 4. p. 23-39.
Pernas, Ana ; Gasparini, I. ; Oliveira, J. P. M. ; Pimenta, Marcelo; 2009. Um ambiente EAD adaptativo considerando o
contexto do usuário. In: I Simpósio Brasileiro de Computação Ubíqua e Pervasiva (SBCUP 2009), 2009, Bento
Gonçalves. Anais do I Simpósio Brasileiro de Computação Ubíqua e Pervasiva (SBCUP 2009) - XXIX Congresso da
Sociedade Brasileira de Computação (CSBC 2009), 2009. v. 1. p. 1151-1156.
Ribas, E. et al; 2009. Tecnologias Digitais: um novo contexto pedagógico na Educação Básica. Colabora, v. 6, n. 19.
Fevereiro.
Severino, Antonio Joaquim; 2000. Metodologia do trabalho científico. São Paulo: Cortez.
Soror, A. A., Minhas, U. F., Aboulnaga, A., Salem, K., Kokosielis, P., And Kamath, S.; 2010. Automatic virtual machine
configuration for database workloads. ACM Trans. Database Syst., 35(1):1–47.
Weiser, M; 1993. Some Computer Science Issues in Ubiquitous Computing. ACM SIGMOBILE Mobile Computing and
Communications Review. July. Volume 3, Issue 3. Page 12.
204
AUGMENTED REALITY IN THE SCIENCE MUSEUM:

LESSONS LEARNED IN SCAFFOLDING FOR
CONCEPTUAL AND COGNITIVE LEARNING
Susan A. Yoon1, Karen Elinich2, Joyce Wang1 and Jacqueline G. Van Schooneveld1
1
Graduate School of Education, University of Pennsylvania
2
The Franklin Institute Science Museum
ABSTRACT
This research follows on previous studies that investigated how digitally augmented devices and knowledge scaffolds
enhance learning in a science museum. We investigated what combination of scaffolds could be used in conjunction with
the unique characteristics of informal participation to increase conceptual and cognitive outcomes. 307 students from nine
middle schools participated in the study. Six scaffolds were used in varying combinations. The first was the digital
augmentation. The next five were adaptations of knowledge-building scaffolds. Results demonstrated greater cognitive
abilities in terms of theorizing can be obtained with particular combinations of scaffolds, however, there appears to be a
trade-off in what students can learn conceptually in informal environments in which learning takes place in short
episodes.
KEYWORDS
Augmented reality technologies, informal environments, knowledge scaffolds, conceptual and cognitive learning.
1. INTRODUCTION
This research seeks to understand how aspects of knowledge building pedagogy (Scardamalia, 2002;
Scardamalia & Bereiter, 2006), known to be successful in formal classrooms, might be adapted to reach
broader informal audiences to improve scientific literacy in museums. The recent National Research Council
(NRC) report on learning science in informal environments (NRC, 2009) highlights the potential of non-
school settings, such as museums, for engaging large populations in scientific investigation. Despite this
potential, the NRC report outlines the need for essential research in three key areas. First, while there is
ample evidence that suggests informal environments increase engagement and interest, fewer studies have
focused on how those experiences result in conceptual gains of science content. Second, in terms of scientific
skills, designed interactives have been shown to increase lower level skills such as manipulating and
observing, however more challenging skills such as critical thinking and theorizing are less frequently
demonstrated. Finally, as digital platforms are increasingly incorporated in informal settings, more research is
needed to determine how they enhance the learning experience.
The research reported here follows on a series of studies that investigate how digitally augmented devices
and knowledge-building activities can enhance learning in a science museum (Yoon et al., in press-a; in
press-b). In the series we use a quasi-experimental design in which students in multiple conditions interact
with a museum device using digitally augmented information and varying arrangements of knowledge
scaffolds. Results thus far have suggested that digital augmentations can help in conceptual development and
that students’ abilities to interpret information about a phenomenon using knowledge-building scaffolds can
improve cognitive understanding in some conditions. However, one major concern that has emerged in our
observations of student interaction in the more highly scaffolded condition is the over-formalization of the
learning experience that appeared to be needed to achieve those results. Specifically, when students were
asked to use learning scaffolds that were delivered in, for example, worksheets with knowledge prompts,
students were less explorative and referred to the scaffolds to dictate their next steps. Students in less
formally structured conditions were generally more playful and experimented on their own—much like
205
ISBN: 978-989-8533-12-8 © 2012 IADIS
students would normally behave in museum exhibits during a school field trip. Informal learning
environments can enhance interest and engagement because the experience does not look like school, i.e., it
is fluid, sporadic, and participant-driven (Falk & Dierking, 1992, 2000). The research in this study aims to
investigate the issue of over-formalization. Specifically, the research question we seek to answer is: Which
combination of learning scaffolds can optimally be used in a science museum taking into consideration the
unique characteristics of informal participation to increase learning outcomes?
2. BODY OF PAPER
2.1 Theoretical Considerations

2.1.1 Augmented Reality as a Scaffold for Learning
In the most recent Horizon Report, the New Media Consortium (2012) discusses the enormous potential
augmented reality (AR) capabilities have on learning and assessment in enabling people to construct new
understanding. AR experiences layer digital displays over 3D real world environments (New Media
Consortium, 2012) providing access to normally hidden data that users can use to develop deeper knowledge
about a phenomenon or take a different immersive perspective to broaden their understanding. In the past
decade, practical uses of AR have emerged in fields such as games, marketing and advertising, films,
navigation, and for medical and military applications (El Sayad et al., 2011). Although newer in education,
over the last few years, there have been studies that illustrate AR’s potential for learning particularly in the
field of science education. For example, Dunleavy and colleagues (2009) document high student engagement
influenced by the ability to collaboratively problem solve and collect data in the real world in their Alien
Contact! handheld AR environment. Squire and Klopfer (2007) detail the impact of their AR game
Environmental Detectives on accessing student prior knowledge by connecting academic content to physical
spaces that students are familiar with. In Outbreak @ The Institute, Rosenbaum and colleagues (2007)
document the affordance of their augmented reality game play to include authentic scientific inquiry and
understanding the dynamic nature of system interactions. In these studies, the indirect correlates of student
learning, i.e., engagement, prior knowledge, and processes in scientific practice, are important outcomes of
the research and provide valuable impetus for pursuing further studies on what and how students learn in
terms of scientific knowledge.
AR technologies have also been incorporated in museums to enhance visitors’ experience by improving
their interest, engagement, and access to information (Baber et al., 2001; Damala et al., 2008; Hall &
Bannon, 2006). For example, Szymanksi and colleagues (2008) examined how two different electronic
guidebook prototypes affected visitors’ social interaction. As visitors toured a historic house, the guidebooks
provided information about the historic artifacts that the visitors encountered. Results indicated that these
guidebooks led visitors to engage in more content-rich discussions with each other. Furthermore, their
exploration of the room and its objects were enhanced (Szymanski et al., 2008). Similarly, Baber et al.
(2008), in testing three different AR platforms in an art museum, discovered that different devices elicited
varying lengths of visitor viewing times and visitor preferences. Damala et al. (2008) also tested an AR-
enabled mobile multimedia museum guide in a fine arts museum and found that visitors enjoyed the playful
content presentation that the museum guide enabled. Finally, Hall and Bannon (2006) investigated the
effects of a digitally augmented exhibit on children. When children were given RFID sensors that could
detect exhibit locations and unlock virtual information, their interest and engagement increased.
Collectively, these studies demonstrate the impact that AR can have on visitors in a museum setting in terms
of increasing engagement, interest, and access to information through scaffolded experiences.
Encouraged by this previous research and in envisioning potential extended uses of AR in museums as
scaffolds for knowledge improvement, the present study addresses the real need for learning research (NRC,
2009) in informal environments through connecting with the notion of scaffolds. In the next section, we
describe our rationale underpinning the use of scaffolds and in particular the use of knowledge-building
scaffolds to improve learning.
206
2.1.2 Learning through Knowledge-Building Scaffolds

The use of scaffolds in educational technology applications has been researched fairly extensively to support
scientific inquiry and cognitive tasks (e.g., Quintana, 2004). In particular, a long-standing program of
research in the learning sciences that is premised on designing learning environments through the intentional
application of technological and pedagogical scaffolds is knowledge building (Yoon, 2008; Bereiter, 2002;
Scardamalia, 2002; Scardamalia & Bereiter, 2006). This approach is centrally focused on the goal of
improving ideas in the same way that knowledge work is done by experts in real world contexts (Scardamalia
& Bereiter, 2006). Primarily applied in school classrooms, knowledge-building studies have been shown to
increase student scientific abilities in explanation, interpreting and evaluating information, and knowledge
advancement (van Aalst, 2009). Students also acquire deep theoretical understanding of scientific
phenomenon through collective sustained inquiry and research on problems that can range from what causes
leaves to change color in the fall in a grade 1 classroom (Scardamalia, 2002) to the complex influences on
genetic engineering research with middle and high school students (Yoon, 2008; 2011).
The technological application, Knowledge Forum and associated pedagogy use educational scaffolds to
enable public, collective contributions that shape the knowledge constructed in the learning community. Such
scaffolds include prompts for consensus building, generalizations, differentiation between evidence and
theories, and peer evaluation. For example, a prompt such as “My theory is...” encourages students to use
evidence to construct a more general understanding of a class of scientific phenomena. Similarly, students
can create a “rise above” note, enabled by the archived database of peer exchanges, which is a distillation of
an idea or theory from a collection of previous peer exchanges that provide students with opportunities to
think across diverse perspectives and to arrive at conclusions about how the collective learning community
views a scientific issue (Yoon, 2008).
Collaboration also factors prominently into the knowledge-building approach. By working with others
discursively to problem solve, evaluate evidence, and identify important shared understanding, students are
able to more deeply reflect on what they know rather than learning independently, or learning through textual
modes. This decentralized, public, and distributed participation promotes what Scardamalia (2002) calls
collective cognitive responsibility where the impetus for learning is generated by consensus within the
community rather than by the teacher. From this set of theoretical and pedagogical descriptions, our series of
studies uses varying degrees of what we collectively refer to as knowledge-building scaffolds which include:
knowledge prompts, a bank of peer ideas, working in collaborative groups, instructions for generating
consensus, and worksheets for recording shared understanding. However, because knowledge building
requires the development of a community with shared understanding, language, and goals, learning events
evolve over longer periods of time than informal environments may afford. Van Aalst (2009) characterizes
learning experiences that are less focused on the community as knowledge construction in which students
may collaborate in small groups on tasks that require less synthesis and reflection on the knowledge
advancement process. We have understood the limitation of our informal setting and population in terms of
achieving a true knowledge building community in previous studies (e.g., Author, in press-b) but have
nevertheless attempted to investigate how aspects of knowledge building pedagogy can be applied in
informal environments given its success in formal classrooms. The issue of over-formalizing the experience
was a phenomenon that emerged as an important context-dependent factor in considering how informal
learning takes place (Author, in press-b), which is the subject of this study. In the next section, we describe
the features of informal environments that make learning unique.
2.1.3 Features of Informal Learning Environments
Learning in informal spaces is fluid, sporadic, social, and participant driven—characteristics that contrast the
highly structured formal classroom experience (Honey & Hilton, 2011, NRC, 2009, Squire & Patterson,
2009). Activities are often experienced in single-visit episodes (Falk et al., 2007) where visitors learn on their
own with little structured follow-up or reflection. McManus (1994) has characterized typical visitors as
demonstrating scouting behaviors within museum exhibits, where they roam around, encounter devices, and
act quickly to discover the intended information. Thus, more systematic learning studies are difficult to
design. However, science museum exhibit developers do intentionally design learning spaces that mix a
variety of supports for learning. For example, exhibit devices (individual interactive kiosks within the larger
exhibit) are deliberately grouped and arranged to encourage progressive engagement with topics (Grinell,
2006) in support of the exhibit’s overall learning objective. Many devices are accompanied by posted graphic
207
ISBN: 978-989-8533-12-8 © 2012 IADIS
panels (Serrell, 1998, 2006) that provide printed content in order to support the interpretation of scientific
phenomena. Finally, visitors often encounter exhibit educators (Serrell, 2006) who are skilled at facilitating
learning in social groups around exhibit devices. They spark conversation, notice ideas, and encourage
theorizing. We see parallels between these aspects of the informal learning environment and the structure of
effective knowledge building in classroom settings.
For this study, we deliberately attempted to construct conditions that probe these parallels. The
application of a digital augmentation to an exhibit device provides a first layer of interpretive support and
acts as a primary scaffold. We then added posters and worksheets with progressively more rigid layers of
structure around the experience to advance the scaffolding for analysis. In some conditions, we prompted or
required group work as a strategy for measuring the benefit of collaboration. We attempted to detect the
frequency of theory building and, in some conditions, prompted it explicitly. As our methods and results will
show, these scaffolds had varying levels of success.
2.2 Methods
2.2.1 Participants and Context
For our study population, we recruited teachers to bring their students who had previously participated in
workshops and other teacher events at the museum or were referred to us to participate in the study by those
teachers. In total, 307 students (52% female, 48% male) on a school fieldtrip from nine local middle schools
(grades 6-8) participated in the study. We chose the middle grades for this study to have confidence that the
students were developmentally able to theorize. Also, the study engaged students with the topic of electrical
conductivity. By grade six, all students would have encountered the topic in their classrooms, as dictated by
the local standardized curriculum. This provided a common ground for the participants, which increased the
chances that they would have similar prior knowledge.
An exhibit device called “Be the Path” provided the context for the study (see Figure 1). The device
consisted of two metal spheres on a table, approximately one foot apart, with one connected by a wire to a
battery and the other connected to a light bulb. The instructions on the device provided little direction, simply
suggesting “try to complete the circuit.” The students attempted different configurations to complete the
circuit and light the bulb. Once the circuit was completed, a projected visualization of electrons flowing
around the complete loop appeared. A more complete description of the technological platform and how we
developed it can be found in (Author, in press-b).
Figure 1. “Be the Path” device with digital augmentation

Students were randomly assigned to conditions, constructed to represent increasing use of scaffolds. Six
scaffolds were used in varying combinations. The first was the digital augmentation. The next five were
adaptations of knowledge-building scaffolds which included: 1) group work; 2) worksheet questions
directing student attention to relevant information; 3) directions about how to work together; 4) embedded
worksheet knowledge prompts; and 5) bank of previous worksheet responses. Condition 1 (C1) served as the
control group with no scaffolds. Condition 2 (C2) represented the digitally-augmented device with no
additional scaffolds. Condition 3 (C3) was the same as C2 but with posted worksheet questions such as What
happened when you touched both metal spheres?; What happened to make the bulb light up? Students in C1–
208
C3 used the device individually and completed individual worksheets afterwards. Condition 4 (C4) students
worked in groups, with posted worksheet questions, completing the worksheet individually after the
experience. C4 was meant to represent the typical informal learning state in which groups of students
encounter posted labels with questions about devices. Condition 5 (C5) students had the same treatment as
C4 students with additional knowledge prompts. Knowledge prompts included, Our hypothesis is; Our
Theory is; and Others have said. A bank of other student worksheet responses was also posted for students to
refer to as they answered the worksheet questions. They completed the worksheet collaboratively after the
experience with one worksheet per group. Condition 6 (C6) students had the same treatment as C5 except the
worksheet was completed collectively during the experience. Table 1 shows the number of students who
participated in each condition.
Table 1. Number of students who participated in each condition
Condition Number of Students
1 30
2 31
3 31
4 24 groups of 3 (72)
5 23 groups of 3, 1 group of 2 (71)
6 24 groups of 3 (72)
2.2.2 Data Sources and Analysis

Three data sources were collected and analyzed through a quasi-experimental mixed-methods approach:
surveys, worksheets, and interviews. Each data source is detailed below. Surveys: A conceptual knowledge
survey was administered to students in each group before and after the intervention. The survey posed five
general multiple-choice content questions each valued at one point, related to the scientific topic of electrical
conductivity and circuits. An additional open-ended question on the survey also solicited responses that
demonstrated knowledge directly related to the device experience, i.e., “Think about an electric circuit that
supplies electricity to a light bulb. What parts make it work so that the bulb lights up?” Responses to this
question were coded on a five-point Likert-scale from no understanding (0) to complete understanding (4).
For the coding scheme, the dataset was reviewed qualitatively by members of the research team in a series of
data analysis sessions during which responses were compared to answers in a teacher’s manual. We also
consulted a physicist on staff at the museum to help the team design the scoring rubric. Once codes were
established, a categorization manual was constructed. Two graduate students, who were not otherwise
familiar with the research project, were trained on the coding scheme, and inter-rater reliability was obtained
on 20% of the data with greater than 90 percent agreement. Collectively, the maximum possible score on the
conceptual knowledge survey was nine points. A paired-samples t-test was conducted to determine statistical
significance in conceptual knowledge gain within each condition after treatment. A between-subjects
ANCOVA was conducted to determine whether the main condition effect was significant. For this survey, we
weren’t able to conduct a multilevel analysis due to the small sample size but we are aware of the non-
independence issues that can potentially impact results. However, as Cress (2008) notes, computer supported
collaborative learning environments are designed to be influenced by group interactions and we intentionally.
designed the conditions to understand the impact of learning scaffolds of which group interaction is one.
Worksheets: A worksheet was used to gather data from all participating students, regardless of condition.
Students in Conditions 1, 2, and 3 completed the worksheet individually immediately after they finished
interacting with the device. They moved away from the device to be seated at tables for the task. Students in
Condition 6 actually completed the worksheet during the group experience with a single worksheet on a
clipboard being used for their collective response. Each group of three was allowed to negotiate the work
process for themselves. For example, some groups elected a single scribe while others passed the sheet
around to share the scribe duties. The worksheet required them to respond to this open-ended question:
“What are you supposed to learn by using this device?” This question was intended to elicit responses that
demonstrated ability to theorize from the interaction with the device, i.e., understanding of electrical circuits
and how the human body functioned as a conductor. Responses were coded on four levels from no
understanding (0) to complete understanding (3). Categorization manual construction, coding and reliability
activities similar to those performed for the open-ended conceptual knowledge question were conducted. An
ANOVA was conducted on the worksheet data set to determine whether there was a statistically significant
209
ISBN: 978-989-8533-12-8 © 2012 IADIS
difference in responses between the conditions and a post-hoc Tukey HSD comparison was conducted to
determine the source of the difference. Interviews: In order to investigate how knowledge-building scaffolds
impacted the nature of the intervention, 66 students (~20% of the population) from C5 and C6 were randomly
selected for short interviews to determine the utility of the scaffolds following their interaction with the
device. We asked the students to evaluate each of the scaffolds they encountered in order to understand
which were more or less impactful on their experience.
2.3 Results
2.3.1 Conceptual Knowledge
Across the conditions, pre- and post-survey means ranged between 2.903 (pre-survey, C3) to 4.014 (post-
survey, C4) out of a possible score of 9. Table 2 shows the results of a paired-samples t-test conducted within
each condition. The table shows that gains in all conditions except C1 were statistically significant. Students
in C4 demonstrated the greatest gains, with students in C3 a close second. However a between-subjects
ANCOVA using the pre-test scores as covariate showed that there was no significant difference in the
outcomes between six groups, F(5, 300) = .546, p=.741.
Table 2. Results of Paired-Samples T-Test Comparing Means within Conditions
Condition Mean Difference SD t df Sig.

(2-tailed)
1 .337 1.691 1.187 29 .245
2 .484 1.313 2.051 30 .049*
3 .677 1.833 2.058 30 .048*
4 .681 1.362 4.241 71 .001*
5 .366 1.355 2.278 70 .026*
6 .472 1.538 2.605 71 .011*
* p<0.05
2.3.2 Cognitive Understanding

Students were asked What are you supposed to learn by using this device? Response scores ranged from
2.233 (low emergent understanding) to 2.800 (nearly partial understanding). Figure 2 shows an increasing
trend from C1 to C6 in the means of theorizing abilities.
Worksheet Responses: Ability to

Theorize
Levels of Understanding
4
3,5
3 2,708 2,8
2,387 2,452 2,431
2,5 2,233
2
1,5
1
C1 C2 C3 C4 C5 C6
Condition
Figure 2. Mean scores for ability to theorize for conditions 1-6

An ANOVA comparing mean scores showed a significant difference in results, F(5)=2.234, p=.052. A
post-hoc Tukey analysis attributed the difference to the higher mean of C6 which was marginally
significantly higher than C1 (p=.056). Differences between all other conditions were statistically not
significant.
210
2.3.3 Impact of Knowledge-Building Scaffolds

C5 and C6 students were asked which knowledge-building scaffolds were helpful. Figure 3 graphs their
responses.
Frequency of student Responses

to Scaffold Helpfulness (n=66)
65
56
46
36
26
Figure 3. Frequency of student responses indicating helpfulness of scaffolds

Many students found both group work and the worksheet helpful while fewer than 1/2 found the
directions on how to work together helpful. Greater than 2/3 indicated that the knowledge-building prompts
were helpful and greater than 1/2 indicated that the bank of previous answers was helpful.
3. CONCLUSION
Based on the conceptual gains shown in Table 2, this study supports previous findings (Yoon et al., in press-
a; in press-b) that digital augmentations can serve as valuable scaffolds for conceptual learning. Furthermore,
where C3 and C4 students had the highest mean difference, we found that worksheet questions, which we
consider to be a form of graphic panel (Serrel, 2006) when posted, can also support conceptual gains. We
also found this to be true in our previous studies. When interviewed, C5 and C6 students said that working in
groups was the most helpful scaffold besides worksheets, which suggests that the combination of group work
and posted worksheets can assist in learning science concepts.
Although the knowledge prompts and the bank of previous answers were selected fewer times as helpful
scaffolds than working in groups and worksheets, the fact that a good portion of the population endorsed
these more formal scaffolds is encouraging. Also, the result that C6 students had the highest score in higher-
order reasoning somewhat supports an argument for their continued use. However, the main reason for
undertaking this research was to understand the extent to which knowledge-building scaffolds enhance the
museum learning experience and which are most impactful. We hypothesized that “over-formalization” in an
informal environment may negatively impact learning. This would include the use of scaffolds other than the
kinds of graphic panels that are normally encountered. Our results indicate that greater conceptual gains
occurred when scaffolds were used in less formal ways, i.e., when the worksheet questions were posted (C3
and C4) and when they looked more like graphic panels typically found in museum settings, which in part
satisfies our hypothesis. The fact that C6 students demonstrated greater theorizing ability suggests, however,
that there may be a trade-off in what students can learn in informal environments in which learning takes
place in short episodes. That is, we may be able to improve student’s conceptual understanding somewhat at
the expense of improved cognitive gains if we consider the unique characteristics of informal settings.
Repeat visitation and use of an exhibit space could counteract this pattern as conceptual understanding
can grow over time and later become more formalized for the sake of cognitive gain. Through repeated use of
an exhibit device, especially with a variety of knowledge-building scaffolds available, students find multiple
learning pathways. The habit of repeated use of an exhibit remains a challenge given the episodic nature of
visitation. On possible way to address this issue is a distributed virtual social learning space that can extend
211
ISBN: 978-989-8533-12-8 © 2012 IADIS
engagement with topics from multiple exhibit devices outside of the museum’s physical space. Such a
collaborative space is currently in development for testing in future phases of our research.
REFERENCES
Baber, C. et al, 2001. Augmenting Museums and Art Galleries. In Human-Computer Interaction, Vol. 16, pp. 439-447.
Bereiter, C., 2002. Education and Mind in the Knowledge Age. Lawrence Erlbaum Associates, Mahwah, NJ.
Damala, A. et al, 2008. Bridging the Gap Between the Digital and the Physical: Design and Evaluation of a Mobile
Augmented Reality Guide for the Museum Visit. In 3rd ACM International Conference on Digital and Interactive
Media in Entertainment and Arts, pp. 120-128. ACM Press, New York.
Dunleavy, M. et al, 2009. Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for
Teaching and Learning. In Journal of Science Education and Technology, Vol. 18, pp. 7-22.
El Sayad, N. et al, 2011. ARSC: Augmented Reality Student Card. An Augmented Reality Solution for the Education
Field. In Computers & Education, Vol. 56, pp. 1045-1061.
Falk, J. and Dierking, L., 1992. The Museum Experience. Whalesback Books, Washington, DC.
Falk, J. H., and Dierking, L. D., 2000. Learning From Museums: Visitor Experiences and the Making of Meaning.
AltaMira Press, Walnut Creek, CA.
Grinell, S., 2006. In Search of New Audiences: Blockbusters and Beyond. ASTC Dimensions. May/June, 1-4.
Hall, T. and Bannon, L., 2006. Designing Ubiquitous Computing to Enhance Children’s Learning in Museums. In
Journal of Computer Assisted Learning, Vol. 4, pp. 231-243.
Honey, M. A. and Hilton, M. (Eds.)., 2011. Learning Science Through Computer Games and Simulations. National
Academies Press, Washington DC.
National Research Council, 2009. Learning Science in Informal Environments: People, Places, and Pursuits. The
National Academies Press, Washington, DC.
Quintana, C. et al, 2004. A Scaffolding Design Framework for Software to Support Science Inquiry. In Journal of
Learning Sciences, Vol. 13, No. 3, pp. 337-386.
Rosenbaum, E. et al, 2006. On Location Learning: Authentic Applied Science with Networked Augmented Realities. In
Journal of Science Education and Technology, Vol. 16, No. 1, pp. 31-45.
Scardamalia, M., 2002. Collective Cognitive Responsibility for the Advancement of Knowledge. In B. Smith (Ed.),
Liberal Education in a Knowledge Society, pp. 67-98. Open Court, Chicago.
Scardamalia, M. and Bereiter, C., 2006. Knowledge Building. In K. Sawyer (Ed.), The Cambridge Handbook of the
Learning Sciences, pp. 97-115. Cambridge University Press, New York.
Serrell, B., 1998. Paying Attention: Visitors and Museum Exhibitions. American Association of Museums, Washington,
DC.
Serrell, B., 2006. Judging Exhibitions: A Framework for Assessing Excellence. Left Coast Press, Walnut Creek, CA.
Squire, K. and Klopfer, E., 2007. Augmented Reality Simulations on Handheld Computers. In Journal of the Learning
Sciences, Vol. 16, No. 3, pp. 371-413.
Squire, K. and Patterson, N., 2009. Games and Simulations in Informal Science Education. Paper commissioned for the
National Research Council Workshop on Gaming and Simulations, October 6-7, Washington.
http://www7.nationalacademies.org/bose/Gaming_Sims_Commissioned_Papers.html
Szymanski, M. et al, 2008. Sotto Voce: Facilitating Social Learning in a Historic House. In Computer Supported
Cooperative Work, Vol. 17, No. 1, pp. 5-34.
Yoon, S., Elinich, K., Wang, J., Steinmeier, C., & Tucker, S. (in press-a). Using Augmented Reality and Knowledge-
Building Scaffolds to Improve Learning in a Science Museum. To appear in the International Journal of Computer
Supported Collaborative Learning.
Yoon, S., Elinich, K., Wang, J., Steinmeier, C., & Van Schooneveld, J. (in press-b). Learning impacts of a digital
augmentation in a science museum. To appear in Visitor Studies.
Yoon, S. A. (2011). Using social network graphs as visualization tools to influence peer selection decision-making
strategies to access information about complex socioscientific issues. Journal of the Learning Sciences, 20(4), 549-
588.
Yoon, S. (2008). An evolutionary approach to harnessing complex systems thinking in the science and technology
classroom. International Journal of Science Education, 30(1), 1–32.
212
TEACHING THE BALANCED SCORECARD THROUGH

SIMULATION
Carlos Capelo, Ana Isabel Lopes and Ana Mata

Universidade Lusófona
Campo Grande, 376, 1749-024 Lisboa – Portugal
ABSTRACT
Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the business
strategy and performance measurement. Many organizations around the world are using the BSC to define, implement
and manage strategy. Nevertheless there exist studies that identify problems and limitations associated with the
implementation and use of the BSC. Those studies show in general terms that managers do not understand the BSC as the
measures and perspectives in use are fairly independent, and do not always mirror the recommended cause-and-effect
logic included in the systems perspective of the BSC approach. This article addresses the effectiveness of teaching the
Balanced Scorecard by means of business simulation. An experiment that uses a business simulator is performed for
testing a set of hypotheses about the influence of simulation on the students’ understanding of the BSC. The simulation
experience was specifically designed by the authors to promote understanding of the BSC concepts. Student feedback and
assessment showed that the simulation significantly enhanced the understanding of the BSC concepts related to the
strategic management and double-loop learning processes and the systems perspective. Results also suggested that
understanding of those BSC concepts positively influences simulation performance.
KEYWORDS
Balanced Scorecard, Business Simulation, Training, Business Education, Teaching Methods.
1. INTRODUCTION
Kaplan and Norton (1992) introduced the Balanced Scorecard (BSC) with the aim of overcoming strategic
management limitations of the traditional performance measurement systems. The BSC approach features a
mix of lead (performance drivers) and lag (outcome measures) indicators, financial and non-financial
measures. These are categorized as follows: (i) financial, (ii) customer, (iii) process and (iv) learning and
growth. The BSC tool helps managers monitor actual financial and market performance, evaluate the results
of short-term processing actions, monitor the intangible development of competencies that will drive future
financial performance and assess the progress of implementing corporate strategy. To support managers in
developing a cause-and-effect perspective and to better understand the business system in which they
participate, Kaplan and Norton (2001) developed the strategy map concept as a complementary tool to the
BSC approach. The strategy map links the performance indicators in a causal chain (causal diagram) that
helps managers to translate, test and communicate their understanding of the business system.
Managers make decisions and learn in the context of feedback loops (Forrester, 1961). In single-loop
learning, managers compare information about the state of a real system to pre-established goals, perceive
deviations between desired and actual states, and make the decisions they believe will move the system
towards the desired state. Single-loop learning does not change the managers’ mental models. A mental
model is a conceptual representation of the structure of an external system used by people to describe,
explain and predict a system’s behavior (Craik, 1943, Johnson-Laird, 1983). In double-loop learning,
information about the business system is not only used to make decisions within the context of existing
frames, but also feeds back to modify the managers’ mental models (Argyris, 1976). As their mental models
change, managers define new strategies and policies.
It is clear that the BSC approach is consistent with the systems and feedback learning perspective of
business management and performance measurement. Kaplan and Norton (2001) argue that the BSC
213
ISBN: 978-989-8533-12-8 © 2012 IADIS
approach supports double-loop learning that facilitates managerial strategic learning, leading to better
performance. In a continual process, managers use the balanced scorecard and the strategy map to re-evaluate
the assumptions used in the previous strategy. They review the assumed cause-and-effect relationships and
identify new ones. Then they improve their understanding of the business system and they determine a new
strategy (Kaplan and Norton, 2001). In other words, BSC triggers a process by which managers can make
explicit improvements to their mental models of the business system. They adapt the company strategy and
define the new short and middle term objectives by simulating their mental models to infer the future
behavior of the business system.
Many organizations around the world are using the BSC approach to define, implement and manage
strategy. However, according to empirical research into the performance implications of the BSC, the
positive contributions of the BSC have not been unambiguously confirmed. There exist studies in the field of
management accounting research that identify problems and limitations associated with the BSC approach.
The inadequate definition and utilization of the performance indicators has been highlighted as a main
drawback of the BSC system (Lingle and Schiemann, 1996; Stivers et al., 1998; Ittner and Larcker, 1998,
2003; Lipe and Salterio, 2000; Malmi, 2001; Speckbacher et al, 2003). Those studies show in general terms
that the measures and perspectives in use are fairly independent, and do not always mirror the recommended
cause-and-effect logic of the BSC approach.
1.1 The use of Simulation for teaching the BSC

Business simulators, business games, management simulators, microworlds, learning laboratory, interactive
learning environments are some examples of terms used to describe computer-based simulations that have
been proposed by researchers from many fields, e.g., management and decision science, psychology,
education and computer science, as important tools to support management learning processes (Maier and
Grobler, 2000). In a broad definition, business simulators for learning purposes comprise virtually everything
connected with the learning process. An interactive learning environment contains more than a computer
simulation model. A simulation model is embedded into a learning environment, which may also include
case descriptions, presentations by a facilitator, modelling tools, background information, source material and
working instructions (Davidsen and Spector 1997).
Simulators are promising tools for teaching in business management domain as they expected to help
students acquire knowledge. Students’ learning processes center around the exploration of the simulation
model. They gather knowledge through an inquiry learning process as they make a broad analysis of the
domain, generate hypothesis, experiment, interpret the outcomes, conclude about the validity of the
hypothesis or form new ideas, and finally reflect on the domain (de Jong and van Joolingen, 2008).
Additionally, by performing simulated tasks, learners understand through experiential processes how
concepts are applied and why they are useful, thus enabling new learning to be more easily (Cannon-Bowers
and Bowers, 2008). Other specific advantages have been pointed out. An often named advantage of computer
simulations is the “compression of time” as they instantly show the results of decisions a user has made.
Simulators help subjects leverage their domain-rich knowledge by allowing them to play through simulated
years, reflect on their actions, modify their mental models, then repeat the process. By compressing time,
business simulators can accelerate learning by enabling them to conduct many such cycles of action and
reflection (Bakken et al, 1994). Providing a safe environment is at the same time an important advantage of
simulators (Sterman 1994), which allows for experiential learning without that stress-related obstacles that
are met in reality. However, subjects may tend to take more risks than in real-life decisions. Also, computer
simulations of business systems address objectively certain special management issues and try to abstract
from details and isolating from confounding factors (Isaacs and Senge, 1994). This abstraction allows a focus
on the learning of important and specific business themes.
As mentioned in previous section, several studies show in general terms that managers do not understand
the fundamental concepts of the BSC approach, in particular those connected with the systems and feedback
learning perspective. Thus, appropriate teaching and training methodologies must be designed in order to
improve subjects learning and comprehension of those concepts. The present study addresses the
effectiveness of teaching the Balanced Scorecard by means of business simulation. An experiment that uses a
business simulator is performed for testing a set of hypotheses about the influence of simulation on the
subjects’ understanding of the BSC. The effectiveness and relevance of simulators for learning is intuitively
214
acknowledged by many researchers. However, there exists scarce research about the effectiveness of
computer-based simulations to support learning. Thus, this study also aims to be a contribution to the
research issue of “effectiveness of computer-based simulations to support learning’’ not been solved so far.
2. RESEARCH HYPOTHESES
This research focuses on how business simulation facilitates the teaching of the Balanced Scorecard
approach, leading to enhanced students’ understanding of its main concepts. The analysis of the subjects’
understanding of the BSC concepts considers two components: the performance measurement system and the
systems and strategic learning perspective. The performance measurement system associated with the BSC
approach is viewed as a comprehensive structure of performance indicators that combine both financial and
non-financial measures. By using a BSC performance measurement system, managers access more relevant
information to interact with the business environment, and they acquire a sharper understanding of the impact
of their decisions. The systems and strategic learning perspective - The BSC approach is consistent with the
systems and strategic learning perspective. By using the BSC strategy map tool, managers will benefit from
more effective double-loop learning as they review the critical cause-and-effect relations through a process
that externalizes and improves their mental models of the business system.
It is assumed that by teaching the BSC through business simulation, students will benefit from more
effective understanding of the BSC performance measurement system concepts. Hypothesis 1: The use of
simulation improves the level of understanding of the BSC performance measurement system.
It is assumed that by teaching the BSC through business simulation, students will benefit from more
effective understanding of the systems and strategic learning perspective of the BSC approach. Hypothesis 2:
The use of simulation improves the level of understanding of the BSC as a strategic learning system.
Hypothesis 3: The use of simulation improves the level of perceived relevance of the strategic learning
system concepts.
This study also assumes that students perform better the business simulation task if they reach both higher
level of understanding of the BSC performance measurement system and higher level of understanding of the
systems and strategic learning perspective of the BSC approach. Hypothesis 4: The level of understanding of
the BSC performance measurement system is positively correlated with Performance. Hypothesis 5: The
level of understanding of the BSC as a strategic learning system is positively correlated with Performance.
Hypothesis 6: The level of perceived relevance of the strategic learning system concepts is positively
correlated with Performance.
3. METHOD
The hypotheses defined in this research and presented in the previous section were tested with a simulation-
based experiment. This section presents an overview of the simulator, describes the subjects and the
experiment conditions, and overviews the research variables.
The business simulator was built by incorporating the same system dynamics model that had been used in
previous research (Capelo and Dias, 2009). The participants run a realistic simulator of a wireless
telecommunications firm by making critical decisions every six months for a simulation period of seven
years. The participant objective was to develop critical and interrelated resources at appropriate rates and
levels in order to gain and retain customers, operate efficiently, and maximize value creation. To succeed in
this simulation task, participants had to identify and understand the cause-and-effect relationships among
critical variables. The simulator provides an interface that represents a balanced scorecard that includes a set
of leading and lagging, financial and non-financial indicators that are graphically separated into four sections
related to the four perspectives associated with the BSC approach. The simulation task also involves drawing
and reviewing a strategy map. The participants produced and reviewed a strategy map linking the critical
concepts as were found in the simulation model. This strategy map represents the participants’ understanding
of the structure of the simulated business system.
215
ISBN: 978-989-8533-12-8 © 2012 IADIS
The subjects consisted of 74 undergraduate students. All the participants were familiar with basic BSC
concepts as they attended previous classes on the BSC approach. The participants had no experience with the
simulator and they also had no prior specific knowledge about wireless telecommunications businesses.
All participants were given a full experimental guide including: description and objective of the
simulation task; case text; instructions for accessing and starting the simulator on the computer network;
instructions for running the simulator; sheets for strategy map review. The decisions made on the simulation
and its results were automatically stored in a protected spreadsheet on the participant’s computer.
Instructions on
Initial Task De scription Instructions about Practice
Ca usal
Q ue stionnaire a nd C a se Te x t Sim ulato r Sim ulation
Diagram m ing
Initial Strate gy Strate gy Map De finitive Final

Sim ulation R e port
Map R e vie w Sim ulation Q ue stionnaire
Figure 1. Experimental procedure

The firm was run by using the balanced scorecard and the strategy map. They analyzed business status
using the simulator interface, used this information to review the strategy and objectives and decision
making, and then repeated the process. The experiment procedure involved four sessions and had the
following steps (Figure 1). Session 1: a) The participants answered a questionnaire on BSC approach. This
questionnaire, comprising twenty five multiple choice questions about the BSC, and a ten-point scale to
evaluate the relative relevance of ten critical concepts (five regarding the performance measurement system,
and other five regarding the systems and strategic learning perspective), captured their initial level of
comprehension of the BSC approach. Session 2: b) The participants read the introduction with the overall
description and the objectives of the simulation and the business case and were instructed to raise any
questions they had as they proceeded with the case. c) The participants read the instructions for accessing,
starting and running the simulator and they were given oral instructions with examples to show simulator
operation. d) A first simulation was conducted to familiarize participants with the game interfaces and
commands, and at this point they were instructed to ask for help at any time. Session 3: e) The participants
received instruction in how to draw the strategy map, and how to review it by cutting or inserting links
between the indicators and defining the arrows that indicated the cause-and-effect relationships. f) The
participants drew the initial strategy map. g) The participants performed the definitive simulation. They were
also asked to review the causal diagram (strategy map). They cut or inserted links so that the causal diagram
accurately expressed their latest understanding of the simulated business system. The participants were also
encouraged to use the strategy map to reflect on strategy, objectives, and decisions. Session 4: h) The
participants filled out the same questionnaire that was used on the session 1 in order to capture their ultimate
understanding of the certain critical concepts of BSC approach. Finally, the participants described their
simulation experience in a written report which was part of the evaluation process.
3.1 Research Variables

Level of understanding of the BSC performance measurement system (LUPMS-BS and LUPMS-AS). These
variables were measured in terms of the evaluation score of participants in answering a questionnaire, before
the simulation experiment (LUPMS-BS) and after the simulation experiment (LUPMS-AS).
Level of understanding of the BSC as a strategic learning system (LUSLS-BS and LUSLS-AS). These
variables were measured in terms of the evaluation score of participants in answering a questionnaire before
the simulation experiment (LUSLS-BS) and after the simulation experiment (LUSLS-AS).
Level of relevance of the performance measurement system concepts (LRPMS-BS, LRPMS-AS and
LRPMS-CA). These variables represent the students’ perception on the relative importance of the
performance measurement system of the BSC approach. LRPMS-BS and LRPMS-AS were obtained from
216
the questionnaire answered by participants, before and after the simulation experiment. The variable
LRPMS-CA results from a quantitative content analysis of the final report.
Level of relevance of the strategic learning system concepts (LRSLS-BS, LRSLS-AS and LRSLS-CA).
These variables represent the students’ perception on the relative importance of the systems and strategic
learning perspective of the BSC approach. They also result from the questionnaire answered by participants,
before and after the simulation experiment. The variable LRSLS-CA was also obtained from quantitative
content analysis of the final report.
Performance - Task performance was measured by total financial value creation.
4. RESULTS AND DISCUSSION

Table 1 shows the results of statistical testing to identify differences in means between stages. Unexpectedly,
after simulation the participants showed on average a lower level of understanding of the BSC performance
measurement system (LUPMS). However, the mean values of LUPMS for after simulation stage were not
significantly different from the equivalent values for before simulation stage (mean difference=-0.32, p=315).
On average, after simulation, the participants showed a better level of understanding of the BSC as a strategic
learning system (LUSLS), a lower level of relevance of the performance measurement system concepts
(LRPMS), and a better level of relevance of the strategic learning system concepts (LRSLS). Table 1 shows
that those differences are significant at p<0.01 (mean difference LUSLS=0.188, p<0.001; mean difference
LRPMS=-3.39, p<0.001; LRSLS=3.63, p=0.001).
Table 1. Tests of significance for differences in means between the stages
After Simulation – Before Simulation
Mean Standard Deviation Significance
Variable Difference p
LUPMS -0.32 2.44 0.315
LUSLS 1.88* 2.85 0.000
LRPMS -3.39* 6.96 0.000
LRSLS 3.63* 7.65 0.001
*p<0.01
Table 2. Tests of significance for differences in means for BSC concepts between stages
After Simulation – Before Simulation
Mean Significance
BSC Concept Difference Standard Deviation p
Concepts included in LRPMS
C 1 – BSC perspectives -2.20** 3.49 0.000
C 2 - Financial measures -0.85* 2.96 0.032
C 3 - Non-financial measures -0.03 3.30 0.937
C 4 - Objectives and targets -0.25 3.42 0.571
C 5 – Key initiatives -0.05 2.96 0.895
Concepts included in LRSLS
C 6 – Systems and dynamical view 0.49 4.20 0.372
C 7 – BSC strategy map 1.00* 3.77 0.046
C 8 - Understanding and validation of the cause-
1.47** 2.87 0.000
and-effect relationships
C 9 – Inference about future performance 0.22 3.62 0.642
C 10 - Double-loop learning process 0.44 3.80 0.376
*p<0.05; **p<0.01
Table 2 shows the results of statistical testing to identify differences in means between the stage before
simulation and the stage after simulation for each BSC concept. The lower mean value of LRPMS revealed
by participants after simulation was mainly due to C1 - BSC perspectives (mean difference =-2.20, p<0.001)
and C2 - Financial measures (mean difference =-0.85, p<0.05) concepts. These results suggest that after
simulation the participants changed their perception about the importance of those concepts as they rated on
average a lower level. Results in table 2 also suggest that the better mean value of LRSLS after simulation
was mainly due to C7 - BSC strategy map (mean difference =1.00, p<0.05) and C8 - Understanding and
validation of the cause-and-effect relationships (mean difference =1.47, p<0.001 ) concepts. Thus, on
opposite, it seems that the simulation experience made the participants changing their view on the relevance
217
ISBN: 978-989-8533-12-8 © 2012 IADIS
of these concepts as they rated on average a higher level. The relevance of those BSC concepts was also
evaluated by means of a quantitative content analysis of the final report (table 3). According to that analysis,
the participants perceive on average that C7 (BSC strategy map), C8 (understanding and validation of the
cause-and-effect relationships) and C10 (Double-loop learning process) are the most relevant BSC concepts.
In order to evaluate the relationship between Performance and each variable, a bivariate correlation
analysis was conducted. As it can be seen in table 4, there is some correlation between Performance and
LUPMS–AS (Corr=0.245, p=0.053). However, that correlation is not significant (at p<0.05). Performance is
significant correlated (at p<0.05) with LUSLS–AS, and with LRSLS–CA.
Table 3. Results of quantitative content analysis of the simulation reports
Quantitative Content Analysis of
Simulation Reports
Number of
BSC Concept Mentions %
Concepts included in LRPMS 63 36.0%
C 1 – BSC perspectives 16 9.1%
C 2 - Financial measures 7 4.0%
C 3 - Non-financial measures 18 10.3%
C 4 - Objectives and targets 7 4.0%
C 5 – Key initiatives 15 8.6%
Concepts included in LRSLS 112 64.0%
C 6 – Systems and dynamical view 6 3.4%
C 7 – BSC strategy map 31 17.7%
C 8 - Understanding and validation of the cause-
32 18.3%
and-effect relationships
C 9 – Inference about future performance 19 10.9%
C 10 - Double-loop learning process 24 13.7%
Table 4. Analysis of correlation (Pearson)

Correlation Analysis
Performance
Variables
Pearson Correlation Significance
LUPMS–AS 0.245 0.053
LUSLS–AS 0.298* 0.018
LRPMS–AS 0.019 0.882
LRSLS–AS -0.030 0.817
LRPMS–CA 0.186 0.144
LRSLS–CA 0.299* 0.017
*p<0.05
Table 5 shows the results of multivariate regression analyses of Performance on the independent variables
(model 1). The regression model was then refined (model 5) by performing a stepwise regression in order to
exclude the variables that did not seem to significantly explain the dependent variable and to preserve the
most significant explanatory variables. Regression analysis of Performance on the most significant
independent variables shows significant effects for LUPMS-AS and LRSLS-CA.
These results confirm five of the six hypotheses (table 6). The present study does not support Hypothesis
H1 as from table 1 the mean values of LUPMS for before and after simulation stages were not significantly
different. On average, the participants in after simulation stage showed better LUSLS and LRSLS, with
significant differences evident in Table 1, supporting Hypothesis H2 and Hypothesis H3. The correlation
analysis (table 4) showed no significant relationship between LUPMS-AS and Performance. However, the
results of multivariate regression analysis of Performance on the independent variables (model 5 - table 5)
indicates a significant effect for LUPMS-AS. Consequently, the results provide support of Hypothesis H4.
The results from correlation analysis (table 4) provide support of Hypothesis H5 as it was found a significant
correlation between LUSLS-AS and Performance. LRSLS was measured through two processes: data
obtained from the questionnaires answered by participants after simulation (variable LRSLS-AS) and data
extracted from quantitative content analysis of participants’ simulation report (variable LRSLS-CA).
Correlation between LRSLS-AS and Performance is not significant. However, results reveal a significant
correlation between LRSLS-CA and Performance. Thus, the results provide support of Hypothesis H6.
218
Table 5. Model 1: regression results for all independent variables; Model 5: regression results obtained through a
stepwise procedure
Dependent Variable
Performance
Independent
Variables Standardized Beta Significance
Model 1
LUPMS – AS 0.226 0.088
LUSLS – AS 0.144 0.273
LRPMS – AS 0.111 0.783
LRSLS – AS 0.053 0.894
LRPMS – CA 0.176 0.151
LRSLS – CA 0.336** 0.009
Model 5
LUPMS – AS 0.292* 0.016
LRSLS – CA 0.340** 0.006
*p<0.05; **p<0.01
Table 6. Summary of Hypothesis Testing

Hypotheses Description Results
H1 The use of simulation improves the level of understanding of the BSC Not supported
performance measurement system
H2 The use of simulation improves the level of understanding of the BSC as Supported
a strategic learning system
H3 The use of simulation improves the level of perceived relevance of the Supported
strategic learning system concepts
H4 The level of understanding of the BSC performance measurement system Supported
is positively correlated with Performance
H5 The level of understanding of the BSC as a strategic learning system is Supported
positively correlated with Performance
H6 The level of perceived relevance of the strategic learning system concepts Supported
is positively correlated with Performance
4. CONCLUSION
Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the
business strategy and performance measurement. Many organizations around the world are using the BSC to
define, implement and manage strategy. Nevertheless there exist studies that show in general terms that
managers do not understand the BSC as the measures and perspectives in use are fairly independent, and do
not always mirror the recommended cause-and-effect logic included in the systems perspective of the BSC
approach. Thus, appropriate teaching and training methodologies must be designed in order to improve
subjects learning and comprehension of those concepts. The present study addresses the effectiveness of
teaching the Balanced Scorecard by means of business simulation. An experiment that uses a business
simulator is performed for testing a set of hypotheses about the influence of simulation on the subjects’
understanding of the BSC. The simulation methodology significantly enhanced the understanding of the BSC
concepts related to the strategic management and double-loop learning processes and the systems
perspective. Results also suggested that understanding of those BSC concepts positively influences
simulation performance.
The findings confirm our assumptions on the effectiveness of using simulation for teaching the BSC
approach as a strategic management system (involving balanced scorecards and strategy maps as suggested
by Kaplan and Norton). The simulator operates by participants developing and combining critical resources
in appropriate levels in order to attract and retain customers, while running an efficient company. To reach
this goal, the participants must understand the interdependence among critical resources and variables of the
business case and they must combine these effectively. As students create a causal model (the BSC strategy
map) representing the critical cause-and-effect relations of the business case, and use that causal model and
the feedback information from the BSC performance system for running the firm, they develop an effective
219
ISBN: 978-989-8533-12-8 © 2012 IADIS
understanding about the meaningless and usefulness of the BSC concepts related to the strategic management
and double-loop learning processes.
REFERENCES
Argyris C. 1976. Increasing Leadership Effectiveness. Wiley: New York.
Bakken B, Gould J, Kim D. 1994. Experimentation and Learning Organizations: A Management Flight Simulator
Approach. In: Morecroft J, Sterman J. 1994. Modelling for Learning Organizations. Portland, Productivity Press.
Cannon-Bowers JA, Bowers C. 2008. Synthetic Learning Environments. In J. M. Spector, M. D. Merril, J. van
Merrienboer, & M. P. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology, (pp
317-327). New York: Lawrence Erlbaum Associates.
Capelo C, Dias J. 2009. A Feedback Learning and Mental Models Perspective on Strategic Decision Making.
Educational Technology Research & Development, 57: 629-644.
Craik K. 1943. The Nature of Explanation. Cambridge University Press: Cambridge.
Davidsen PI, Spector JM. 1997. Cognitive complexity in system dynamics based learning environments. In 15th
International System Dynamics Conference: Systems Approach to Learning and Education into the 21st Century,
Barlas Y, Diker VG, Polat S (eds). System Dynamics Society: Istanbul: 757–760.
de Jong T, van Joolingen WR. 2008. Model-Facilitated Learning. In J. M. Spector, M. D. Merril, J. van Merrienboer, &
M. P. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology, (pp 457-467). New
York: Lawrence Erlbaum Associates.
Forrester J. 1961. Industrial Dynamics. Pegasus Communications: Waltham, MA.
Isaacs W, Senge P. 1994. Overcoming Limits to Learning in Computer-Based Learning Environments. In: Morecroft J,
Sterman J. 1994. Modelling for Learning Organizations. Portland, Productivity Press.
Ittner C, Larcker D. 1998. Innovations in performance measurement: Trends and research implications. Journal of
Management Accounting Research 10: 205-238.
Ittner C, Larcker D. 2003. Coming Up Short on Nonfinancial Performance Measurement. Harvard Business Review Nov:
88-95.
Johnson-Laird P. 1990. Mental Models: Towards a Cognitive Science of Language, Inference and Consciousness.
Harvard University Press: Cambridge, MA.
Kaplan R, Norton D. 1992. The Balanced Scorecard – Measures that Drive Performance. Harvard Business Review Jan-
Feb: 71-79.
Kaplan R, Norton D. 2001. The Strategy-Focused Organization: How Balanced Scorecard Companies Thrive in the New
Business Environment. Harvard Business School Press: Boston, MA.
Lingle J, Schiemann W. 1996. From Balanced Scorecard to Strategic Gauges: Is Measurement Worth It ?. Management
Review 85: 56-61.
Lipe M, Salterio S. 2000. The Balanced Scorecard: Judgmental Effects of Common and Unique Performance Measures.
The Accounting Review 75: 283-298.
Maier F, Grobler A. 2000. What Are We Talking About?—A Taxonomy of Computer Simulations to Support Learning.
System Dynamics Review, 16(2), 135–148.
Malmi T. 2001. Balanced Scorecards in Finnish Companies. Management Accounting Research 12: 207-220.
Speckbacher G, Bischof J, Pfeiffer T. 2003. A Descriptive Analysis on the Implementation of Balanced Scorecards in
German-speaking Countries. Management Accounting Research 14: 361-387.
Sterman JD. 1994. Learning in and about complex systems. System Dynamics Review 10(2/3): 291–330.
Stivers B, Covin T, Hall G, Smalt S. 1998. How nonfinancial performance measures are used. Management Accounting
44: 46-49.
220
FACILITATING EXPLORATORY LEARNING IN

SCHOOLS THROUGH VIRTUAL WORLDS:
EXPERIENCES FROM A COURSE RUN AT A SCHOOL
Ahmer Iqbal
University of Jyväskylä
Agora, Mattilanniemi 2, 40100 Jyväskylä, Finland
ABSTRACT
The following paper examines the results of a research study in which a virtual world, Quest Atlantis (QA), was used to
engage students in exploratory learning to teach about water quality issues. The main aim of the research was to find out
how new digital learning environments and educational technology, such as virtual worlds, can be introduced in schools;
what will be the response of the administration, teachers, parents and students; what could be the best teaching
methodology for these digital learning environments; and what could be possible hurdles and hindrances. The research
was comprised of comparison between different teaching methods based on student outcomes and the results of a survey
which was conducted at the end of the study. The research was too big to include all results in this paper thus the findings
of the survey and observations of the researcher are presented here. The survey was answered by students (N=33) and it
included questions about the learning environment, perceptions about the learning process and problems that the students
might have encountered during the study. During the study, the students were divided into three different groups, or
classes, and each group studied using a different teaching method. Thus, there were three teaching methodologies or
learning strategies: direct instruction and using QA; group discussions in class and using QA; and self-exploration using
QA. The results are presented on an overall basis and also on the basis of these teaching methods as it could have
influenced the answers given by the students. The results of the survey shows that majority of the students liked learning
through the virtual world, Quest Atlantis in this case, and would like to continue using it both at home and at their school.
Irrespective of the teaching method used, they perceived that they learned a lot about water quality and think that they
could not have learned more in a traditional class. The biggest hurdles or problems reported by students were: working in
pairs; wrong time to conduct this research; short session times; and technical problems related to computers and the
internet.
KEYWORDS
Virtual worlds, digital learning environments, exploratory learning, teaching methodologies.
1. INTRODUCTION
Educators all over the world throughout the history have been striving to make the process of learning more
engaging for the learners. For this purpose, educators also try to make use of any new technology especially
those which are popular among learners. However, integration of those technologies into the settings of
formal learning institutions, schools and universities, can be a challenge. Virtual worlds are one of those
technologies for which many researchers have proved its potential for education. For example, Iqbal et al
(2010: 3199) presented learning gains in virtual worlds as is presented by many researchers. Barab et al
(2008), Barab et al (2009), Hickey et al (2009), Arici (2008) have shown learning gains when learners used
the virtual world of Quest Atlantis (QA). Similarly, Ketelhut et al (2005) has also shown learning gains using
River City.
However, many researchers have indicated that there are many obstacles which hinder the integration of
virtual worlds in the school or university settings. Dickey (2011: 15-17) discusses that visual representation,
security and costs were main hindrances as perceived by the teachers in his research. Klopfer et al (2009: 18)
presents 9 barriers to adoption for learning games which are curriculum requirements, attitudes, logistics,
support for teachers, assessment, evidence, uses of games, limited view and social and cultural structures.
221
ISBN: 978-989-8533-12-8 © 2012 IADIS
Technical problems, steep learning curve, student security and university liability was identified to be the
major concerns in Second Life (Woods, 2010: 105-108). Koh et al (2012: 56) found that insufficient time,
limited resources, high costs, non-relevance to curriculum, reactions from parents and no or less support from
school are key reasons for not using games in schools in Singapore as perceived by teachers. Ryan (2008), a
PhD student at Lancaster University, presented sixteen ways to incorporate Second life into a classroom.
Most of this research is conducted in western countries. These problems could amplify or more problems
can arise in developing countries. Here a study is presented which is significant as it compares different
teaching methods and also because it was carried out in Pakistan to find out the relevant problems in
integrating virtual worlds in a developing world context. This paper presents a research in which students
learned in QA through three different teaching methods. The research was focused on comparison of learning
gains and experiences between these three different teaching methods. Here we present the experiences of the
students and hindrances that they faced which were collected as survey answers.
1.1 About the Virtual World

The virtual world used in this research is Quest Atlantis (QA) which is currently maintained under the
Atlantis Remixed Project (Atlantis Remixed Project: Online). QA engages students in, what Atlantis
Remixed Project team defines as transformational play through content that is comprised of online, in QA
virtual world and on websites, and off-line activities. The Atlantis Remixed Project team claims to have
demonstrated learning gains in science, language arts and social studies.
1.1.1 The Topic
The topic chosen for this research was water quality which is known as ‘Taiga: the water quality unit’ in QA.
The researchers wanted to choose a topic which was unfamiliar with the students in Pakistan. None of the
students who took part in this study had ever learnt about water quality although some of them were familiar
with the concepts of pH, acids and bases. They were unaware of the effects that water quality can have on the
ecosystem.
The learning takes place in QA at ‘Taiga National Park’ where the park ranger had asked the students to
help him to identify the reasons for declining number of all species of fish in ‘Taiga River’ (Taiga Unit Plan:
Online).
Figure 1. Map of ‘Taiga National Park’.
222
Along the banks of Taiga River live three main groups of people. ‘Mulu’ indigenous people use the river
to grow crops and raise different animals. ‘Build-Rite Lumber Company’ cuts and logs the trees in the park.
Lastly, there is a fishing company, ‘K-Fly Fishing Company’, which arranges fishing tours. ‘Taiga National
Park’ also has a ranger station, lab and 3 water monitoring stations as is shown in Figure 1.
The material used in this research was provided by the Atlantis Remixed Project team in ‘Taiga Unit
Plan’ (Taiga Unit Plan: Online) which is available online.
1.1.2 The Missions
There are five missions in ‘Taiga: the water quality unit’ but due to time constraints it was decided that the
students will try to solve first three missions including an introduction week to using QA. The educational
year in most of the schools in Karachi, Pakistan, runs from April to March. This study started in January and
as the final exams were approaching thus the school administration was not ready to give more time.
The first mission is called ‘Getting a handle on Taiga’ and in this mission the students meet with all the
stake holders of the Taiga National Park. They talk with all of them and get their respective versions of the
possible problems of water quality and who may be responsible for the situation. In doing this, the students
start understanding that it’s not as straight as it might look initially and that the situation is complex. At the
end of the mission the students have to fix responsibility for declining population of fish.
The students continue to talk with different people in the second mission, ‘Digging deeper’, and take
photos of Taiga River. Then they compare the photos taken and try to analyze them in order to come up with
a hypothesis. Students learn about distinguishing between guess and hypothesis.
In mission 3 students collect water samples from three different sampling sites and then take it to the park
lab to analyze it. This mission is called ‘Building a case’ and students learn about and analyze 6 water quality
indicators which are pH, nitrates, phosphates, dissolved oxygen, turbidity and temperature. Based on the
results of the analysis the students inform the park ranger about the group responsible for the problems.
2. METHODOLOGY
The research had mainly two parts. Firstly, to compare different teaching methods based on student out
comes. In order to do this the students (n=48; 27 male, 21 female) were divided into three classes which were
organized according to three different teaching methods. Teaching method 1 (TM1) had 16 students.
Teaching method 2 (TM2) had 15 students. While, teaching method 3 (TM3) was comprised of 17 students.
The teaching methods are briefed later in this section.
The second part was to get an insight into student’s experiences by asking them to answer a survey. The
total number of students who answered the survey was 33 (19 male, 14 female). Out of that, 13 were from
TM1, 12 were from TM2 and 8 were from TM3.
Although, the survey was supposed to be conducted online but the teachers suggested that it will be easier
and quicker to do it on paper. Thus, the students took the survey on paper and later on their answers were put
in the computer.
Throughout the study the researcher observed the whole learning process and had many discussions with
owners, administration and teachers.
2.1 Context of Research

The research was carried out in a privately owned school in Pakistan by a university in Europe (Nordic
countries. The name of the institution is not mentioned to maintain the anonymity of the author). On the
contrary to Europe, the schools in the educational sector in Pakistan can be divided into government-owned
or privately-owned. Privately-owned schools in Pakistan are considered to be of better quality as compared to
the government-owned schools. Therefore, mostly all privately-owned schools, except those run by non-
profit or charity organizations, charge a monthly fee to meet their expenses and to generate profit on their
investments. In return they provide better facilities, teachers and resources.
This context of research is important to mention here because the researcher had to convince owners of
the school, administration, teachers and parents about the importance of the research in order to justify using
the resources of the school. In this research, certificates of participation were given to the school, teachers
223
ISBN: 978-989-8533-12-8 © 2012 IADIS
and students. The school was also given a complimentary gift by the research unit which conducted this
study. The students were so enthusiastic when they heard about this study that they played a crucial role in
convincing their parents.
Another aspect that has to be mentioned here is that the teachers and a group of volunteers went through a
week of training period in which they used Quest Atlantis as the teachers were not accustomed to playing
video games and/or using virtual worlds. This proved to be very helpful when the research was carried out.
2.2 Teaching Methods

All students who took part in the study were divided into three groups or classes and each group or class
learned through a different teaching method or learning method. These three methods can also be called
classroom implementation strategies. These teaching methods are mentioned as under.
2.2.1 Teaching Method 1 (TM1): Explanatory Instruction and Exploratory Inquiry-Based
Learning in QA
In this teaching method the students (n=16) learned through traditional direct instruction method where the
teacher explained the topic and the main concepts in the class. After that, students went to the computer lab
to use QA in pairs in which they undertook the missions.
2.2.2 Teaching Method 2 (TM2): Group Discussion and Exploratory Inquiry-Based Learning
in QA
The students (n=15) used QA in the computer lab in groups of two (pairs) however one of the groups had
three students. Afterwards, the students went into class where they took part in group discussions where they
discussed the topic in larger groups consisting of three to four students and then shared their understanding
with the whole class.
2.2.3 Teaching Method 3 (TM3): Self-Exploration Through Guided Inquiry Instruction Using
QA
There were 17 students who were divided into groups of two (pairs) however one of the groups had three
students in this teaching method. The students learned through self-exploration by using QA on their own
and they were allowed to ask questions and get guidance from teachers if they wish to.
2.3 Survey Design

The survey can be hypothetically divided into three main parts. As the survey was supposed to be taken at the
end of the comparison study, thus the survey contained questions about the experience of learning through a
virtual world, QA in this case. First question asked the students whether they liked learning through QA or
not? They had to respond by choosing one of the following: ‘yes, it was very fantastic’, ‘yes, it was good’, ‘it
was ok’, ‘no, it was not good’ and ‘not at all, it was really bad’. Another question was ‘would they like the
school to continue using QA?’ And they had to choose between ‘no, not at all. It is useless’, ‘yes, it is very
good educational software’ and ‘may be. I am not sure’. Thus, the students had equal positive and negative
and a neutral statement to choose from. The students were also asked whether they would be using QA at
home or not. They answered by choosing ‘yes, definitely’, ‘yes, sometimes’, ‘no, never’ and ‘I do not know’.
The other part of the survey asked students about their perception of learning through QA. They were
asked how much they perceive that they have learned in this research. The answer choices were: ‘I learned a
lot’; ‘I learned something’; ‘I did not learned much’; ‘I learned nothing at all’; and ‘I do not know’. Thus, the
students had equal positive and negative and a neutral statement to choose from. They were also asked
whether they would prefer learning through ‘traditional class’, ‘through QA’ or ‘through a combination of
traditional class and QA’. There were two questions related to this theme, one about the perceived learning
benefit and the other about enjoyment.
In a question they were also asked about the features of QA that they liked. The features about which the
students were asked about were: graphics; teacher guidance system; missions / quests; gaming elements;
interface; customization of the avatar; story; topic; answering system and the learning process.
224
The last main part of the survey was about the difficulties, hurdles and hindrances that the students faced
during the study. They were asked about the problems that might have occurred due to the way it was
conducted. They were also inquired about the problems that they encountered in QA or technical problems
about computers and internet.

The total number of students who answered the survey was 33 out of the 48 students who took part in the
research. The students mainly liked learning through QA in all teaching methods and they would like to
continue learning from QA. Three questions specific to the missions in QA were also asked from the students
in the survey but there results are not presented here as they do not fit in the theme of this article. Main
results are discussed under the following sections.
3.1 About Learning Process in QA

The students were asked three questions about the learning process. First question inquired if they believe
that they have learnt in this research or not. A total of 18 (54.55 %) students agreed that they had learnt a lot
in this study which is more than half of the students as is shown in Table 1. Eight (24.24 %) students
expressed that they learnt something. If we combine the number of students who expressed that they had
learnt a lot with those said that they had learnt something, then the total number of students, 26 (78.79 %),
who expressed that they learnt using QA is quite large. On the contrary, only 6 (18.18 %) students mentioned
that they did not learn much. There can be many reasons for not being sure if they learnt something or
nothing. One of that could be that the topic was totally unfamiliar to them and it was not covered in a
tradition course book as is in Pakistani schools the curriculum is strictly taught through an assigned book.
Thus, if they were not learning from a pre-assigned book they were not sure what and how much were they
learning. Curriculum requirements are one of the barriers to integrate video games, and educational
technology, as was indicated by Klopfer et al (2009: 18). Secondly, they were not made aware of the post-test
results which they could have compared with their peers.
Students of TM2, 8 (66.67 %), were more sure that they learnt a lot than students of TM1, 6 (46.15 %), or
the students of TM3, 4 (50 %). Thus, one can infer that group discussions in class do result in a belief that
one has leant a subject. This could be because the students get a chance to share and adjust what they have
learnt through discussing with peers.
Table 1. How much do you think you learned about water quality from Quest Atlantis?
I learned a lot I learned I did not learned I learned nothing I do not know
something much at all
TM1 (n=13) 6 (46.15 %) 3 (23.08 %) 3 (23.08 %) 0 1 (7.69 %)
TM2 (n=12) 8 (66.67 %) 3 (25 %) 1 (8.33 %) 0 0
TM3 (n=8) 4 (50 %) 2 (25 %) 2 (25 %) 0 0
Total (n=33) 18 (54.55 %) 8 (24.24 %) 6 (18.18 %) 0 1 (3.03 %)
The students were also asked to specify in which of the following methods they could have learnt the
most about water quality: traditional class learning; learning through QA; or a combination of both traditional
class and QA. This was asked to find out their preferences after taking a course which involved a virtual
world, in this case QA. There were 32 students who answered this question. Majority of the students, 22
(68.75 %) as is shown in Table 2, expressed that they could have learnt the most through QA across all
teaching methods. Students of TM3, 7 (87.5 %) out of 8, believed the most that they could not have learnt
more than learning through QA. A majority of students of TM1, 8 (66.67 %), and TM2, 7 (58.33 %), also
thought the same.
225
ISBN: 978-989-8533-12-8 © 2012 IADIS
Table 2. In which of the following method do you think you could have learned the most about water quality?
Traditional class learning Learning through Quest Traditional class learning +
Atlantis (Taiga) Learning through Quest
Atlantis (Taiga)
TM1 (n=12) 1 (8.33 %) 8 (66.67 %) 3 (25 %)
TM2 (n=12) 1 (8.33 %) 7 (58.33 %) 4 (33.33 %)
TM3 (n=8) 1 (12.5 %) 7 (87.5 %) 0
Total (n=32) 3 (9.38 %) 22 (68.75 %) 7 (21.88 %)
In a similar question to that of the previous one, the students were asked about the method in which they
could have enjoyed the most while learning about water quality. Surprisingly, most of the students of TM3, 5
(62.5 %), expressed that they could have enjoyed most in the traditional class learning as is shown in Table 3.
this could be because, as the researcher observed, the students of TM3 were very unsure about whether they
are going in the right direction or not during the study. This created a sense of uncertainty in them. This could
be because they are not used of learning outside the class, that too, from a video game or a virtual world.
Most of the students of TM1, 8(61.54 %), and half of the students of TM2, 6 (50 %), thought that they
could have enjoyed the most while learning through QA. one can infer from these results that it is necessary
to hold simultaneous explanatory lessons in class to keep the confidence of the students in their learning
process.
Table 3. In which of the following method do you think you could have enjoyed the most about water quality?
Traditional class learning Learning through Quest Traditional class learning +
Atlantis (Taiga) Learning through Quest
Atlantis (Taiga)
TM1 (n=13) 2 (15.38 %) 8 (61.54 %) 3 (23.08 %)
TM2 (n=12) 2 (16.67 %) 6 (50 %) 4 (33.33 %)
TM3 (n=8) 5 (62.5 %) 2 (25 %) 1 (12.5 %)
Total (n=33) 9 (27.27 %) 16 (48.48 %) 8 (24.24 %)
3.2 About Likeness of QA

The students were inquired if they had liked using QA during this research or not. About half of them, 16
(48.48 %), agreed that QA is fantastic (Table 4). Another 10 students, 30.30 %, expressed that it was good to
use QA. Thus, if we combine both who said it was fantastic and those who said it was good then most of the
students, 26 (78.79 %), liked using QA. The number was again low for the TM3 group as only 3, 37.5 %,
who agreed that using QA was a fantastic experience.
Table 4. Did you like using Quest Atlantis?
Yes, it was very Yes, it was good It was ok No, it was not Not at all, it was
fantastic good really bad
TM1 (n=13) 7 (53.85 %) 4 (30.78 %) 1 (7.69 %) 1 (7.69 %) 0
TM2 (n=12) 6 (50 %) 4 (33.33 %) 2 (16.67 %) 0 0
TM3 (n=8) 3 (37.5 %) 2 (25 %) 1 (12.5 %) 2 (25 %) 0
Total (n=33) 16 (48.48 %) 10 (30.30 %) 4 (12.12 %) 3 (9.09 %) 0
In another question, the students were asked if their school should continue to use QA or not. Most of the
students, 21 (63.64 %), agreed that the school should continue to use QA (Table 5). The number was highest
in the TM2 group where 10, 83.33 %, agreed with the statement and it was lowest in TM3 in which only 3,
37.5 %, agreed.
Table 5. Do you think that your school should continue to use Quest Atlantis?
No, not at all. It is useless Yes, it is very good May be. I am not sure
educational software
TM1 (n=13) 2 (15.38 %) 8 (61.54 %) 3 (23.08 %)
TM2 (n=12) 1 (8.33 %) 10 (83.33 %) 1 (8.33 %)
TM3 (n=8) 2 (25 %) 3 (37.5 %) 3 (37.5 %)
Total (n=33) 5 (15.15 %) 21 (63.64 %) 7 (21.21 %)
226
The students were also asked if they would be using QA at home or not and 41.67 %, 10, chose ‘yes,
definitely’ while 19.17 %, 7, chose ‘yes, sometimes’. It means most of the students, over 70 % of them, take
QA as a legitimate virtual world for leisure as well.
These results show that overall, with the exception of TM3, most of the students liked using QA and they
think that it is enjoyable and schools should use it in future. However, the way it shall be used in schools
shall be considered carefully as it can affect the satisfaction of the students.
3.3 Features of QA that were Liked

The students expressed their likeness for different features of QA in a question in which they had to specify
their level of likeness for ten features which were: missions / quests; graphics; teacher guidance system;
gaming elements; topic; story; answering system; learning process; customization of avatar; and interface. 28
students answered this question. Most of the students (n=28), 18 (64.29 %) liked the missions/quests the
most (Table 6). Graphics was liked by 17, 60.71 %, and teacher guidance system was liked by16, 57.14 %,
students. The most disliked was the interface as 12, 42.86 %, students said it was ok and about 21.43 % said
that they did not liked it much. The interface of any game, and virtual worlds, should be simple enough
(Kohler, 2012) but it seems it was not in the case of QA. The same feelings were conveyed to the researcher
by the students and by the teachers many times during the research. One of the reasons could be that the
students and teachers were using a virtual world for the first time.
Table 6. How much did you like the following in Quest Atlantis?
Very much It was O.K. Not much Not at all
Missions/ Quests 18 (64.29 %) 9 (32.14 %) 1 (3.57 %) 0
Graphics 17 (60.71 %) 7 (25 %) 3 (10.71 %) 1 (3.57 %)
Teacher guidance system 16 (57.14 %) 10 (35.71 %) 1 (3.57 %) 1 (3.57 %)
Gaming elements 15 (53.57 %) 11 (39.29 %) 2 (7.14 %) 0
Topic 13 (46.43 %) 12 (42.86 %) 3 (10.71 %) 0
Story 13 (46.43 %) 7 (25 %) 2 (7.14 %) 6 (21.43 %)
Answering system 12 (42.86 %) 11 (39.29 %) 2 (7.14 %) 3 (10.71 %)
Learning process 12 (42.86 %) 10 (35.71 %) 5 (17.86 %) 1 (3.57 %)
Customization of the avatar 11 (39.29 %) 11 (39.29 %) 4 (14.29 %) 2 (7.14 %)
Interface 10 (35.71 %) 12 (42.86 %) 6 (21.43 %) 0
3.4 Problems and Hindrances

The problems that were mentioned in the survey can be largely divided into the ones due to the design of the
research and the technical problems. The owner of the school, administration, teachers, students, 13 (39.39
%), and parents, all of them expressed to the researcher that it was a wrong time to conduct this research as
final exams of the students were near and the school nor their students wanted to divert their attention as
promotion of the students to the next grade depends on that. 14 students, 42.42 %, also mentioned that
working in pairs was problematic. The main reason for that, as was mentioned by students during the study,
was that the students could use only one QA account to proceed. The student having the other account was
feeling left behind and some students tried QA from home to cover the difference. Students and teachers also
complained about the short session times as was also mentioned by Koh et al (2012: 56). Each session was
about 40 minutes long as per guidelines from Atlantis Remixed t team but it proved to be short on many
occasions. It is recommended that each session shall be at least 60 minutes long for non-native speakers of
English.
Most of the students, 16 (48.48 %), and teachers also complained about the slowness of the internet.
Although, the internet had sufficient speed but the bandwidth proved to be lacking when all computers tried
to access QA at the same time. This resulted in QA getting stuck many times or the answers not sent
properly. The computers barely met the minimum requirements and had to be upgraded for this study which
resulted in some spending for the school. The technical problems were also a cause of concern in Second Life
(Woods, 2010: 105-108) and have to be sorted out to make the experience better.
227
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. CONCLUSION
This was an important study for many reasons. Firstly, it was one of the few researches that have been
conducted in developing nations on the use of virtual worlds in education especially at schools. Given the
Pakistani context, the research can prove to be very beneficial for future studies and can serve as a starting
point to incorporate digital educational technology in schools. To do this, one must take into consideration
the timing of research, the nature of school system, local context, curriculum and culture.
Mostly, students liked using virtual world in schools and would like to continue using it at their schools.
However, the teaching methodology has to be chosen carefully to make sure the students are satisfied with
the learning process and enjoy it too.
ACKNOWLEDGEMENT
I would like to thank the owners, administration, teachers, students and parents of ‘Winbury Grammar
School’, Karachi – Pakistan, especially Mrs. Ghazala Naushad, for helping me in conducting this research
and taking part in it.
REFERENCES
Arici, A., 2008. Meeting Kids at Their Own Game: A Comparison of Learning and Engagement in Traditional and 3D
MUVE Educational-Gaming Contexts. Doctoral Dissertation. Indiana University, Bloomington, USA.
Atlantis Remixed Project. [Online], Available: http://atlantisremixed.org/#44 [26 July 2012].
Barab, S. et al, 2007. Situationally Embodied Curriculum: Relating Formalisms to Contexts. In Science Education, Vol.
91, No. 5, pp 750-592.
Barab, S. et al, 2009. Transformational Play as Curricular Scaffold: Using Videogames to Support Science Education. In
Journal of Science education and Technology, Vol. 18, No. 1, pp 305-320
Dickey, M. D., 2011. The Pragmatics of Virtual Worlds for K-12 Educators: Investigating the Affordances and
Constraints of Active Worlds and Second Life with K-12 In-Service Teachers. In Educational Technology Research
and Development, Vol. 59, No.1, pp 1-20.
Hickey, D., et al, 2009. Designing Assessments and Assessing Designs in Virtual Educational Environments. In Journal
of Science Education and Technology, Vol. 18, pp 187-208.
Iqbal, A. et al, 2010. Engaging Learners through Virtual Worlds. In Procedia Social and Behavioral Sciences, Vol. 2, No.
2, pp 3198-3205
Ketelhut, D. J., et al, 2005. Inquiry Teaching for Depth and Coverage via Multi-User Virtual Environments. Paper
presented at the National Association for Research in Science Teaching, Dallas, USA.
Klopfer, E., 2009. Moving Learning Games Forward: Obstacles, Opportunities & Openness. The Education Arcade.
Massachusets Institute of technology, [Online], Available:
http://education.mit.edu/papers/MovingLearningGamesForward_EdArcade.pdf [26 July 2012].
Koh, E., 2012. Teacher Perceptions of Games in Singapore Schools. In Simulation & Gaming, Vol. 43, No. 1, pp 51-66.
Kohler, C., 2012. GDC: 5 Things Every Addictive Game Needs, [Online], Available:
http://www.wired.com/gamelife/2012/03/rj-mical-gdc-speech/ [26 July 2012].
Ryan, M., 2008. 16 Ways to Use Second Life in Your Classroom: Pedagogical Approaches and Virtual Assignment,
Lancaster University.
Woods, J., 2010. Improving the Student Experience – Student Support in Virtual Enviornments. In The Journal of Virtual
Worlds and Education, Vol. 1, No. 1, pp 90-110.
Taiga Unit Plan. [Online], Available: http://atlantisremixed.org/centers/Taiga_Unit_Plan_Final.pdf [26 July 2012].
228
EDUCATIONAL SUPPORT SYSTEM FOR EXPERIMENTS

INVOLVING CONSTRUCTION OF SOUND PROCESSING
CIRCUITS
Atsushi Takemura
Tokyo University of Agriculture and Technology
2-24-16, Naka-cho, Koganei-shi, Tokyo 184-8588, Japan
ABSTRACT
This paper proposes a novel educational support system for technical experiments involving the production of practical
electronic circuits for sound processing. To support circuit design and production, each student uses a computer
during the experiments, and can learn circuit design, virtual circuit making, and real circuit making. In the
proposed system, an analysis system that runs on the internet translates the composition of the circuit that is
designed and constructed by the experimenter into the general circuit description language (SPICE). This circuit
translation into SPICE enables the simulation of circuit behavior and indicates the presence of incorrect parts in
the circuit. The analysis system also provides simulated sound, which is synthesized on the basis of circuit
designed or constructed by an experimenter. Therefore, students can individually learn practical circuit making,
behaviors of sound-processing circuits, and sound simulation. The proposed system is applicable to various circuit
types and learning environments, such as computer-aided circuit design and the manufacture of both virtual and real
circuits. The usefulness and effectiveness of the proposed system was evaluated by analyzing circuits made by five
university students in an actual class.
KEYWORDS
Electronic circuit experiment, circuit making, sound signal processing, SPICE
1. INTRODUCTION
Knowledge of the design and production of electronic circuits and their practical applications is important
for education involving technical experiments. Recently, several educational support systems have been
developed to improve students’ understanding of electronic circuits and their circuit-making ability.
Remote learning systems and virtual laboratories are useful for students lacking experimental
facilities (Gurkan, 2008; Oliver, 2009). The simulation systems were developed for learning electronic
circuits used in the field of engineering (Abramovitz, 2011; Fitch, 2011). However, these conventional
systems cannot cope with the wide variety of circuits constructed by individual experimenters because they
are based on all-purpose or ready-made learning tools and are applicable to only specific circuits within a
subject area. Moreover, technological experiments in educational institutions have the following problems:
(1) student-to-instructor ratios are usually very large and (2) mistakes made in an experimenter’s circuits
can cause serious accidents such as electric shocks or fire. It is therefore difficult to ensure efficient and
secure individual instruction.
To overcome these disadvantages, I have proposed an educational support system for distance learning of
experiments involving virtual circuit making (VCM) and real circuit making (RCM) (Takemura, 2011). The
previous system (Takemura, 2012) was developed so that a student can learn circuit design and make
practical applications of the produced circuit in an electric guitar (Fender Stratocaster). However, the
graphical user interface (GUI) used in the preceding VCM systems is not sufficient for practical use. In this
paper, a new VCM system including a more user-friendly GUI is proposed (cf., subsection 2.1). Moreover,
this study improves the applicability of the system so that a student can learn practical circuits for sound
signal processing. The proposed system is applicable to various educational modes and media (e.g.,
virtual laboratories and e-Learning) for practical electronic circuits and experiments including
231
ISBN: 978-989-8533-12-8 © 2012 IADIS
practical circuit making, such as circuit design, VCM, and RCM (cf., subsection 2.3). The usefulness
of the proposed system was evaluated by analyzing the circuits constructed by five students.
2. METHODOLOGY
Figure 1 illustrates the proposed educational support system for experiments involving the design and
construction of sound processing circuits. This system consists of individual users’ computers and a remote
analysis system. To support circuit design and production, each student uses a computer and transmits the
circuit image to the analysis system that runs on the internet. The analysis system performs circuit
recognition of the designed and constructed circuit using image processing techniques. The analysis system
can automatically recognize the circuit construction, and it can also translate the structure of the circuit into a
general circuit description language (SPICE) (Rabaey, 2012). This SPICE translation technique (Takemura,
2011) can indicate how the circuit works, and it can identify incorrect parts that may exist in the circuit. This
is an important step that aims to improve the efficiency and prevent the occurrence of accidents. The
proposed system improves the circuit recognition technique for the VCM and RCM to make it applicable to
educational support systems for the practical circuit making of a sound processing circuit. As shown in Fig.
1, the analysis system can provide each experimenter with simulated sound, which was synthesized using the
designed or constructed circuit. Therefore, individual experimenters can better understand the effects of
sound processing achieved with their circuits.
Figure 1. Schematic of the proposed educational support system for experiments involving production of the sound-
processing circuits.
2.1 Educational Support System for Circuit Design and Virtual Circuit
Making
The VCM system is useful for students lacking the facilities or equipment for RCM (e.g.,
laboratories, circuit components, and measurement instruments). The previous VCM system (Takemura,
2012) allows individual experimenters to use general graphics editors, which are usually provided in a user’s
computer. Therefore, this system does not require software to be used exclusively. A virtual circuit image is
transmitted to the analysis system along with information pertaining to the color coding of each drawn line,
232
which indicates connected circuit components. The analysis system can differentiate between circuit
components and wiring on the basis of line colors, which are defined by the experimenter. However, the
virtual circuit image based on the user-defined color lines is actually poor. Thus, the GUI of the preceding
system is not user friendly.
This paper improves the GUI of the VCM system so that connections of circuit components on a virtual
circuit can be indicated by an experimenter by placing the virtual circuit components and characters
representing their parameters (e.g., IC, resistance, and capacitance) on a template circuit-board (breadboard)
image using the experimenter’s preferred graphic editor. Figure 2 shows examples of the virtual circuit
components. Each experimenter can download these virtual circuit components and the template image from
the analysis system. The experimenter indicates wirings in the virtual circuit by drawing colored lines on the
template image using the graphical editor. The analysis system can perform circuit recognition using image
processing techniques (pattern matching) between the image of a virtual circuit and the database of each
circuit component in the server computer of the analysis system. The analysis system translates the virtual
circuit into SPICE without errors because circuit recognition is based on data that is related to the input of the
virtually constructed circuit.
(a) (b) (c) (d) (e)
Figure 2. Circuit components for the VCM system: (a)–(d) virtual circuit components prepared for connection on the
circuit board image (resistor, potentiometer, capacitor, and IC, respectively), and (e) circuit board image used to construct
a virtual circuit.
2.2 Educational Support System for Real Circuit Making

An experimenter transmits the image of a constructed circuit to the analysis system after making the circuit.
The analysis system can perform pattern matching between the virtually made circuit (drawn using the VCM
system) and the circuit made using the RCM system. This allows the analysis system to automatically
differentiate the layout of each device and the circuit wiring. In addition, the analysis system performs circuit
recognition based on the training data obtained from this approximate differentiation and the circuit device
database (Takemura, 2012). This pattern matching process between the virtual circuit and the real circuit
improves the accuracy of circuit recognition and translation into SPICE, and decreases the computational
cost of the circuit recognition and translation.
2.3 Combination of the VCM and the RCM Systems

The proposed system has an advantage that individual users can choose preferred modes (combination
of VCM and RCM) and media depending on the required purpose or environment. For instance, the
proposed system is applied to the following purposes:
z A student lacking circuit components or equipment (measurement instruments) can virtually
experience circuit making and measurement.
z An experimenter can learn the behavior of the circuit constructed using the VCM or the RCM system on
the basis of circuit translation into SPICE. Measurement instruments are not needed to check the circuit
behavior.
z An experimenter can construct the circuit of an electric guitar (Fender Stratocaster) and practical circuits
for sound signal processing using the VCM or the RCM system depending on the purpose or
233
ISBN: 978-989-8533-12-8 © 2012 IADIS
environment. Moreover, the experimenter can experience the simulated sound obtained from the
constructed circuit without the need for audio equipment (e.g., audio amplifier and loudspeaker).

The proposed system is evaluated by analyzing circuits, which were designed and constructed by five
experimenters in an actual class at Tokyo University of Agriculture and Technology. Figure 3 shows an
example of the results based on a practical sound processing circuit (low-pass filter) that was used to improve
the sounds of an electric guitar. Figure 3(a) shows the diagram of the circuit to be made. Figure 3(b) shows
the incorrect circuit of the sound processing circuit, which was drawn using the experimenter’s computer.
The analysis system automatically indicated the incorrect parts on the circuit image using red broken lines.
The experimenter then corrected the circuit accordingly. Based on the corrected circuit [shown in Fig. 3(c)],
the analysis system recognized the circuit construction, and translated the circuit into SPICE. Figure 3(d)
shows an example of the simulation results (frequency characteristics) obtained from the results of the SPICE
translation. The analysis system provided this simulation result and simulated the sounds, and the
experimenter could therefore learn the behavior of the circuit and the effect of sound processing (synthesized
sound) on the basis of virtual circuit. Figure 3(e) shows the circuit constructed using the RCM system
depending on the correct circuit, which was constructed virtually using the VCM system [Fig. 3(c)]. The
results of the circuit simulation and sound processing obtained from the real circuit made using the RCM
system were the same as those obtained from the virtual circuit, which was previously made using the VCM
system. The user can then verify that the circuit made using the RCM system behaves accurately.
(a) (b) (c)
(d) (e)
Figure 3. Results of automatic circuit translation and simulation using the VCM and RCM systems: (a) diagram of a
circuit to be made; (b) incorrect circuits made using the VCM system; (c) correct circuits made using the VCM system;
(d) SPICE simulation based on the circuit translation of the circuit (e); and correct circuit constructed using the RCM
system.
234
The proposed system was evaluated by analyzing the circuits made by five students at Tokyo University
of Agriculture and Technology. The following positive responses, which pertain to the usefulness and
efficiency of the proposed system, were obtained from all of the students:
z The proposed systems are useful because it enables remote learning of topics such as circuit design
and experiments involving practical circuit making for sound signal processing.
z The proposed system is instructive because individual students can learn the practical circuits
for sound signal processing, and also the behavior of circuits on the basis of SPICE and
simulated sounds.
z The GUI for VCM is user-friendly and useful.
However, there were also a few technical disadvantages and suggestions for improvement:
z A remote educational system to learn both analog and digital circuits for signal processing is
expected.
z The size of the experimenter’s system is expected to be reduced.
4. CONCLUSION
This paper proposed an educational support system for experiments, which are performed in different
environments such as computer-aided circuit design, VCM, and RCM. The system can teach methods related
to the production of basic circuits for processing sound signals. The usefulness and effectiveness of the
system were verified by analyzing electronic circuits made by five students in an actual university class. The
following steps are necessary to practically realize the proposed system:
z The usefulness of the proposed system must be first validated by a larger number of experimenters.
z There is a need for an educational support system, which can be used by users to learn various
techniques and practical circuit making (e.g., larger scale circuits).
ACKNOWLEDGMENT
This study was partly supported by a Grant-in-Aid for Scientific Research (KAKENHI) 22500878 from the
Japan Society for the Promotion of Science (JSPS).
REFERENCES
Abramovitz A., 2011. Teaching behavioral modeling and simulation techniques for power electronics courses, In IEEE
Trans. Education, Vol. 54, No.4, pp.523–530.
Fitch A. L. et al., 2011. An Analog Computer for Electronic Engineering, In IEEE Trans. Education, Vol. 54, No.4,
pp. 550–557.
Gurkan D. et al., 2008.Remote laboratories for optical circuits, In IEEE Trans. Education, Vol. 51, No. 1, pp. 53–60.
Oliver J. P. et al., 2009. Lab at Home: Hardware Kits for a Digital Design Lab, In IEEE Trans. Education, Vol. 52, No. 1,
pp. 46–51.
Rabaey J. M., The Spice Page, <URL: http://bwrc.eecs.berkeley.edu/Classes/IcBook/SPICE/> (accessed Aug. 25, 2012).
Takemura A., 2011. E-learning system for experiments involving virtual and real electronic circuit making by using
network-based image processing technique, Proc. IADIS Conference e-Learning 2011, Rome, Italy, Vol. II,
pp.176–180.
Takemura A., 2012. E-learning system for experiments involving electronic circuit making and sound signal processing,
Proc. IADIS Conference e-Learning 2012, Lisbon, Portugal, pp.407-411.
235
ISBN: 978-989-8533-12-8 © 2012 IADIS
OAEditor - A FRAMEWORK FOR EDITING ADAPTIVE

LEARNING OBJECTS
João Carlos Rodrigues Pereira, Lucídio dos Anjos Formiga Cabral, Ronei dos Santos Oiveira,
Lucimar Leandro Bezerra and Nisston Moraes Tavares de Melo
Universidade Federal da Paraíba / Centro de Informática / Brazil
ABSTRACT
Distance Learning supported by the WEB is a reality which is growing fast and, like any technological or empirical
innovation, it reveals positive and negative aspects. An important aspect is in relation to the monitoring of the activities
done by the students since an accurate online assessment of the knowledge acquired is an open and, therefore, worrying
subject. A way of minimizing such situation is by using technology to monitor the general performance of the student in
the virtual learning environment. This work invests on the construction of the OAEditor framework which allows
teachers to create learning objects in the form of educational hyper documents representing conceptual graphs that can
adapt to the interaction of the students according to their performance, besides enabling a pedagogical monitoring based
on self-evaluative questionnaires which are automatically generated, pondering the navigation and performance obtained.
KEYWORDS
DLE, Web, learning objects, hyper documents.
1. INTRODUCTION
This paper describes a framework to edit the Learning Objects (LOs) able to adapt to single characteristics of
the students in the process of Distance Learning (DL). Most of the LOs produced nowadays do not provide
feedback information supporting their efficiency in the teaching process, registering, for example, if the
student is interacting in the right way with the object and if it is reaching the planned objectives Moodle
(2012), BlackBoard (2012), Amorim (2005). Studies warn that the learning process is made easier when the
teaching methods are in agreement with the learning style of the student Bohlen (1993, pp. 280-281), Graf
(2008).
The solution proposed here is the OAEditor framework integrated to the AVA Moodle that performs the
following functions:
• It enables the creation of learning objects in the format of educational hyper documents representing
conceptual graphs;
• It allows the interaction between students and these learning objects;
• It generates logs in the interaction student-object;
• It stores a question bank;
• It automatically generates questionnaires to evaluate the level of learning of the students by choosing
questions from a question bank that involve their navigation history and their development;
• It generates data reports of the logs and the students’ assessments.
Moodle, the most popular Distance Learning Environment (DLE) among the eLearningGuild users Davis
(2009), allows the aggregation of new blocks of functions – plugins – which broadens its functionalities. The
OAEditor was designed as a plugin of Moodle and it was developed with the PHP language.
1.1 Structured Hyperdocuments as Conceptual Graphs

According to Bush (1945), the human mind operates by associations, following an intricate web of
representations. For that reason there is a necessity of breaking with the conventional hierarchical structure
236
that we find in books, trying to adapt the didactic material to the nature of the human reasoning. The
hypertexts allow the non-linear writing/reading where one can edit and manage the texts in a distributed
environment. With the use of this facility the users have the possibility of interacting, creating new texts,
introducing new documents or modifying the documents which are already there. The OAEditor uses this
process in the creation of the LOs in which the professor creates fragments of information and also the
connections among these fragments implementing Conceptual Graphs (CG), a concept proposed by Sowa
(2004) to a graphic representation of knowledge, where the nodes represent the concepts and the edges
represent the links among the concepts.
1.2 Applications Sensitive to Context

Medeiros (2010, pp. 15-16) presents a summary of some structures in the adaptive environments in relation
to the following characteristics: data acquisition, storage, processing, characteristics of context use, patterns
of the adopted content and communication infrastructure. The OAEditor has some characteristics of these
structures, composing a new proposal, as shown in Table 1:
Table 1. Synthesis of structures and synthesis of the OAEditor
Synthesis of adaptive structures by Medeiros (2010, pp. 15-16)
Synthesis of the
Semantic Adaptation of
CTRL GlobalEdu OAEditor
Learning Space Learning Scenarios
Interactions
with users, Learner Profile, Interactions with the Interactions with the
Physical and
Acquisition Model of the Elements on social user, Model of the user, Model of the
Virtual sensors
student and and Physical Context student student
Curriculum
Ontologies of
Storage content, context XML Ontologies Objects Repository XML
and domain
First-order logic. Inference on the base
Machine learning
Bayesian nets, Logic Rules of knowledge using Logic Rules IF
Processing algorithm-
Fuzzy logic and of IF THEN the behavior rules THEN
(KNearestNeighbor)
probabilistic logic WHEN IF THEN
Recommendation of
Recommendations of
learning objects and
Recommendation Intelligent Adaptation of learning objects and
Use of context the support to the
of contents tutoring learning scenarios support to the
interaction of the
learner´s interaction
learner
LOM, SCORM,
LOM (Learning
IMS Learning
objects) and IMS
Patterns of content Design, OKI, Not applicable SCORM Conceptual graphs
Learning Design
Ariadne and Dublin
(Learning Scenarios)
Core
Infrastructure of
Centralized Distributed Centralized Centralized Centralized
communication
The great difference of the OAEditor is the proposal of structuring the hypertexts as conceptual graphs,
providing the framework a big range of possibilities of implementation.
2. ARCHITECTURE/CHARACTERISTICS OF THE OAEDITOR

The Fig. 1 exposes the number of use cases in the specification of the OAEditor: maintaining learning
objects, maintaining the question bank, logs, interacting with the learning objects, evaluating the learning
process and Data evaluation.
237
ISBN: 978-989-8533-12-8 © 2012 IADIS

Figure 1. OAEditor Use Case Diagram
2.1 Use Case: Maintaining Learning Objects – Available to Teachers/Tutors

In the OAEditor the text is typed in a similar way to the editor of a regular text. One of the most important
characteristics of the OAEditor is that it is adaptable to the student’s profile, and its concepts are available in
three levels: ‘synthetic’, ‘detailed’ and ‘learn more’, where each level presents the same concept, however, it
is presented in different ways in a growing sequence of detailing.
2.2 Interacting with the Learning Objects – Available to Students

The OAEditor uses the authentication model of Moodle to get all the information about the user who is using
the system, information such as login, profile, relationship discipline-student, section, and so on. Two users’
profiles are considered: Teacher/Tutor and Student. The teachers can create/update/exclude LOs, feed the
question bank and follow students individually. The students will be able to navigate through the LOs created
by the teacher, answer questionnaires and see their personal records.
2.3 Logs
The interaction of the students with the LOs produces registers related to their navigation on the LO. With
these logs many questions can be explored:
• Did the student really choose a satisfactory knowledge path?
• Which concepts were not chosen to be navigated?
• A certain concept was not chosen because the student already mastered it or because it was a lack of
attention?
The great gain that it generates to the teacher is the possibility of knowing the individual profile of the
students offering them more precise feedback.
2.4 Maintaining the Question Bank – Exclusive to Teachers

A question bank with good quality questions related to all concepts present in the LO is necessary to make
the generation of questionnaires possible. The defined architecture suggests multiple choice questions with
238
five alternatives and only one possible answer, where each question is related to a specific concept present on
the LO.
2.5 Evaluating the Learning Process

This evaluation is done when the student requests it. Each questionnaire has ten questions which are chosen
in a way to have all the concepts of the lesson, favouring the concepts which were not navigated and the ones
which obtained the lower results quantitatively. With this, the OAEditor breaks the conventional paradigm
that the same questionnaire is applied to all students. When it is answered and submitted, the OAEditor sends
the results back to the student’s assessment, showing the mark and correct answers.
2.6 Data Evaluation

The pedagogical assessment will take place through an individual monitoring with each student. With the
OAEditor the teacher has access to a monitoring area of the students attending the discipline where the way
each student interacted of the learning object is shown. It is possible to visualize how many times each
student accessed each concept, and there is also a table showing the number of questions about each concept
and the number of right and wrong answers to each question together with all the tests taken by the student.
All the tests can also be visualized.
On the page for monitoring the students you can see the tree of concepts of the unit on the left, and it
shows how many times the student accessed each concept. There is also a table informing the number of
questions related to each concept in all the tests done by the student as well as the number of right and wrong
answers. And, lastly, the teacher can access the tests taken by the students.
3. CONCLUSION
The OAEditor framework implemented enables the creation of learning objects to the teacher in a transparent
way. The teacher does not need to have specific knowledge in programming to create a high quality LO. The
OAEditor transforms the act of creating an LO into a simple typing activity. From that moment, the system
works in an automatic way, keeping the navigation logs of the students, automatically generating tests based
on the navigation of the tests already solved, and also generating analytical reports to the teacher. The quality
of the learning object created is directly associated with the expression of the conceptual graph in terms of
completeness/coverage of concepts and established relationship among the concepts.
4. FUTURE WORKS
The main challenge that can be the motivation to a development of future studies is the incorporation
resources of the artificial intelligence (AI) to the OAEditor. The use of the AI would help the development of
a structure which permits:
• The programming of the LOs to suggest individual contents, based on cases, increasing the quantity of
parameters to be considered;
• The creation of a mechanism of intelligent tutoring where the system itself can give the tutoring by
suggesting contents, questions, external material among other things registering cases of success. Besides,
it suggests paths the students should take based on these cases;
• The increase in the number of criteria which compose the generation of assessment questionnaires and
establishing metrics able to detect if the questionnaires were enough to an evaluation criteria;
• The improvement of the reports in order to obtain more consistent data to take the decision, that is,
establish more precise filters to build the reports.
Other perceptions that can be applied latter on are:
• Making the system become collaborative so the students can insert contents in the LOs that could be
assessed by the teacher and made available to the community, making it something similar to a wiki.
239
ISBN: 978-989-8533-12-8 © 2012 IADIS
• Using the data mining techniques to obtain a better benefit from the information related to the students’
logs;
• Making the system, without the intervention of the teacher, suggesting materials for studying, questions to
solve, and so on;
• Identifying if there are patterns or tendencies in the navigation and answers of the students.
REFERENCES
Amorim, J., Armentano, V., Miskulin, M., Miskulin, R. 2005. Uso do teleduc como um recurso complementar no ensino
presencial. Revista Brasileira de Aprendizagem Aberta e a Distância, 4 (1). pp. 1-10.
BlackBoard, 2012, Available at: < http://www.blackboard.com/International/LAC.aspx?lang=en-us />, [Accessed in: 20
de Mar. de 2012].
Bohlen, G. A., Ferrat, 1993. T. W. The effect of learning style and method of instruction on the achievement, efficiency
and satisfaction of end-users learning computer software. Proceeding SIGCPR '93 Proceedings of the 1993
conference on Computer personnel research. Pp. 273 – 283.
Bush, V.1945. As We May Think. Atlantic Magazine. , [online]. Available at: <http://www.theatlantic.
com/magazine/archive/1945/07/as-we-may-think/3881/4/?single_page=true>. [Accessed in: 03 de Nov. de 2011]
Davis, B., Carmean, C., Wagne, E., 2009. Moodle Moves To the Front of the LMS Adoption Pack. Learning Solutions
Magazine. [online]. Available at: <http://www.learningsolutionsmag.com/articles/111/>. [Accessed in: 03 de Nov.
de 2011]
Graf, S., Liu, T., 2008. Identifying Learning Styles in Learning Management Systems by Using Indications from
Students’ Behaviour, in Acvanced Learning Technologies, Eight IEEE International Conference, pp. 482-486.
Medeiros, F., Tedesco, P., Gomes, A. 2010. Arquiteturas de Suporte à Aprendizagem Colaborativa Sensível ao Contexto.
In: Universidade Federal de Pernambuco - Núcleo de Estudos de Hipertexto e Tecnologias na Educação, 3º Simpósio
Hipertexto e Tecnologias na Educação. Recife, Brasil 02-03 december 2010. Available at:
<http://www.ufpe.br/nehte/simposio/anais/Anais-Hipertexto-2010/Francisco-Medeiros&Patricia-Tedesco&Alex-
Gomes.pdf>. [Accessed in: 03 de Nov. de 2011]
Moodle, 2012. Available at: <http://www.moodle.org/>, [Accessed in: 20 de Mar. de 2012].
Sowa, J. F.2004. Conceptual Graphs. Available at: <http://www.jfsowa.com/cg/. 2004. [Accessed in: 03 de Nov. de 2011]
240
PROTOTYPE OF A MOBILE SOCIAL NETWORK FOR

EDUCATION USING DYNAMIC WEB SERVICE
Sandra Costa Pinto Hoentsch, Felipe Oliveira Carvalho, Luiz Marcus Monteiro de Almeida Santos
and Admilson de Ribamar Lima Ribeiro
UFS – Universidade Federal de Sergipe
Av. Marechal Rondon, Jardim Rosa Elze - São Cristóvão/SE
ABSTRACT
This article presents the proposal of a social network site SocialNetLab that belongs to the Department of Computing -
Federal University of Sergipe and which aims to locate and notify users of a nearby friend independently of the location
technology available in the equipment through dynamic Web Service; to serve as a laboratory for research in social
networks and to promote collaborative learning and integration among students, teachers and researchers, providing the
institution’s services and educational resources in full time.
KEYWORDS
Mobile Social Network, Social Network for Education, Dynamic Web Service, Technology Location.
1. INTRODUCTION
The increasing use of mobile devices, especially cellular and its derivatives such as smartphones, have
favored the expansion of available services through the Internet. The site of the ComputerWord published in
February of 2011 that more than half of computing devices sold in the coming years will not be computers
but smartphone, tablets and netbooks (Fonseca, 2011).
According to the magazine Época, published in May of 2010, the world-wide average of virtual friends is
195 persons per user, when in Brazil is 365, being that more than 80% of the Internet users are registered in
at least one social network, classifying Brazil as the country most sociable of the world (Ferrari, 2010).
After these considerations we realize the importance of conducting research on social networks. The
objective of this article is to present the mobile social network SocialNetLab (Social Network Laboratory)
that will serve of laboratory for researches on social networks, will allow users to find friends and be notified
of the proximity of them according to configuration of distance defined by the user, through a movable
device independent of the technology of location available in the equipment, may also provide tools to the
academic community. Besides this introduction, the article is divided in more four sections described below.
In section 2 we conceptualize and describe some social network sites and present the advantages of
SocialNetLab in relation to other social network sites. In section 3 we present the social network site
SocialNetLab as well as the functionalities and applications that will make up the site. In Section 4 we show
the progress of implementation of SocialNetLab. In section 5 we present the final considerations and the
future works.
2. MOBILE SOCIAL NETWORKS

The social network sites or SNS are web-based services that allow individuals to: 1) construct a public or
semi-public profile within a bounded system; 2) articulate a list of other users with whom they share a
connection; and 3) view and traverse their list of connections and those made by others within the system
(Boyd, 2011).
241
ISBN: 978-989-8533-12-8 © 2012 IADIS
There are online social networks (Web-based), mobile social networks and hybrid social networks. A
hybrid social network is initially being developed as online, and migrates or extends to mobile platforms
later (Tong, 2008). What differentiate an online social network of a mobile social network are the
technological facilities of the new mobile devices combined with the features of the social networks online.
The idea of implementing a mobile social network own arose because of the non-existence of a mobile
social network non-commercial and open source to accomplish the researches of the institution.
3. SOCIALNETLAB
The SocialNetLab is an SNS that will allow a user to have access to educational tools and also enable the
user to locate and be notified of the proximity of a friend regardless of location technology available in they
equipment and according with the configuration of distance established by the user, making the SNS more
attractive.
To do the location of the mobile device we will use initially the API of Geolocation that is already
incorporated to the HTML 5 (Meyer, 2011). This API is a hybrid positioning technology proposed by the
working group Ubiquitous Web Applications of the W3C (World Wide Web) consortium. The idea is that
getting the coordinates (latitude and longitude) will occurs when the user accesses the SocialNetLab site.
As the Geolocation API serves only for four types of location technologies, we realized the need to
implement another solution that will serve the higher number of location technologies available in the
market. In an attempt to guarantee the independence of location technology in mobile device is that the idea
appeared of using the Web Service Dynamic (Murray et al, 2006), so that the system will go automatically
search for the appropriate service for the technology of available location in the equipment. In the figure 1
we can see the architecture of the SocialNetLab SNS.

Figure 1. Architecture of the SocialNetLab
In this environment will be provided educational tools that will serve as manager of educational contents,
will function as remote laboratory in electronics, will serve as a judge online and as a tool for learning
through play. These tools are detailed below. Besides these features, SocialNetLab will also serve as a
laboratory for research on social networks.
The figure 2 shows a prototype of the website's homepage of the SocialNetLab. The site of the
SocialNetLab will have the Tools’s link in which will be available the tools directed to education as the
LEW (Laboratory Engineering Web), the ERLab (Electronic Remote Laboratory), the ProgWeb and the
JOnline. These tools are already implemented and will be incorporated in future to the SocialNetLab.
The LEW is a tool that is already in use through a portal and which provides an environment for the
managing information related to teaching, research and extension. (Ribeiro et al, 2011). The JOnline is a
online judge which seeks to support the programming learning (Santos et al, 2011). The ProgWeb is a tool
whose purpose is to build an environment for teaching programming through games using the programming
language Python. (Ribeiro et al, 2011).
242

Figure 2. Prototype of the initial web page of SocialNetLab
The ERLab is a middleware for remote access laboratories (Ribeiro et al, 2012). It allows users to access
various resources (oscilloscopes, signal generators, multimeters, reprogrammable hardware, among others) at
a distance, through Internet. (Feistauer et al, 2010). In figure 3 we can see the homepage of the ERLab and
an oscilloscope in functioning through the ERLab.
Figure 3. ERLab and oscilloscope
4. PRELIMINARY RESULTS
The social network SocialNetLab is being implemented and was divided into three scenarios as shown in
figure 4.
Figure 4. Implementation Scenarios of the SocialNetLab
243
ISBN: 978-989-8533-12-8 © 2012 IADIS
The scenario 02 is in the final phase of the implementation and is being developed for any device (mobile
or desktop) with Internet access can run the software, find friends and be located, without this being
necessary to install and run any application at the client.
The ultimate objective is to implement successfully the scenario 03, allowing, in addition to mobile
devices with Android, all other devices with Internet access have access to dynamic localization. For this we
are studying and researching the best solution to integrate the Dynamic Web Service application (scenario
01) with the API Geolocation application (scenario 02). Such integration could be done by connection A,
where access to the Web service would be made while accessing the browser, or through connection B,
where access would be made when the application accessing the server.
5. FINAL CONSIDERATIONS
The mobile social network SocialNetLab is functioning with the location system via Web Service for mobile
devices that use Android. As for the other types of devices (mobile or desktop) the application is in the final
phase of development, so that the main objective of the implementation is the integration between the
solutions of location Web Service dynamic and API Geolocation making the application always available to
the user and can be run on any device with Internet access. Finally we will take the integration of educational
tools, the tests and the analysis of results found.
As future work we will be integrating all the tools that will compose the educational environment, as well
as adding new features that add value to the SNS. Issues of privacy and security will be also covered in more
detail later.
REFERENCES
Boyd, D. M. e Ellison, N. B. 2011. Social network sites: Definition, history, and scholarship. Disponível em:
http://jcmc.indiana.edu/vol13/issue1/boyd.ellison.html.
Feistauer, L.; Ribeiro, A. 2010. Myerlab: Comunidade de Pesquisa On-Line para a Utilização de Laboratórios de Acesso
Remoto. In: II Encontro de Iniciação em Desenvolvimento Tecnológico e Inovação. Aracaju-SE.
Ferrari, B. 2010. Onde os brasileiros se encontram. Disponível em:
http://revistaepoca.globo.com/Revista/Epoca/1,,EMI143701-15224,00.html
Fonseca, P. 2011. “Boom” nos dispositivos móveis, bolha nas redes sociais. Disponível em:
http://www.computerworld.com.pt/2011/02/09/boom-nos-dispositivos-moveis-bolha-nas-redes-sociais/.
Lecheta, R. R. 2010. Aprenda a Criar Aplicações Para Google Android. Novatec. São Paulo-SP.
Meyer, J. 2011. O Guia Essencial do Html 5. Ciência Moderna. São Paulo-SP
Murray, G; Singh, I.; Brydon, S. 2006. Projetando Web Services com a Plataforma J2ee 1.4 - Tecnologias Jax-rpc , Soap
e Xml. Ciencia Moderna, São Paulo-SP.
Ribeiro, A. R. L; Chella, M.T. ; Santos, L.M.M.A. ; Santos, J.C.S. ; Melo, W.R. 2012; ERLab: A Middleware for
Remote Access Electronic Laboratories. EATIS 2012 - Euro American Conference on Telematics and Information
Systems, 23 a 25 de maio de 2012. ISBN 978-1-4503-1012-3
Ribeiro, A. R. L; Ribeiro, A. R. L; Chella, M.T. ; Santos, L.M.M.A. ; Santos, J.C.S. ; Melo, W.R. 2011. LEW:
Laboratório de Engenharia Web para ensino, pesquisa e extensão. In: II ENINED - Encontro Nacional de Informática
e Educação. Cascavel-PR.
Santos, J.; Ribeiro, A. 2011.Uma proposta de um juiz on-line didático para o ensino de programação. In: II ENINED -
Encontro Nacional de Informática e Educação. Cascavel-PR.
Tong, C. 2008. Analysis of some popular mobile social network system. Helsinki University of Technology.
244
DEVELOPMENT OF ONLINE COGNITIVE AND

ALGORITHM TESTS AS ASSESSMENT TOOLS IN
INTRODUCTORY COMPUTER SCIENCE COURSES
Aimee Theresa Avanceña1, Akinori Nishihara1 and John Paul Vergara2

1
Tokyo Institute of Technology, 2-12-1-W9-108 Ookayama, Meguro-ku, Tokyo, 〒 152-8552 Japan
2
Ateneo de Manila University, Loyola Heights, Quezon City, 1108 Philippines
ABSTRACT
This paper presents the online cognitive and algorithm tests, which were developed in order to determine if certain
cognitive factors and fundamental algorithms correlate with the performance of students in their introductory computer
science course. The tests were implemented among Management Information Systems majors from the Philippines and
among the students of the Information Systems track in a technology high school in Japan. The cognitive factor, Spatial
Scanning, correlated significantly with the performance of both student groups in introductory computer science. Among
the algorithms, Maze Tracing correlated with the university students’ final grade and Binary Search with the midterm
exam result of the high school students. The online tests also showed significant correlations among Searching, Sorting
and Maze Tracing algorithms. The obtained correlations may be considered in creating assessment tools and in designing
learning strategies for introductory computer science courses for both university and high school levels.
KEYWORDS
Computer Science Education, Cognitive factors, Algorithms.
1. INTRODUCTION
Researchers, particularly in the field of Computer Science Education, have endeavored to find ways on how
to address the high drop-out rate and the poor performance of students in computing. These studies (Irons D.,
1982, Rountree, N., et.al., 2004, Simon, et.al, 2006) attempt to identify the factors that determine success in
computing and measure aptitude in programming. In relation to this, the importance of mastering
programming early was emphasized by the ACM Computing Curricula 2001 Final Report which proposes
programming as a prerequisite for many advanced courses in computer science (ACM, 2001). It is also
important to find ways to improve computer science for high school as it is now being considered as an
essential subject along with the traditional STEM (science, technology, engineering and mathematics) (CSTA,
2005).
The main goal of this research is to determine which among known cognitive factors and fundamental
algorithms correlate with the actual student performance in introductory computer science through online
tests. The tests can be used as reference for designing assessment tools for both university and high school
students who plan to study computer science and other technology and computing-related degrees. The
correlation results may also be used in determining possible ways to help students succeed in introductory
computer science, specifically in programming.
2. METHODOLOGY
The first phase of this research covered the initial development of the online cognitive and algorithm tests
and their preliminary implementation among sixty-one (61) second year Management Information Systems
(MIS) majors at the Ateneo de Manila University. These participants have taken up Introduction to
Computing I (CS 21A) and were enrolled in Data Structures and Algorithms (CS 21B) when they took the
245
ISBN: 978-989-8533-12-8 © 2012 IADIS
tests in May 2007. The second phase involved the translation of test instructions to Japanese and their
implementation in September, 2011 among forty (40) students from the Information Systems track of
TokyoTech High School. At the time of the implementation of the tests, these students have already finished
introductory programming but have not yet studied the fundamental algorithms considered in this research.
2.1 The Cognitive Factors Tests

Two sets of online tests were developed for the purposes of this research. One set was based on known
cognitive factors from the Kit of Factor-Referenced Cognitive Tests by the Educational Testing Services
(ETS), a US-based educational institution. Each cognitive factor has corresponding paper tests. The first
cognitive test, Object-Number for the cognitive factor Associative Memory, assesses the ability of the student
to learn combinations of words and numbers. The second is Letter Sets Test for Induction, which aims to find
the rule that makes the letter sets similar and to identity which set does not fit the rule. The third,
Diagramming Relationships test for Logical Reasoning, entails finding the relationships among things in a
group, represented by overlapping circles. The last is Map Planning test for Spatial Scanning, the goal of
which is to find the shortest route between two points in a city map the shortest possible time and according
to a set number of rules (Ekstrom, et.al, 1976). A sample map for the Map Planning Test is shown in Figure 1
below. Previous researches have shown that the aforementioned factors are related to programming aptitude
(Irons, 1982; Scanlan, 1988). For the Japanese high school students, Diagramming Relationships Test was
not included because this particular test uses English words and the authors have no license to modify and
translate actual exam contents.
Sample routes
A to Z passes building 1
E to S passes building 5
P to J passes building 3
Figure 1. Sample map and routes in Map Planning Test
2.2 The Algorithm Tests

Fundamental algorithms, commonly taught in introductory computer science were designed as games that
can simulate how the algorithms work. The first set of games is Card Searching which includes Linear
Search and Binary Search. The goal is to locate a certain number from among ten (10) numbered cards. The
next is Sorting games, where the goal is to simulate Insertion Sort and Selection Sort while sorting ten
numbered cards in ascending order. The third game is Maze Tracing which attempts to simulate how breadth-
first and depth-first search algorithms work. The last algorithm, Tower of Hanoi, checks the student’s ability
to think recursively. It was not implemented among the high school students because recursion is not covered
in their course syllabus. Figure 2 shows the screen components of Card Sorting.
Figure 2. Unsorted cards and data boxes in Card Sorting
246
3. DATA ANALYSIS
The scores from the online cognitive factors tests were collected. For the algorithm tests, data based on the
execution of the games were used to compute the test scores. The test results were then correlated with the
actual performance of the students in their introductory computer science course, the grades in CS 21A of the
university students and the scores in the mid-term exam on algorithms of the high school students. Data
analysis was done using Pearson’s R correlation function in SPSS.
3.1 Correlations of Online Tests with Performance in Introductory Computer

Science
Among the cognitive factors, Spatial Scanning correlated with the performance of the university students in
CS21A (r=.331, R=.1095, sig=.010). This finding is similar to the study of Irons where this factor is one of
those considered to be most relevant to programming aptitude. Spatial Scanning is associated with debugging
skills as it entails finding “particular configurations representing a pathway through the field”. This is
considered as a more psychologically relevant factor in predicting programming skill and is greatly related
with debugging ability (Irons, 1982). Spatial Scanning also correlated with the midterm exam results of the
high school students specifically with their score in the Binary Search problem (r=.383, R=.1466, sig=.015).
This can be related with the ability to apply Binary Search algorithm where a certain scheme is employed for
locating values in an array. Among the algorithms, only Maze Tracing correlated significantly with the
performance of the university students in their introductory computer science class, CS21A (r=.310, R=.0961,
sig=.017). In the case of the high school students, the result of the online test on Binary Search correlated
significantly with the score in the Binary Search problem in the midterm exam (r=.352, R=.1239=, sig=.026).
This particular result implies that it is more substantial to find the correlations between the online tests with
the performance in the actual application of the algorithms rather than with the final grade.
3.2 Correlation Results among Algorithms

The results of the online algorithm tests were also correlated against each other as shown in Table 1. Two
significant correlation results are similar for both university and high school participants, Linear Search with
Binary Search and Insertion Sort with Selection Sort. Linear Search and Binary Search tests are expected to
correlate because both are search algorithms and have the same objective, that of locating a value from a
given set of cards. The same applies to the correlation between Insertion Sort and Selection Sort.
Table 1. Correlation results among the algorithm tests
Correlation values
Algorithms
University students High school students
Linear Search with Binary Search r=.555, R=.3080, sig=.000 r=.859 R=.7378, sig=.000
Insertion Sort with Selection Sort r=.517, R=.2672, sig=.002 r=.511, R=.2611,sig=.001
Binary Search with Maze Tracing r=-.314, R=.0985, sig=.023 r=.465, R=. 2162,sig=.002
Binary Search with Insertion Sort r=.501, R=.2510, sig=.003
Binary Search with Selection Sort r=.375, R=.1406, sig=.011
Maze Tracing with Tower of Hanoi r=.345, R=. 1190,sig=.007
Maze Tracing with Linear Search r=.390, R=.1521, sig=.013
There are a number of differences, though, between the test results of the university students and high
school students. For the university students, Binary Search and Maze Tracing are inversely correlated. These
students who already knew Binary Search could have implemented the algorithm’s strategy in the
corresponding online test but tried free search in Maze Tracing. However, for the high school students,
Binary Search and Maze Tracing have positive correlations. This may be explained by the fact that searching
is naturally applied in maze traversal. Both processes entail some movement from one point to another.
Another possible reason is that the high school students have not yet studied Binary Search at the time of the
online test implementation. Thus, they could have applied free searching in both algorithm tests, that is, not
following a learned method. The same reason may be applied with the positive correlation between Maze
247
ISBN: 978-989-8533-12-8 © 2012 IADIS
Tracing and Linear Search. Binary Search further correlated with Insertion and Selection Sort and Maze
Tracing with Tower of Hanoi for the online test results of the university students but not for the high
school students. These other correlations may be again due to the fact that the university students have
already studied algorithms by the time of online test implementation. Their approach in executing algorithms
is more unified and structured. Generally, the correlations may also be due to the natural connection between
the concerned algorithms. Search strategies are naturally employed when sorting or tracing a maze.
4. CONCLUSION
Online cognitive and algorithm tests were developed and implemented among university students who are
majoring in information systems and high school students who are in a similar course track. The aim was to
find out correlations among the known cognitive factors and fundamental algorithms with the performance of
the students in their introductory computer science course.
A number of significant correlations were obtained and similarities as well as differences in the
correlations between the two groups of participants were observed. The correlation results indicate
consistency in the students’ performance in the online test and demonstrate certain reliability among the
developed algorithm tests. Moreover, the obtained results show close connections among the cognitive factor
Spatial Scanning and Maze Tracing and Binary Search, making them possible factors to be considered in
learning computing fundamentals.
In summary, the online cognitive factors and algorithm tests that correlated with the performance of the
students either with the final grade or exam score may then be considered in designing tools in assessing
student ability to succeed in an introductory computer science course. The said cognitive factors and
algorithms may also be taken into account in the development of a course outline for introductory computer
science and in preparing actual problem exercises for the students.
Future work for this research includes considering other cognitive factors and algorithms for the online
tests. The other data on the online test performances that were not used in scoring the algorithm tests may
also be used in constructing other formulas for scoring the tests. These data may then be used to further
study particular aspects with which the students implement algorithms and determine other factors that can
assess aptitude in introductory computer science courses.
REFERENCES
ACM, 2001, ACM Computing Curricula 2001. Computer Science Volume, In Chapter 7 Introductory Courses, December,
2001, Available at http://www.acm.org/sigcse/cc2001/cs-introductory-courses.html.
Commission on Higher Education, 2001. Revised Policies and Standards for Information Technology Education (ITE).
CHED Memorandum Order, No. 25 Series of 2001, Available at http://www.ched.gov.ph/aboutus/index.html.
CSTA Curriculum Improvement Task Force, 2005. The New Educational Imperative: Improving High School Computer
Science Education, Using worldwide research and professional experience to improve U.S. Schools. ACM, New
York, USA.
Ekstrom, R. B., et.al, 1976. Manual for kit of factor-referenced cognitive tests 1976. Educational Testing Service,
Princeton, New Jersey, USA.
Irons, D. M., 1982. Cognitive Correlates of Programming Tasks in Novice Programmers. Proceedings of the 1982
conference on Human factors in computing systems, ACM Press, New York, USA, pp. 219-222.
Rountree, N., et. al, 2004. Interacting Factors that Predict Success and Failure in a CS1 Course. In ACM SIGCSE Bulletin,
Vol. 36, No.4, pp 101-104.
Sabin, M., et.al, 2005. Designing a pre-college computing course. In Journal of Computing in Small Colleges, Vol. 20
No.5, pp.176-187.
Scanlan, D.A., 1988. The Mental Abilities Associated With Programming Aptitude. Proceedings of the 1988 ACM
sixteenth annual conference on Computer Science, ACM Press. New York, USA, p.737.
Simon, et al, 2006. Predictors of Success in a First Programming Course, Proceedings of the Eighth Australasian
Conference on Computing Education, Darlinghurst, Australia, Vol. 52, pp.189-196.
248
EVALUATION OF MATHEMATICAL
SELF-EXPLANATIONS WITH LSA IN A
COUNTERINTUITIVE PROBLEM OF PROBABILITIES
Jordi Majà Guiu

Universitat Illes Balears, Department de Psicologia
Ctra Valldemossa km 7,5, 07122 Palma (Spain)
ABSTRACT
In this paper different type of mathematical explanations are presented in relation to the mathematical problem of
probabilities Monty Hall (card version) and the computational tool Latent Semantic Analyses (LSA) is used. At the
moment the results in the literature about this computational tool to study texts show that this technique is appropriate for
the case of expository and narrative texts, but there is not evidence with mathematical texts. The technique could help us
to identify which are the better explanations and if this is relevant to explain correct responses.
KEYWORDS
Self-explanation, LSA, implicit inference, reasoning
1. INTRODUCTION: DIFFERENT EXPLANATIONS FOR THE MHD

In this paper different type of mathematical explanations are presented in relation to the mathematical
problem of probabilities Monty Hall dilemma (MHD) (card version, see Tubau and Alonso 2003). We will
see that there are several ways of arriving at the correct response to the dilemma without resorting to the
application of the theorem of Bayes. We will explore these explanations in order to introduce the
explanations provided by the participants.
A simple and completely correct explanation for the participants’ continued belief in the probabilities
after the elimination appeals to the invariability of the probabilities of the initial sets. The elimination is not
informative from the point of view of the probabilities of the initial sets: since the beginning we have known
that at least 1 of the 2 cards held by the informant is different from the ace; the elimination is informative
from the point of view of the individual probabilities of the cards (since the cards of the informant change
from having equal probabilities of 1/3 to having probabilities of 0 and 2/3), but these do not affect the
probabilities of the initial sets.
The illusory response and a possible explanation
The initial probability of each card being correct is the same, 1/3. Once the informant has showed one of
his cards (situation of elimination), we tend to make the illusory inference that “if there are only two cards,
the probability of each of them of being the ace is the same (1/2)”, when the probabilities are in fact the same
as they were at the beginning: for the decision-maker, p (ace) = 1/3, and for the informant, p (ace) = 2/3.
A possible explanation for such an illusory response could draw from a heuristic of similarity, where in
situations in which there are two alternatives, we think the probabilities are equal.
Necessary and sufficient information for the correct resolution of the dilemma
It is important to realise that the key point in the explanation of the dilemma is the situation of
elimination. Any of the factors that facilitate the resolution of the dilemma in fact facilitate the
comprehension of the implications of the situation of elimination. It is at this moment that it is necessary to
bear in mind that the performance of the informant is conditioned, and this is relevant to their response.
Let us consider, furthermore, that to overcome the strong illusory inference, it is not enough only to
consider the variety of possible cases of performance of the informant (that is to say, to think that “if John
has the ace and the 7, he will always show the 7, whereas if he has the 7 and the 8 he will show half of the
249
ISBN: 978-989-8533-12-8 © 2012 IADIS
time the 7 and half of the time the 8), but it is necessary also to consider the globality (or generic sum) of
these and to compare the number of times in which there is the ace and the 7 and the number of times in
which there is the 7 and the 8. For in the case where the informant has the 7, for example, the reasoning
would be as follows: “Every 3n/2 times that he shows 7, n will be with the ace and n/2 with 8.”, and
specifying for a natural number n=2, “Every 3 times that he shows 7, 2 of them will be with the ace and 1
with the 8.” Therefore, if the informant shows the 7, it will be with the ace more times than with the 8; this
reasoning have been explored in several questionnaires (see, for example, Tubau, 2008).
2. EXPERIMENT
A version of the problem was designed (see Appendix for details of the game) and the experiment was
separated among 3 conditions; they were administered at different consecutive stages.
The first condition was a control condition to be compared with other versions in which an explanation is
asked. Condition A included a “Help-sentence” that shows to them the ways in which the informant might
perform (following the line suggested by Johnson-Laird, Legrenzi, Girotto, Legrenzi, & Caverni (1999)) that
was also included in the other versions.
In the conditions B and C, a “Double question” was designed in order to help participants to answer
correctly the dilemma that asked them for an explanation (justification) for their responses. The aim of this
experiment was to see the efficacy of self-explaining the problem and to see, in condition C, if the same
correct explanation given by the participants in condition B would be enough to correctly understand the
problem1.
2.1 Method
Material. See Annex of Majà, 2009. In condition B, question “(a)” will always be correctly answered if the
statement was correctly comprehended. Question “(b)”, which in fact already implies the illusory inference,
is the one that theoretically could imply serious problems; more specifically, it could be similarly difficult to
the two questions (Decision and Probability) that finally were considered in order to evaluate the correct
resolution of the dilemma. Participants. 22 students of the University of Barcelona participated in exchange
for course credit (one of the participants was eliminated due to previous familiarity with the problem); in
condition B, 52 students; in condition C, 58 students of the University of Balearic Islands participated.
2.2 Results
The results show a surprisingly high percentage of switches (correct responses) in the first illusory question
and a strong decrease in the later questions, despite all of them inviting the same illusory inference.
As was expected, in the Decision question (in which the participant was only given the option of
switching or sticking with the initially selected card), better results than in the Probability question (in which
the exact probabilities were also asked) were found in both the B and C conditions (following the tendency
that there is more difficulty expressing the answers by exact numbers rather than relative probabilities).
The difference between the questions of condition A and condition B was significant in the Decision
question; it was not significant in the Probability question. In the condition B, by seeing the difference
between questions, significant differences comparing the Decision question and the Probability question were
also found (45% vs. 20%; x2 (1, N=102) = 7.57, p<.01). Therefore, the fact of giving explanations seem to
especially helpful for the correct comprehension o the problem.
1
In Condition C, instead of the Help sentence, the complete explanation given for participants in condition B was included: i.e., the
correct response, followed by the connective ‘because’, and followed by the explanation provided by most of the participants in
condition A.
250
2.3 Analyses of Justifications
40 38
40
Percentage
30 27 of Justif. 28
Percentage
30 24 26 of Justif.
19 Decision
20 20 16 Decision
13
10 10 10 9
8 8
6 7
10 4 4 4 4 44 4 Probability 10 55 5 75 753 73 Probability
2 2 2
0 0
Inc H C S E N Inc H C S E N
Figure 1. Type of justification and correct responses (%) in the Decision and Probability questions for Condition B and C
Comparing the justifications of the conditions B (in which the justification ‘C’ was the most common one)
and C, there was an increase in the condition C of the type of justifications ‘H’ (not being significant). There
was a significant difference between the type of justification and correct responses to the ‘Decision’ and
‘Probability’ questions, being the correctness of both the Decision and the Probability questions more
predictable from the type of justifications “S” (perhaps more creative).
Table 1. Classification of the justifications of the participants (conditions B and C)
Types of justification2 Examples of justifications (of the LSA LSA (with Difference
participants) coherence3 implicit between
inference) LSA_i and
coherence LSA
C: “It’s more probable that John shows 0.391 0.438 0.047
Repetition of the the ace because if he has the ace and
consideration of the Cases of the 7 he always shows the 7 [and if he
the Help-sentence has the 7 and the 8 he only shows it
half the time]”
N: Long version exemplified with 6 cases 0.47 0.491 0.021
Consideration (explicit) of a “It’s more probable that John shows
consecutive Number of trials the ace when he has the ace and 7
because if they play several times and 0.295 0.34 0.045
always show the 7 it would be that
hides the ace.”
E: “If John shows the 7 it’s more 0.36 0.412 0.052
Double sense (Equivalency) probable that he has the ace because if
of the relationship between John has the ace and the 7 it’s more
the cards that John has and probable that shows the 7.”
the card that he shows
S: “If John shows the 7 it’s more 0.311 0.33 0.019
Invariability of the probable that he has the ace because
probabilities of the initial Sets John always has 2 cards”
H: “If John shows the 7 it’s more 0.284 0.333 0.049
John Hides the ace. probable that he has the ace because
the ace, because he is forced to hide
it.”
“If John shows the 7 it’s more 0.29 0.339 0.049
probable that he has the ace because he
hides the 8 only half of the time
whereas he always hides the ace.”
2
Considering only the correct responses (‘Inc’ expresses ‘Incorrect Responses’)
3
It is considered the value LSAassa 'LSA, Sentence to Sentence, adjacent, mean', the one that show more differences in the results.
251
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.4 Analyses with LSA

There are several authors (for example, Louwerse, 2010) that hold that LSA is not only a computational tool
for text analyses but also it has also implications in order to understand the process of cognition. One reason
to argue this would be that when the computational technique LSA estimates that the coherence of a sentence
is high, it says that on the basis of a mechanism of semantic activation similar to a connectionist one in which
the activation (or coherence) of a sentence happens in relation to the activation of another (a mechanism of
co-ocurrence) (as developed in Kintsch 1988, 1998, for example).
Therefore, LSA could be used in order to study these justifications (explanations) and to see if a specific
implicit inference ('i') would be in the mind of some of the participants that correctly answered the problem
until the end of the questionnaire; the LSA values obtained in a specific text (which includes both (a) a
specific justification and (b) the implicit inference ‘i’) could show that this implicit inference is involved in
the comprehension of the participants.
It is hypothesized that there is an implicit inference on the basis that the literature (Gigerenzer and
Hoffrage, 1995) show that, in probabilistic reasoning, natural frequencies are easier to understand than
relative frequencies, and this is in relation to the fact that a number of consecutive trails in probability have to
be assumed, as it was emphasized at the beginning of the paper. The implicit inference therefore would be:
“If I consider, on a consecutive number of trials, a sum of the diverse possible cases in which John shows 7, I
see that John shows more times 7 when it is with the Ace than when it is with 8.”
In order to obtain the LSA cosine values, the informatic tool Coh-Metrix is used; the full long texts with
justification with/without an implicit inference were introduced. The analyses showed that the difference
between LSA and LSA_i was marginally significant for the comprehension question Decision, t(95)=2.465;
p=0.016, and was significant for the question Probability, t(95)=3.648; p<0.01.
The difference between LSA and LSA_i cosine values seem to show that the implicit inference could be
more present in the case of the participants who would give the longer explanations N and S. Comparing the
results of condition C with condition A, there is a significant improvement in correct responses in the
Probability question; as this increase is only with the increase of number of justifications ‘H’, it cannot be
said that the overcome of an associative effect is accompanied with the processing of the implicit inference.
Looking at the large difference in values for the justification H between LSA vs LSA_i, LSA would predict
that the implicit inference is not necessarily involved in the correct comprehension of the problem.
LSA seem to able to evaluate more creative explanations as well as both short and large explanations.
LSA seem to be a helpful tool to evaluate understandable explanations for education.
REFERENCES
Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats.
Psychological Review, 102, 684-704.
Johnson-Laird, P.N., Legrenzi, P., Girotto, V., Legrenzi, M., & Caverni, J-P. (1999). Naive probability: a mentalmodel
theory of extensional reasoning. Psychological Review, 106, 62-88.
Louwerse, M.M. (2010). Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science
(TopiCS).
Majà, J. (2009). Process of explanation and implicit knowledge in a counterintuitive problem of probabilities. In N. A.
Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 159-
154). Austin, TX: Cognitive Science Society.
McNamara, D.S., Louwerse, M.M., Cai, Z., & Graesser, A. (2005, January 1). Coh-Metrix version 1.4. Retrieved
[September 2012], from http//:cohmetrix.memphis.edu.
Tubau, E. & Alonso, D. (2003). Overcoming illusory inferences in a probabilistic counterintuitive problem: The role of
explicit representations. Memory and Cognition, 31 (4), 596-607.
Tubau, E. (2008). Enhancing probabilistic reasoning: The role of causal graphs, statistical format and numerical skills.
Learning and Individual Differences, 18(2), p. 187-196.
252
CLASSIFICATION OF BRAIN SIGNALS IN BCI SYSTEMS
Josimara A. Silveira and Diana F. Adamatti

Programa de Pós-Graduação em Modelagem Computacional – Universidade Federal do Rio Grande (FURG)
Av. Itália km 08, 96201-900, Rio Grande, RS , Brazil
ABSTRACT
The paper presents a propose of a methodology to treat brain signals in BCI systems. It is an initial work and it tries to
apply artificial intelligence to solve some problems in the area.
KEYWORDS
Brain Computer Interface, Brain Signals, Artificial Intelligence, Classification
1. INTRODUCTION
The interfaces to human use machines, as keyboard and mouse on computers, cameras and joystics on video
games, are commonly used by the most of people. However, there is a part of population that they may need
other ways to interact with electronic equipment due to health problems or accidents, that lead to severe
paralysis. The development of Brain Computer Interfaces (BCI) systems aims to provide this communication
without the use of muscle movements.
BCI systems are enable to communicate based to on neural activity generated by the brain, independent of
the muscle movements. In the case of people with severe paralysis, but the brain is not compromised, the
neural activity to perform a movement is generated, but it does not reach the corresponding muscles, or those
muscles are weak to obey such commands [VALLABHANENI et al., 2005].
The potential of BCI systems to help people with disabilities is obvious. There are several designed
interfaces for people with disabilities [Wickelgren, 2003 apud VALLABHANENI et al., 2005]. Most of these
systems, however, utilizes some type of muscle control, as the movement of the head, neck, eye, or other
facial muscles.
The BCI systems use signals generated voluntarily by the user. Interfaces based on involuntary brain
activities, as generated during a seizure, utilize many of the same principles and components of BCI systems,
but are not included in this study area.
A BCI system has four modules. The first is the acquisition of brain signals, the second is realized pre-
processing of these signals and the third consists of extracting relevant characteristic signals, and the latter is
to classify the sets of characteristics.
This paper present the steps of development of the study and the analysis of brain signals to BCI systems
using Artificial Intelligence (AI) techniques, especially in the stages of feature extraction and classification.
In addition to the steps above define, a fifth step is performed, the validation of the classification.
The paper is strutured in 4 sections. The section 2 presents the main concepts of BCI systems and system
components, techniques for capturing brain signals and concepts and techniques of feature extraction and
translation or classification of brain signals. The section 3 presents the proposed methodology to develop BCI
systems. In section 4, the conclusion are presented.
2. BCI SYSTEMS
In 1929, Hans Berger present a device that later became the electroencephalogram (EEG), which could
record electrical potentials generated by brain activity. After that, some researches speculate that these
253
ISBN: 978-989-8533-12-8 © 2012 IADIS
devices could be controlled using these brain signals. For a long time, it was just a speculation. Just in the
70's, the first ideas and technologies were developed.
Nowadays, BCI systems are tools that can help users to communicate and perform daily activities, despite
a limited success and they are still mainly in research environments. The main users are individuals with
small to severe muscle deficiency. BCI systems have also been developed for users with certain mental
disabilities such as autism or for people who do not have any disabilities, primarily as entertainment in the
gaming industry.
The goal of a BCI system is to allow the user to interact with a device. This interaction is enabled through
a variety of intermediary components, control signals and feedback loops, as presented in Figure 1.
Intermediary components play specific roles to convert the intent into an action. This means that the user and
the device are also integral parts of a BCI system. The interaction is also possible through feedback loops that
serve to inform the status of each system component.
Figure 1. Intermediary components and feedback loops in a BCI System [VALLABHANENI et al., 2005].
A BCI system has the following modules:
a) Acquisition of brain signals techniques:
The brain signals can be captured in invasive or non-invasive way. BCI devices are called “pure” that
they use cortical implants (electrodes implant into the cranium of the patient). It is an invasive technique.
There are many non-invasive methods to measure brain activity. Non-invasive techniques reduce the risk
to users, since it does not require surgery or permanent fixture of electrodes. Techniques such as
Computerized Tomography (CT), Positron Electron Tomography (PET), Single-Photon Emission Computed
Tomography (SPECT), Magnetic Resonance Imaging (MRI), functional Magnetic Resonance Imaging
(fMRI), magnetoencephalography (MEG) and electroencephalography (EEG) have been used to measure
brain activity non-invasively.
However, the electroencephalography is the most widely used method for the BCI signal acquisition.
EEG has a high temporal resolution. It is able to measure the activity every millisecond. Modern devices of
EEG also have a spatial resolution signals with reasonable, superior to 256 electrodes, simultaneously.
EEG is used massivaly in laboratories and real-world environments. It is portable and the electrodes can
be easily placed on the scalp of the user, simply wearing a cap. Furthermore, the EEG systems have been
used in numerous fields since their creation.
b) Pre-process of brain signals and extraction of relevant characteristic:
The study of BCI systems is focused on improving methods to extract features of the collected signals
and translate them into logical control commands to devices that attend more accurately the user's intent. A
existent feature in a signal can be seen as reflecting a specific aspect of physiology and anatomy of the
nervous system [Wolpaw et al., 2000b apud VALLABHANENI et al., 2005]. The goal of feature extraction
methods is to get the specific physiological aspect of the nervous system through a specific serie. The steps
involved in feature extraction and translation.
254
The goal of processing and extraction of features (or attributes) is to describe an item through its
attributes, which should be very similar for the same category, but very different for different categories of
items. The best solution is when the most relevant characteristics within the numerous options available are
chosen.
c) Classification of the sets of characteristics:
The purpose of training (and classification) is to get that users voluntarily produce EEG signals that are
detectable and can be altered to achieve a specific result. While the user is not aware to how and when the
signals are generated, the signal generation process can only be activated by voluntary user actions. However,
these signals can be produced voluntarily through conscious mental activity, such as adding numbers, or an
automatic response to a situation, that requires little conscious effort, as riding a bike.
Translation techniques are developed algorithms to convert the input characteristics (independent
variable) in commands to control device (dependent variable) [WOLPAW et al., 2002]. Some translation
techniques which are widely used in other signal processing areas are adapted to BCI.
There are several types of translation characteristic algorithms. Some use simple features, such as
amplitude or frequency. Others use individual characteristics, while advanced algorithms use a combination
of spatial and temporal characteristics. Current algorithms use linear classifiers, common spatial subspace
decomposition, artificial neural networks, support vector machines and hidden markov models
[VALLABHANENI et al., 2005].
Research are being developed by various groups around the world. The work of Kroupi et al. (2011), at
Ecole Polytechnique Fédérale Lausanne (EPFL), Switzerland, discloses a method for removing noise from
eye movements in EEG signals. The study is a comparison of the use of independent component analysis and
adaptive filtering, concluding that the first is more robust but slower, and it can be used in approaches offline,
while the second is faster but less accurate and it can be used in some approaches whenever speed is more
important than the accuracy [KROUPI et al., 2011].
Waldert and colleagues (2008), at Universitt Tübingen, Germany, presented a study of EEG and MEG
signals during movement of the hand in order to analyze the direction of movement. Still using the Shannon
entropy to quantify the extracted information of the signals. The study showed a modulation in the signals in
both periods of rest and motion. This modulation was stronger in motor areas of the brain. There were 67\%
accuracy and concluded that both the EEG signals and MEG can generate good results [WALDERT et al.,
2008].
3. METHODOLOGY
A methodology to develop BCI systems is proposed in this paper. It is divided in five steps. In the first step,
all brain signals are selected. In our tests, we will be used the signals of the PhysioBank (http://physionet.org)
group. It is a database of biomedical signals and the set of brain signals has 109 healthy peoples (64 electrods
in each individual), used the 10-20 system, the international distribution of the electrodes on the scalp. These
signals correspond to the resting state of the person or to do one of the following tasks: i) imagination of
hand movement; ii) imagination of foot movements; iii) movement of the hands; iv) movement of feet.
The second step is the pre-processing of brain signals. This step is performed to remove noise, making the
relevant information present in the signal easier to be identified. EEG signals often have noise from muscle
movements, especially of the face, as blinking. These movements generate noise that stand out for their
amplitudes in the EEG signals. The pre-processing of these signals can be done by the application of spatial
or temporal filters, or more sophisticated filters, as Independent Component Analysis.Besides the removal of
noise, even in the pre-processing step, the signals can be divided into frequency bands, ie, beta or mu
rhythms and they can be filtered to be analyzed separately. Both noise removal and separation of the
frequency bands will be performed using the software BCI2000 (www.bci2000.org).
The extraction of features of brain signals preprocessed is the third step of the methodology. The amount
of acquired information in a few minutes is huge, and it increases with the number of used electrodes. In this
way, it becomes necessary to extract the relevant characteristics of this signal for classification. It is very
important that the extracted features must be relevant and represent the brain activity required to control the
desired application, to obtain success in the classification step. The step of feature extraction of signals can
255
ISBN: 978-989-8533-12-8 © 2012 IADIS
be removed from the process and the “pure” brain signals can be passed directly to the classifier, but when
features were chosen, there is an increase of the performance of classification algorithms.
Several techniques to BCI feature extraction have been proposed in the literature, which can be divided
into three groups [GODOI, 2010]: i) Methods that analysis the temporal information in the signal; ii)
Methods that analysis the information related to the frequency; iii) Hybrid methods, based on time and
frequency representations.
The classification is the fourth step and it is the conversion of the characteristics into commands for the
applications to be controlled. This step may be achieved by regression or classification algorithms. The goal
of classification is to automatically associate a feature vector to a particular class. This class will identify the
type of cognitive task performed by the user. The classification algorithms, called classifiers, learn to
associate the class to the vector feature vector. For example, a data set is recorded from an individual with a
collection device of brain signals, where that individual must imagine that he/she is moving the right hand by
a certain time, at another time this individual will imagine that he/she is moving the left hand also for some
time. These signals, together imagination signals of other movements, will train the classifiers.
The fifth step consists in to validate the results. To validate the classification capacity of the used
algorithms, the set of available data is separated into two parts. A party will be used to train the classifier and
the other will be used to test it. The validation is usually made together with the classification step, where the
data are separated automatically. A online validation can be done and the brain signals of a person must be
sent in real time to the already trained classifier, which analyze the data received and return the action desired
by the user.
4. CONCLUSION
Further works in BCI technology should focus on basic and applied research. New resources are needed for
the user to have more precise and faster control. Algorithms that use artificial intelligence techniques must be
developed to feature extraction and classification step to become these steps more efficient. Advances in
methods of signal acquisition, such as invasive techniques are also great promise to increase the spatial and
temporal resolution. Better and faster training methods will become BCI devices easier to use (popularization
of the techniques). Increase the consciousness of the potential of BCI technology will be essential to generate
public interest and government support.
This paper presented a initial work in the BCI area. The study of BCI systems requires knowledge from
many different areas such as medicine, computing, electronics, physiotherapy, psychology, and so on, which
makes it highly interdisciplinary. In this way, to start this kind of work, it is not a easy job.
The main contribution that we hope of this works is in the extraction and classification steps of
methodology, focusing in artificial intelligence techniques to solve the open problems in these steps.
REFERENCES
GODOI, A. C. B. Detecção de Potenciais Evocados P300 para Ativação de uma Interface Cérebro-Máquina. Master in
Engineer. Escola Politécnica da Universidade de São Paulo. 2010.
KROUPI, E.; YAZDANI, A.; VESIN, J.-M.; EBRAHIMI, T. Ocular Artifact Removal from EEG: A Comparison of
Subspace Projection and Adaptative Filtering Methods. In: EUROPEAN SIGNAL PROCESSING CONFERENCE
(EUSIPCO 2011), 2011.
VALLABHANENI, A.; WANG, T.; HE, B. Brain-Computer Interface. In: HE, B. (Ed.). Neural Engineering. New York:
Kluwer Academic, 2005. p.85–121.
WALDERT, S.; PREISSL, H.; DEMANDT, E.; BRAUN, C.; BIRBAUMER, N.; AERTSEN, A.; MEHRING, C. Hand
movement direction decoded from MEG and EEG. Journal of Neuroscience, [S.l.], v.28, n.4, p.1000–1008, 2008.
WOLPAW, J. R.; BIRBAUMER, N.; MCFARLAND, D. J.; PFURTSCHELLER, G.; VAUGHAN, T. M. Brain-computer
interfaces for communication and control. Clinical neurophysiology, [S.l.], v.113, n.6, p.767 – 791, june 2002.
256
A PROJECT-BASED LEARNING SETTING TO

HUMAN-COMPUTER INTERACTION FOR TEENAGERS
Cornelia Geyer and Stefan Geisler

Hochschule Ruhr West, University of Applied Sciences, Tannenstr. 43, Bottrop, Germany
ABSTRACT
Knowledge of fundamentals of human-computer interaction resp. usability engineering is getting more and more
important in technical domains. However this interdisciplinary field of work and corresponding degree programs are not
broadly known. Therefore at the Hochschule Ruhr West, University of Applied Sciences, a program was developed to
give teen-aged pupils insights into this area in a project-based learning environment with professional tools. Within the
last 18 month this project was successfully conducted several times with participants of different age.
KEYWORDS
Project-based learning, computer-based learning, user interface
1. INTRODUCTION
In Germany, the number of beginners of technical studies and especially computer science is comparably
low, though the employment situation in this area is quite good. For this reasons several initiatives have been
started, to demonstrate the wide range of activities in technical studies and jobs. Several universities
cooperate with schools and organize open days, guest lectures of professors in schools or project days in
universities.
One additional aspect is the advancement of women in technical jobs. In Germany, every year in April the
Girl’s Day is declared nation-wide, where the school girls can visit companies or universities and especially
get an introduction to occupational fields which are normally dominated by men.
Within this situation we wanted to define a project which gives insights into an area of computer science
which is presumably attractive to young pupils who are interested in technology but also human sciences
and/or design to show that computer science is more than pure coding. We chose a project-based approach
(Blumenfeld, 1991) as we wanted to offer an experience which differs from normal school days, results in a
product, that can be taken home and that stays in mind for a longer time though the project execution time is
rather short.
In the same project, professional tools and workflows should be presented to the participants as these
normally are not available in schools and to demonstrate a more realistic working environment than academic
or especially educational software development tools (see Terwelp and Dahm, 2011 for an overview) could
do. On the other hand, we wanted the group to work on their own and create software which would run on
their own computer or smartphone and have good memories on this day. Last but not least the project should
be able to be carried out with participants of different ages and different levels of previous knowledge in
computer science or programming.
2. FIELD REPORT: UI-DEVELOPMENT WITH TEENAGERS

The project “My personal MP3-Player” was executed several times since April 2011 at the Hochschule Ruhr
West, University of Applied Sciences with 44 pupils aged 14-19. These workshops were integrated in
different forms of cooperation with schools or nationwide programs like Girl’s Day.
257
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.1 The MP3-Player Software

To allow participants with different knowledge in computer science to work in this project, a tool was
required that allows the UI development without any programming, but also should have the possibility to
discuss and change source code if wanted. The Adobe Flash Builder 4.5 was chosen, as it fulfills this
requirement and is due to the Flash base more flexible on the graphical side than other GUI development
tools. Furthermore this tool allows platform independent development incl. Windows-PC, Mac OS X,
iPhone/iPad and Android devices.
With this tool a MP3-Player software was developed in advance in a way, that the UI can easily be
modified, so that everyone can create the desired look & feel without the need of programming.
Programming only is required if the behavior should be changed. The software structure therefore is prepared
to support this with quite low effort.
To avoid that participants stay close to the start configuration of the UI, the buttons and labels are
completely disordered, every button has a different style, every text a different color, etc. Figure 1 shows this
configuration. Though the interface looks very confusing it is fully functional.
2.2 The Workshop

At the beginning of the workshop a short introduction into the field of human-computer interaction was
given. Jointly the typical controls and displayed information of MP3 players were discussed. The topic of
meaningful grouping of elements was dealt with by using cards showing the controls and displayed
information. The group was asked to organize these cards in a reasonable order at the blackboard. In addition
to this UI introduction some options of image editing were shown.
The first phase of creative work in this project was performed unstructured by intention. The participants
worked depending on their preference alone or in pairs on the user interface of a MP3-Player software for
desktop PCs or Android phones. By this, every participant was able to develop own ideas in own speed. To
support the realization with the software tools two tutors for eight groups were present. Button graphics were
downloaded in most cases from Web Sites or self-drawn. Downloaded graphics were edited (mainly change
of colors and cropping single buttons out of images) with either Gimp or Photoshop.
The given user interface was modified by using the graphical editor of Adobe Flash Builder 4.5. Changes
to the prepared behavior could be realized on a limited scale with support of the tutors. But this kind of
requests occurred only rarely, presumably due to the fact, that the behavior of an MP3 player is in main
aspects standardized. Depending on the interest, age and knowledge of the pupils parts of the source code
were discussed with the participants.
Figure 1 shows the start situation for the participants and pictures from the workshops.
Figure 1. Pictures from the workshop and start configuration of the MP3 player
2.3 Results of the UI Development

The pupils developed the user interface of an MP3-Player according to their own preferences within
approximately four hours. The main focus was on the appearance and styling, but also different questions of
interaction concepts were considered.
258
The results have a high variety in their appearance, but are quite similar in functional behavior. This is
likely due to the case, that a well-known stereotype for operating music players exists, which was discussed
in the introduction to the project work.
Another observation is related to the gender. Girls were more focused on aesthetics and the design of the
overall concept while boys often tried to bring in symbols or graphics related to their interests or hobbies
(e.g. favorite soccer club). Figure 2 shows different outcomes of the project.
Figure 2. Different outcomes from the project day
3. EVALUATION
After some of the executed workshops a survey was carried out. The results of the N=24 pupils who took part
in the evaluation overall was very positive. Especially they liked the possibility of creativity in the project
and the practical work at the computer. This strengthens our approach of offering project-based workshops
instead of pure trial lectures to elate someone.
The overall average results on a five-level Likert scale with 1: very good, 5: insufficient of the most
important items were:
• I could gather a first insight into the topic: 1:42%, 2:58%
• The project course fulfilled my expectations: 1:33%, 2:50%, 3:17%
• The content of the project course was interesting: 1:63%, 2:38%
• The content was comprehensibly communicated: 1:58%, 2:42%
Figure 3. Feedback comparison of pupils with and without courses in computer science (c.s.) at school.
Looking more into detail the comparison of pupils with courses in computer science at school (N=13)
with those who don’t (N=11) for most interesting items are shown in figure 3.
The discussion of these results is within the limitation of the small sample size, but is enriched by verbal
comments given back over the teacher a few days after the workshop. So they give a first indication how to
improve the course.
In average it shows that the flexible concept of the workshop, which allows adaptable amount of work
with the source code, is suitable for participants with and without school knowledge in computer science.
Some of the participants with already higher knowledge in computer science tend to prefer some deeper
programming work than was offered so far. On the other hand this needs to be done carefully so that people
with less knowledge will not get lost.
259
ISBN: 978-989-8533-12-8 © 2012 IADIS
Another interesting item to evaluate is, if the course content is able to arouse interest in computer science
and in human-computer interaction in particular. As the third graph in figure 3 shows this is the case.
Especially the person with less knowledge (and lower age) learned more about different aspects of the tasks
of software engineers. Positively mentioned by a high number of participants was the practical work, self-
reliant in defining the goal and technically as long as possible, but getting support whenever required and at
the end having their own software to take home.
4. IMPLICATIONS FOR USABILITY ENGINEERING PROCESSES

The success of working together with teenagers in a professional software tool for user interface
programming could also have some implications on the usability engineering process (Mayhew, 1999) for
software targeted for this group.
A participatory design process (Muller, 2003) seems applicable in a manner that professional UI design
software can be used with the teenagers. Normally either low-fidelity prototyping, e.g. showing screens of
the user interface on paper, is used or a special software version is created. The first variant has the
disadvantage that the important user experience (Buchenau and Fulton Suri, 2000) differs significantly from
the final product, the latter one that additional effort in programming is required.
Further research will be done to explore the range of products and the depth of interaction design which
can be covered with teenagers in such a project.
5. CONCLUSION
The developed project “My personal MP3-Player” was performed successfully several times to inform and
elate teenagers about topics in the area of computer science and especially human-computer interaction. The
project-based approach with professional design and software development tools was according to the
participants’ feedback very well accepted. The goal to arouse interest also was achieved.
In further modifications of the project the aspect of integrating programming blocks for participants with
previous knowledge need to be worked out. The current project duration is around four hours. The base
concept of the workshop allows extending the duration by including other parts of the usability engineering
life cycle like user testing into the project. Both together give the opportunity for a long-term project or
regular course over a complete semester to hold contact between pupils and university. Details need to be
worked out and evaluated.
With the success of this project, another research topic for the future will be the integration of teenagers
into the usability engineering process.
REFERENCES
Blumenfeld, P.C. et al., 1991. Motivating Project-Based Learning: Sustaining the Doing, Supporting the Learning.
Educational Psychologist, Vol. 26, Iss. 3-4, pp. 369-398
Buchenau, M. and Fulton Suri, J., 2000. Experience prototyping. Proceedings of the 3rd conference on Designing
interactive systems: processes, practices, methods, and techniques (DIS '00). ACM, New York
Mayhew, D. J., 1999. The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design.
Morgan Kaufmann Publishers, San Francisco.
Muller, M.J., 2003. Participatory Design: The third space in HCI. The human-computer interaction handbook, LEA,
Mahwah
Terwelp, S. and Dahm, M., 2011. Entwicklungsumgebungen für Informatik-Anfänger. Mensch & Computer 2011:
überMEDIEN|ÜBERmorgen. Oldenbourg Verlag, München, pp. 371-374.
260
USING BIG DATA TO PREDICT STUDENT DROPOUTS:

TECHNOLOGY AFFORDANCES FOR RESEARCH
David Niemi1 and Elena Gitin2

1
Kaplan Inc., Fort Lauderdale, FL
2
Kaplan University, Fort Lauderdale, FL
ABSTRACT
An underlying theme of this paper is that it can be easier and more efficient to conduct valid and effective research
studies in online environments than in traditional classrooms. Taking advantage of the “big data” available in an online
university, we conducted a study in which a massive online database was used to predict student successes and failures.
We found that a pattern of declining performance over time is a good predictor of the likelihood of dropping out, and that
having dependents or being married or in the military reduces the risk of dropping out. The risk of dropping out was
higher for older students, females, and students with previous college education or transfer credits. These results provide
a foundation for testing interventions to help students who are at risk and will also help to inform the development of a
“research pipeline” that will enable rapid experimental studies of new tools and strategies.
KEYWORDS
Virtual education, Online learning, Technology
1. INTRODUCTION
The early years of the 21st century have witnessed an explosion of interest in new technologies for learning
and teaching at all levels of education. Notable examples of applications of technology in education include
the Khan Academy and the massive open online courses (MOOCS) offered by edX, Coursera, Udacity and
other entrepreneurial partnerships. Non-profit and for-profit online universities now provide a wide variety of
postsecondary training, certificates, course credits, and degrees, and traditional colleges and universities have
greatly expanded their online learning opportunities. These and a startling proliferation of other online
opportunities have made it possible for students and educators to pursue a range of alternative learning and
teaching paths not available to them before.
Whether new forms of online instruction can be effective in educating students on a large scale across the
world remains a vexing question, however. At the moment, the evidence is lacking, but the opportunity to
build evidence does exist. The growth of online instruction across the globe in fact opens new opportunities
for data collection, analysis and reporting and for studies of the effectiveness of research-based instructional
strategies, among other things (e. g., Baker, in press; Campbell et al., 2007; Cen et al., 2006; Desmarais et al.,
1996; Means et al., 2010; Romero et al., 2008; Romero et al., 2011).
These opportunities include the possibility of using routinely collected data to evaluate the relative
effectiveness of current and innovative instructional approaches, as well as the opportunity to provide better
information to guide learners and their teachers. In online environments it is possible to collect detailed
real-time data on every action taken by every learner. The existence of these data for thousands, tens of
thousands, and even millions of students studying the same topics or using the same curriculum under
different conditions, gives us new leverages for studying the influence of contextual factors on learning and
learners. This “big data” affordance can help learners by identifying which learning paths might be best for
them, teachers by recommending approaches for helping students who are struggling, and researchers by
enabling them to test principles of learning and instruction in authentic learning environments at scale.
As a first step toward capitalizing on these opportunities, we conducted an initial investigation intended to
use large existing datasets to predict student success and failure in an online university program. Having
261
ISBN: 978-989-8533-12-8 © 2012 IADIS
developed algorithms to flag students needing additional support, we then plan to use both data mining and
experimental methods to test which types of support will be most effective and efficient for which students.
2. BODY OF PAPER
2.1 Analyzing “Big Data” to predict Student Dropouts

The study we conducted involves the mining and analysis of “big data”, which in our case refers to large
existing datasets that can be analyzed to discern patterns in student and teacher performance, identify at-risk
students, and study relationships among important variables, such as attendance, learning, and student
satisfaction.
We conducted this study in an online university that offers approximately 1000 different online courses to
about 60,000 students and that has assembled an extensive database of information on students and their
performance. This university tends to enroll students who are older and for various reasons need an
alternative to a traditional classroom-based university program.
The database we analyzed contains student and faculty background data as well as measures of learning,
student satisfaction, retention, engagement, teacher performance, and postgraduate success. These data make
it possible to test a wide range of research and evaluation questions on a very large scale, as well as to
monitor and respond to student performance, engagement, and motivation.
The study was specifically designed to examine how students’ academic and demographic characteristics
relate to their dropout rates. We analyzed academic and demographic characteristics of degree-seeking
students (N = 14791) enrolled during a two-year period in the online university. Demographic variables
analyzed were age, gender, marital status, military status, previous college education, estimated family
financial contribution, and number of transfer credits from other universities. Academic variables included
measures of student performance available in the online eCollege platform, such as final exam, discussion,
project, and other assignment scores.
Survival analyses (a type of logistic regression analysis) revealed that measures of student performance
that declined over time were significant predictors of the likelihood of dropping out. With respect to the
demographic predictors, we found that age was a significant predictor of retention, with older students more
likely to drop out. Military and married students retained at a higher rate than non-military and unmarried
students, respectively, and students with prior college experience and higher financial contributions from
their families retained at lower rates than students without these characteristics.
Results of the analyses are presented in Table 1, with the last column showing the effect of each predictor
variable on the risk of students dropping out. The outcome we modeled is student retention, coded as a
dichotomous variable with students who dropped out receiving a “1” score on the variable.
For comparison purposes we also calculated odds ratios based on a logistic regression analysis; this
analysis produced the same outcomes as the survival analysis.
262
Table 1. Predicting student dropouts

Parameter Standard Chi- Pr > Hazard Effect on
Predictor DF Estimate Error Square ChiSq Ratio dropout risk
Reduces risk
Age between 29 and 38 1 -0.50719 0.04485 127.9116 <.0001 0.602
by 40%
Reduces risk
Age between 38 and 45 1 -0.07808 0.04767 2.6833 0.1014 0.925
by 7.5%
Increases risk
Older than 45 1 0.26797 0.04479 35.7943 <.0001 1.307
by 30.7 %
Increases risk
Has transfer credits 1 0.86241 0.04427 379.5799 <.0001 2.369
by 236 %
Increases risk
Enrolled in 200 level courses 1 0.07112 0.07777 0.8365 0.3604 1.074
by 7.4%
Reduces risk
Enrolled in 300 level courses 1 -0.31259 0.07711 16.4345 <.0001 0.732
by 26.8%
Reduces risk
Enrolled in 400 level courses 1 -1.07894 0.08296 169.1268 <.0001 0.34
by 66%
Reduces risk
In the military 1 -1.3035 0.08964 211.4631 <.0001 0.272
by 72.8%
Increases risk
Previous college education 1 0.12529 0.04324 8.3962 0.0038 1.133
by 13.3 %
Increases risk
Female 1 1.20342 0.06681 324.4385 <.0001 3.331
by 330 %
Estimated financial contribution Increases risk
1 0.15667 0.03569 19.275 <.0001 1.17
from family by 17 %
Reduces risk
Married 1 -0.44225 0.03807 134.98 <.0001 0.643
by 35.7%
Reduces risk
Has dependents 1 -0.86645 0.05448 252.9138 <.0001 0.42
by 58%
Above median score on Reduces risk
1 -0.72133 0.0497 210.6254 <.0001 0.486
discussions by 51.4%
Above median score on final Reduces risk
1 -1.4674 0.16955 74.9057 <.0001 0.231
exam by 76.9%
Reduces risk
Above median score on projects 1 -0.96057 0.06153 243.6801 <.0001 0.383
by 61.7%
Above median score on course Reduces risk
1 -1.04862 0.05681 340.7227 <.0001 0.35
review by 65%
Above median score on other Reduces risk
1 -0.98675 0.05474 324.9456 <.0001 0.373
teacher-graded assignments by 62.7%
3. CONCLUSION
3.1 Moving Toward Rapid Experimentation

The finding that declining performance over time is related to dropout tendencies, while not surprising, is a
particularly useful one for us. It represents a first step on the path toward generating data for faculty and
administrators that will enable them to provide additional support to students who have a high likelihood of
failing or dropping out. Results on student background variables will serve a similar function, enabling us to
build profiles of students who may need more support to succeed in the program. Some of our findings for
these variables were surprising; for example that females, unmarried students and students with transfer
credits or dependents are more likely to drop out. We can speculate about these relationships but it will be
more useful to conduct follow-up investigations, including qualitative studies with representative samples of
students, to explore why some groups drop out at higher rates.
As we continue to collect data over time we can also confirm or disconfirm the strength of the
relationships we have found and possibly discover new ones. We are currently developing prior knowledge
measures in several courses and expect, based on extensive previous research on the effects of prior
263
ISBN: 978-989-8533-12-8 © 2012 IADIS
knowledge on learning (e. g., National Research Council, 1999; Sweller et al, 2011), that these will be strong
predictors of student success, failure and retention in online courses. Ultimately we will be able to determine
which combinations of indicators and performance patterns constitute “red flags” requiring immediate, strong
intervention and which may call for less intensive strategies.
As valuable as these analyses of routinely-collected big datasets may be, however, they give us only part
of the information needed to build more effective evidence-based educational programs. Another critical
piece is the testing of new instructional, motivational and support strategies to help students who are having
difficulty and may be likely to drop out. To determine what kinds interventions will be most effective for
which students will require experimental studies in which students are randomly assigned to different
instructional conditions, so that alternative hypotheses for observed changes or differences in student
performance (such as differences in initial student ability levels) can be ruled out. In this case we can take
advantage of another technology affordance for research: online learning systems make it possible to run
large numbers of randomized control trials far more rapidly than would otherwise be possible (Shadish and
Cook, 2009).
We are currently developing “research pipelines”, or platforms and processes that will to enable us to
randomly assign individual students, class sections, or classes to different instructional experiences. The
pipelines will make it possible to build up knowledge on scalable improvements incrementally, testing the
impact of one variable at a time, but quickly over time. Big data analytics in combination with experimental
testing will thus enable us take advantage of the tremendous scale of the online university to find out what
works more quickly than would otherwise be possible–and have the data to prove it. (As a final note, it is
worth mentioning, however, that the online university in this study primarily serves a high percentage of
students who for one reason are another are not able to attend or are not interested in attending traditional
college and university programs, so results may not generalize to students in those programs.)
REFERENCES
Baker, R.S.J.d, in press. Data Mining for Education. To appear in McGaw, B., Peterson, P.,Baker, E. (Eds.) International
Encyclopedia of Education (3rd edition). Elsevier, Oxford, UK.
Campbell, J. P., DeBois, P. B., and Oblinger, D. G., 2007. Academic Analytics: A New Tool for a New Era. EDUCAUSE
Review. EDUCAUSE, http://net.educause.edu/ir/library/pdf/erm0742.pdf
Cen, H., Koedinger, K., Junker, B., 2006. Learning Factors Analysis - A General Method for Cognitive Model Evaluation
and Improvement. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan,
pp. 12-19.
Desmarais, M.C., Maluf, A., and Liu, J., 1996, User-expertise Modeling with Empirically Derived Probabilistic
Implication Networks. In User Modeling and User-Adapted Interaction, Vol. 5, No. 3-4, pp. 283-315.
Mayer, R. E., amd Alexander, P. A. (Eds.), 2011. Handbook of Research on Learning and Instruction. Routledge, New
York, USA.
Means, B., Toyama, Y., Murphy, R., Bakia, M. & Jones, K., 2010. Evaluation of Evidence-based Practice in Online
Learning: A Meta-analysis and Review of Online Learning Studies. US Department of Education Office of Planning,
Evaluation, and Policy Development. http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf
National Research Council, 1999. How People Learn. National Academy Press, Washington DC, USA.
Romero, C., Ventura, S., Pechenizkiy and Baker, R.S.J.d, (Eds.), 2011. Handbook of Educational Data Mining. CRC
Press, Boca Raton, FL, USA.
Romero, C., Ventura, S., Espejo, P.G., Hervas, C., 2008. Data Mining Algorithms to Classify Students. Proceedings of
the First International Conference on Educational Data Mining. Montreal, Canada, pp. 8-17.
Shadish, W. R. and Cook, T. D., 2009. The Renaissance of Field Experimentation in Evaluating Interventions. In Annual
Review of Psychology, Vol. 60, No. 1, pp. 607–629.
Sweller, J., Ayers, P., & Kalyuga, S., 2011. Cognitive Load Theory. Springer, New York, USA.
264
EFFECTS OF PEER-TUTOR COMPETENCES ON

LEARNER COGNITIVE LOAD AND LEARNING
PERFORMANCE DURING KNOWLEDGE SHARING
Ya-Ping (Amy) Hsiao, Francis Brouns, Jan van Bruggen and Peter B. Sloep
Centre for Learning Sciences and Technologies (CELSTEC), Open University of the Netherlands
P.O. Box 2960 6401 DL Heerlen The Netherlands
ABSTRACT
In Learning Networks, learners need to share knowledge with others to build knowledge. In particular, when working on
complex tasks, they often need to acquire extra cognitive resources from others to process a high task load. However,
without support high task load and organizing knowledge sharing themselves might easily overload learners’ limited
cognitive capacities because learners first have to find relevant peer tutors (i.e., those who provide help) and then
maintain the social interaction. We propose to design a peer-support system that selects tutors and provides support
during knowledge sharing. The pilot study reported here investigated the effects of two peer tutor competences, tutoring
skills vs. content knowledge, on tutees’ (i.e., those who need help) cognitive load and learning performance. The results
show that tutees supported by tutors with tutoring skills experienced lower cognitive load and had better learning
performance than did tutees supported by tutors with content knowledge. This is in line with our assumption, but for
confirmation we need to gather more data in a full study. We need to first use a task that requires learners to rely on
others to trigger higher cognitive skills to deal with high task load. Secondly we need to find a modus to ensure that the
tutors follow the instructions to apply the particular competence.
KEYWORDS
Peer-tutor competence, content knowledge, tutoring skills, knowledge sharing, cognitive load, tutor selection.
1. INTRODUCTION
A Learning Network (LN) is a particular kind of online social network that is dedicated to learning (Sloep
2009). In LNs, learners are themselves responsible for sharing knowledge with their peer learners. When
organizing knowledge sharing themselves, learners first have to find out relevant knowledge sharers and then
maintain the social interaction with others to reach a shared understanding and knowledge building. To help
finding relevant knowledge sharers, our colleagues have developed two tutor selection systems that
automatically assign peer learners with relevant content knowledge to act as peer tutors to help answer each
others’ content-related questions (Van Rosmalen 2008; De Bakker 2010). However, due to the heterogeneous
group composition of LNs it is not always possible to find tutors with content knowledge related to tutees’
questions. In addition, the questions or problems that participants of LNs have are often rather complex: they
are authentic problems originating from real-life contexts, such as the working place. When working on
complex tasks, learners have to allocate many of their cognitive resources to process numerous information
elements and element interactivity and this imposes a high cognitive load (Sweller et al. 1998). Cognitive
load (or mental workload) refers to the learner’s limited cognitive capacity actually allocated on performing a
particular task and it has been recognized as an important factor that influences learner performance (Sweller
et al. 1998; Hart 2006). We surmise that tutors who provide content-related knowledge only cannot alleviate
their tutees’ cognitive load.
Furthermore, maintaining the social interaction to reach a shared understanding or build knowledge on
complex tasks requires certain pedagogical and process-facilitation skills (King 2007; Roscoe and Chi 2008).
These skills go beyond the knowledge transfer or information exchange that content tutors often offer. If it is
not possible to always find tutors with content knowledge precisely related to tutees’ questions, nor do tutors
with content knowledge guarantee effective tutee learning, it is therefore necessary to consider tutors with
265
ISBN: 978-989-8533-12-8 © 2012 IADIS
tutoring skills. However, while most tutoring studies have examined effects of tutors’ content knowledge on
diverse dependent variables of tutees, there have not been many studies that measured effects of tutors’
tutoring skills on tutees’ learning. Studies of reciprocal peer tutoring have compared the effects of supporting
learners with and without tutoring skills on learner performance (King 1994; King et al. 1998; Nath and Ross
2001). Their support of tutoring skills was to make sure that learners would demonstrate tutoring skills to
elicit certain social interactions and trigger cognitive processes that contribute to learning. Some studies have
shown that the learners with tutoring skills outperformed the control groups on knowledge or comprehension
tests (King 1994; King et al. 1998). However, in LNs knowledge sharing involving peer support is more akin
to asymmetric rather than reciprocal peer tutoring. From these studies we can therefore only speculatively
claim that tutoring skills are likely to promote tutee learning. Considering human limited cognitive capacities,
we need to redesign our previous tutor selection system to figure out what makes an effective tutor and to
design support for the social interaction process. To achieve this, we first of all need to investigate the effects
of peer-tutor competences, i.e. content knowledge vs. tutoring skills, on tutee learning. In this pilot study we
investigate whether it is possible to determine an effect of the two types of peer-tutor competence when
tutors do not receive prior training but instead receive a tutoring guide to follow.
2. METHOD
Two computer science classes jointly consisting of 28 students from a pre-university secondary school in the
Netherlands took part in this pilot. We assigned students in half of classes to act as tutors and the other half as
tutees. The tutor role was split: a tutor was either a tutoring skills (TS) or a content knowledge (CK) tutor. In
total there were seven TS and seven CK pairs. The intervention of this study was to provide tutors with a
tutoring guide including additional course materials (for CK tutors) or instructions of tutoring skills (for TS
tutors) to assure that tutors had either CK or TS competence. The instructions that TS tutors received
consisted of general rules for pedagogical and process-facilitation skills as well as specific step-by-step
directions that guide tutors to perform pedagogical and process-facilitation skills (King 2007). This study
consisted of two sessions in 2.5 hours. Two websites were set up for these two sessions: the course site
contains the learning material of Sex and Evolution as well as the task site contains task instructions, chat and
wiki tools. During the first session, students studied the learning material on the course site and then they
took the pre-test and filled in the Tutoring Skills Questionnaire. There was a 15-minute break between these
two sessions. The second session was started with a brief introduction of peer tutoring, task requirements and
use of chat and wiki tools. Then students had 50 minutes to work in pairs to complete the essay task. While
performing the task, students no longer could consult the course material. When working on the task,
students had to use a chat to communicate with each other and they were not allowed to talk face-to-face.
Tutees had to write the essay in a wiki and to “publish” the wiki page frequently to allow the tutors to read up
on the tutee’s progress. Tutors could only read the wiki page, but they could not edit it. To view the updated
wiki page, tutors had to “refresh” the wiki page frequently. Tutors used a respective tutoring guide to help
tutees work on essay answers through the chat. After the task, students had 20 minutes to indicate
experienced cognitive load on the NASA-Task Load Index (see Table 1) (Hart 2006), to take the post-test
and to fill in an evaluation survey.
3. FINDINGS
Pre-measures. Both groups of tutees estimated their tutoring skills to be quite high, 63 out of possible 75.
The low pre-test scores showed that tutees of both groups did not have sufficient prior knowledge though TS
tutees scored slightly higher than CK tutees. For both pre-measures, there was no difference between tutees
assigned to TS or CK pairs.
Tutee learning performance. Tutees’ low post-test scores indicated that working on the essay task did
not help students learn much related to the course materials. In addition to the same items of the pre-test,
there were five task-related items in the post-test. CK tutees performed better on five task-related items than
TS tutees: this means that additional course materials might help tutees understand the topics included in the
essay task better than instructions of tutoring skills. Both groups of tutees performed equally well on essay
266
answers though TS tutees performed slightly better than CK tutees. Qualitatively, TS tutees answered more
topics completely than CK tutees whereas CK-tutees answered more topics partially than TS-tutees.
Tutee cognitive load. TS tutees’ total cognitive load is lower than CK tutees’ and lower on all six
dimensions. In particular, TS tutees experienced much lower cognitive load on physical demand and
frustration than CK tutees.
Table 1. Tutee cognitive load on six dimensions of NASA-TLX
Mental Physical Temporal
demand demand demand Performance Effort Frustration Total
Median 12.00 3.00 10.00 6.00 9.00 2.00 41.00
TS
SD 4.27 3.82 6.05 3.70 4.08 5.68 14.60
Median 14.00 11.00 10.50 8.00 13.00 10.50 68.00
CK
SD 5.59 4.32 4.66 4.67 3.93 2.78 20.01
Tutee cognitive load and learning performance when their tutors applied the tutoring guide. Chats
were analyzed for evidence of adherence to the assigned tutoring competence. Only two of the TS tutors
applied the specific step-by-step instructions and five of the CK tutors referred to the additional course
material. Table 2 shows that TS tutees performed better on all performance measures and they experienced
lower cognitive load than CK tutees.
Table 2. Performance measures and cognitive load of TS tutees and CK tutees supported by tutors who actually applied
intervention during chats
TS-tutees (n = 2) CK-tutees (n = 5)
M SD M SD
pre-test 5.00 2.83 3.75 .96
post-test 6.50 3.54 4.40 1.52
5 task-related items 4.00 0 3.60 1.67
essay 7.92 1.29 6.71 1.78
total cognitive load 51.00 14.14 68.20 10.85
4. CONCLUSION AND DISCUSSION

Although TS tutees had higher scores on the essay task and post-test, CK tutees seemed to have higher scores
on the five task-related items than TS tutees. From how multiple topics were answered in the essays, we
surmise that the different tutor competences influenced how tutees worked on the task. The diverse examples
included in the additional course materials helped CK tutees answer more topics partially while the
instructions that TS-tutors used helped TS tutees answer topics more completely. When looking into the data
of tutees supported by tutors who indeed applied the intervention of additional course materials and tutoring
skills, the results in Table 2 turned out to be different from the group data: TS tutees performed better than
CK tutees on both test and task measures. During knowledge sharing, CK tutees in general experienced more
cognitive load than TS tutees. In particular, the much lower score on frustration corresponds to findings of
our previous study (Hsiao et al. in press). We observed that some CK pairs completed the task much earlier
than the expected time of 50 minutes. Although using additional course materials helped CK tutees complete
the task earlier, CK-tutees still indicated a high cognitive load.
The findings of this pilot are hopeful in the sense that discerning CK and TS tutoring competences seems
to make sense. In order to corroborate our first findings, we need to improve three aspects for the full study.
First, in this study we tried out different performance measures in order to find out which could clearly
answer the research question in the full study. The pre- and post-test that measured students’ understanding
of the entire course could not appropriately gauge the effects of the intervention on the essay task since this
essay only covered several topics of the course. Thus, to better measure learning performance, we should
either expand the number of task-related items to a longer test or use the essay answers that directly reflect
how students learn from working on the task. Second, tasks with different complexities require different
amount of cognitive resources to deal with the task load. Complexities of essay questions not only depend on
the interactivity of information elements but also on the level of cognitive skills required. The essay question
267
ISBN: 978-989-8533-12-8 © 2012 IADIS
of this study is still a relatively simple task: comparing and contrasting facts require a lower level of cognitive
skills, namely understanding. From satisfactory essay scores, we surmise that tutees might be able to answer
the essay by merely retrieving learned factual knowledge from their own or their tutors’ memory instead of
performing higher cognitive skills to process the task load. This corresponds to our model that when working
on simple tasks learners do not need to resort to knowledge sharing or that knowledge sharing does not
contribute to better learning effects (Hsiao et al. 2011). Third, considering the online characteristics of LNs,
we supported tutors with certain competences by giving them ready-to-use tutoring guides instead of giving
them a prior training, as is commonly done in online collaborative learning studies or peer tutoring in face-to-
face classrooms. The results showed that five of seven CK tutors and two of seven TS tutors and applied the
respective intervention. Our findings confirm the findings of peer tutoring studies: without training, peer
tutors seldom demonstrate certain skills to fulfill their role tasks (Nath and Ross 2001). Our observations
during the experiment might explain why TS tutors did not use tutoring skills: i) tutors might not have
sufficient time to read all of the textual instructions, ii) tutors might not realize the relevance of using the
general instructions with answering the essay question, and iii) students might have developed certain
internal scripts of dealing with such essay questions comparing facts and thus they did not need to use our
intervention.
To conclude, for the full study we need to oblige tutors to apply the support of competences as we
expected, in particular for tutors to apply tutoring skills. In addition, we should implement the full study with
a more complex task that requires higher-order cognitive skills such as analysis, evaluation and synthesis, to
make sure that learners benefit from the kind of knowledge sharing with others that triggers extra cognitive
processes.
REFERENCES
De Bakker, G., 2010. Allocated online reciprocal peer support via instant messaging as a candidate for decreasing the
tutoring load of teachers. PhD doctoral dissertation, Eindhoven University of Technology.
Hart, S. G., 2006. NASA-Task Load Index (NASA-TLX): 20 Years Later. The Human Factors and Ergonomics Society
50th Annual Meeting. Santa Monica: HFES., San Francisco, pp. 904-908.
Hsiao, Y. P., et al., 2011 Cognitive load and knowledge sharing in Learning Networks. Interactive Learning
Environments, pp. 1-12. DOI: 10.1080/10494820.2010.548068.
Hsiao, Y. P., et al., in press. Designing optimal peer support to alleviate learner cognitive load in Learning Networks.
IADIS International Conference Web-Based Communities and Social Media 2012. Lisbon, Portugal.
King, A., 1994. Guiding Knowledge Construction in the Classroom: Effects of Teaching Children How to Question and
How to Explain. American Educational Research Journal, Vol. 31, No. 2, pp. 338-368.
King, A., 2007. Scripting Collaborative Learning Processes: A Cognitive Perspective. In F. Fischer, I. Kollar, H. Mandl
and J. M. Haake (Eds.), Scripting Computer-Supported Collaborative Learning (pp. 13-37). Springer US, New York,
NY.
King, A., et al., 1998. Mutual peer tutoring: Effects of structuring tutorial interaction to scaffold peer learning. Journal of
Educational Psychology, Vol. 90, No. 1, pp. 134-152.
Nath, L. R. and S. M. Ross, 2001. The influence of a peer-tutoring training model for implementing cooperative
groupings with elementary students. Educational Technology Research and Development, Vol. 49, No. 2, pp. 41-56.
Roscoe, R. D. and M. T. H. Chi, 2008. Tutor learning: the role of explaining and responding to questions. Instructional
Science, Vol. 36, No. 4, pp. 321-350.
Sloep, P. B., 2009. Fostering Sociability in Learning Networks through Ad-Hoc Transient Communities. In M. Purvis
and B. T. R. Savarimuthu (Eds.), Computer-Mediated Social Networking (pp. 62-75). Springer, Heidelberg,
Germany.
Sweller, J., et al., 1998. Cognitive Architecture and Instructional Design. Educational Psychology Review, Vol. 10, No. 3,
pp. 251-296.
Van Rosmalen, P., 2008. Supporting the tutor in the design and support of adaptive e-learning. PhD doctoral dissertation,
Open University of The Netherlands.
268
INTERACTIVE BOOKS FOR PRIMARY AND SECONDARY

EDUCATION FOR THE COURSE OF RELIGION IN
GREECE
Vasiliki Mitropoulou
Aristotle University of Thessaloniki
University Campus, 541 24 Thessaloniki, Greece
ABSTRACT
The Greek Ministry of Education has initiated the project “Digital School” since September 2011. One of the actions of
the Digital School concerns the development of digital interactive books (in html) for all school courses. These
interactive books are enriched with digital components with activities which are embedded in them. The Course
Management System ‘eClass’ was selected as the platform to support the project and to provide the place for
communication and collaboration among the teachers participating in the project. It is supported that teaching with such
interactive books offers some advantages such as offering to the students more opportunities for engaging and interacting
with the books content, multiple representations of the content information, which are expected to result to the better
imprint in the students’ memory. We envisage that this kind of interaction with the content of the books of Religion will
create favorable conditions for the teaching approach of the course of Religion and it will be an initiative for renewal of
its teaching strategies.
KEYWORDS
Interactive book, education, religious education, interactive components
1. INTRODUCTION
The Greek Ministry of Education since September 2011, has initiated the project “Digital School”.
(http://digitalschool.minedu.gov.gr/) One of the actions of the project concerns the design and development
of digital interactive books. The interactive books consist an additional form of the e-books already uploaded
on the e-platform of the greek ministry of education and can contribute to the better exploitation of the
interactive boards in the classrooms. The interactive books are in html format, and on them are embedded
various digital components with activities e.g. Interactive maps, crosswords, quizzes, concept maps,
interactive exercises, power point presentations, links to wikipedia, dictionaries and other similar sources etc.
The digital components appear as small pictures on the interactive book. When the cursor is moved on the
picture, a pop-up menu opens with all the information of the component (title or kind of the activity (e.g.
Concept map). When clicking on it, a window opens with the activity and information about it e.g. Chapter,
lesson, title, guidelines of use etc). (Kapaniaris & Papadimitriou, 2012)
2. E-PLATFORM OF THE ‘DIGITAL SCHOOL’

As place for the actions of the project “Digital School” was selected the Course Management System
‘eClass’ (digitalschool.minedu.gov.gr). It is used as platform where there are uploaded the digital
schoolbooks (already in pdf and gradually in html format) of all courses and it provides a common space for
communication and collaboration of the teachers working on the design and development of the digital
components. The html format of the books will render easier on one hand their enrichment with the digital
components (interactive maps, digital map etc) and thus their change into interactive.
269
ISBN: 978-989-8533-12-8 © 2012 IADIS
Among the aims of this project are: (a) to provide the greek students the possibility to have access to the
digital and interactive schoolbooks from their homes, at their free time, so that they will be able to practice
with the activities at their own pace and time (b) to provide the teachers the possibility to have immediate
access to a great variety of teaching material so as to be able to use it for teaching with interactive whiteboard
but also to enrich and upgrade their teaching approach. (Μitropoulou, 2011)
‘eClass’ Course Management System is used to facilitate the communication and collaboration among the
teachers working on the development of the interactive books. There are 10 teacher-teams working on the
following courses: Modern Greek Language, Physics, English, Informatics, Mathematics, French, Music,
Chemistry, Geography and Religion. The field of the platform used for this purpose is called”
“Synergazomai-Symmetexo” (Collaborate-Participate) (http://synergasia.minedu.gov.gr/) and provides to all
teams a place to communicate, upload and store the components/activities they design and develop. Then the
components are incorporated as links on the html schoolbook, which, finally, is uploaded on the platform of
the Digital school.
3. THE INTERACTIVE E-BOOKS FOR THE COURSE OF RELIGION

All the schoolbooks for Religion both for Primary and Secondary Education are already uploaded on the
eClass platform of the Digital School. Up-to-day, those which have been enriched are the books for
Secondary Education (High School: 1st, 2nd, 3rd grades, and Upper High School [Lykeion]: 1st, 2nd grade). The
team has just finished the enrichment of the book of 6th grade of Primary school and is currently working on
the books of 3rd, 4th, 5th grade.
On the eClass platform are also uploaded the aims and targets of each course per grade and per subject, as
they are published in the national state Curricula (Government Paper, 2003: 303/13.03.2003, v. Β’ &
Government Paper, 1998: 406/05.05.1998, v. Β΄) as well as the Teacher’s Book for each course.
Religion is an obligatory course in the greek educational system taught at both Primary (4 out of 6
classes) and Secondary Education (3 years High School and 3 years upper high school [Lykeion]. Its content
contains information on history, practice, art, on religions and their role as way and expression of life
(Christianity (mainly) and references to other religions e.g. Islam, Judaism, Buddhism, Hinduism etc).
The interactive components/applets developed and embedded on the interactive books of Religion have a
great variety in their formats and interactivity. More specifically, there are: (a) introductory presentations
(presenting the basic points of the lesson (written and orally) (b) interactive maps and pictures (c) photo
galleries, (d) videos, (e) links with Wikipedia, (f) links with a greek online Dictionary, (g) activities for
consolidation, practice and/or feedback (quizzes, crosswords, puzzles), (h) music and songs (i) comics in
books.
For the development of the interactive activities for the course of Religion were used the following
software programs: Articulate Studio 2009, Pro Adobe Flash Pro CS5, Wondershare, QuizCreator 4.1.0,
Αudacity, Colagedit, ispring free, Eclipse Crossword, Hot potatoes, Acrobat Reader, Comic lab.
4. PEDAGOGICAL AND TEACHING APPROACH OF THE

INTERACTIVE E-BOOKS FOR THE COURSE OF RELIGION
The interactive schoolbook can be used in the teaching of the course of Religion with interactive whiteboard.
The teaching of Religion course in schools is delivered till today in the traditional way, teaching in groups,
accompanied with a power-point presentation prepared just for the purposes of the specific lesson.
Occasionally, when the teacher is more familiar with computers, he may use educational software. Usually,
though ICT is used in the teaching, just as visual medium without activating the students, because they still
sit and watch and not participate actively. This is important in the course of Religion because it is a course
based on experiences from life and the above described way of teaching does not exploit the possibilities of
the computer in this direction. Also it does not help the students realize the interdisciplinarity of the course of
Religion with other courses (e.g. History, Geography, Visual Arts). Among the advantages of teaching with
such interactive books is the possibility offered to the student to participate actively during the lesson and
make correlations and creative comparisons with either other lessons or other courses thus supporting the
270
interdisciplinary learning approach (Μitropoulou, 2011). The content uploaded on the interactive books of
Religion is expected to offer multiple representations of the various information, which is envisaged to result
to their better imprint in the students’ memory, as «the information is presented in a more attractive, direct,
pleasant and comprehensible way» through interactive pictures, interactive maps, comics, applets etc.
The introductory presentation created with Articulate software is a presentation with the main points of
each lesson of the book (duration 3-5 min). The student can both see these points in text and listen (or not
listen if he does not want) them at the same time. When possible, the text is accompanied by a relevant
picture. The introductory presentation can be used by the teacher to introduce the lesson to the students
(presenting the new information so that the students will use it as a base to construct their new knowledge) or
for review at the end of the lesson (review or feedback, and consolidation of the information learnt) or before
the next lesson to help the students recall in their memory relevant information and connect them with the
new information that they will be taught, thus facilitating the assimilation and adjustment. The students on
their part can review the main points of the lesson at their own place and time.
The photo galleries present different views of a theme, e.g. photos of Mt Thavor or river Jordan, or icons
of Transfiguration or Buddha. The photo-galleries developed for the interactive books, depending on the
topic, they use photographs, byzantine icons, paintings of famous painters, etc. The students can select the
picture they wish to observe in e.g. a carousel or scattered cards. By clicking on it once they can enlarge it
and notice the details of it, e.g. posture or facial expression, and by clicking once again they can read details
about it, e.g. if it is a famous painting, the students can get information on its title, artist, date, museum where
it is etc. Thus the students can actively practice to observe the paintings and notice e.g. the common elements
or persons that there are on them and their posture e.g. in Transfiguration, Jesus is dressed in white clothes
and is inside a circle of light, on his right and left there are depicted Moses and prophet Elijah, while at the
bottom of the painting there are fallen down his three students Peter, John and Jacob. The students are asked
to observe that e.g. in all the paintings, the students are fallen on the ground, Jesus is in white, they are on a
mountain (Mt Thavor) etc. This activity responds to the special needs of Religion painting, which are
symbolic and therefore not understood. Another component which helps in this direction concerns the
interactive icons. The student, when clicking with the mouse on a person of the icon, can get information
about him/her e.g. who he is, what is the symbolism, why he is there. With the photo-galleries and the
interactive pictures the students can actively see and understand why a theme is depicted in this specific way
and which persons participate in this event (e.g. Baptism), or see the parts and objects inside/outside a sacred
place (mosque, synagogue, hindu temple). This activity has a direct use for the course of Religion as it is
related with its symbols, without which it is not possible to understand e.g. historical or art issues. This kind
of activities help the students practice their observation skills and their critical thought, as they are involved
actively in creative comparisons e.g. compare the religious symbolisms in the byzantine icons with the same
symbolisms in the paintings of famous painters (e.g. El Greco) thus involving in interdisciplinary approach
with the Visual Arts Course.
The words which are links to the online modern greek dictionary (http://www.greek-
language.gr/greekLang/modern_greek/tools/lexica/triantafyllides/index.html) are keywords, useful to help
the students understand the content of the lesson as they also provide the religious meaning of the words, thus
aiming to help the students become familiar with it and understand better the content of the lesson. When the
students try to solve the crosswords they will need to know about the specific meaning a word has/acquires in
the religious context. Because a word or phrase matching exercise like those in a crossword “represent the
new knowledge with words, which –according to Bruner- consist ‘abstract symbol systems’ (symbolic
representation of knowledge)”. (Samara, 2011, Μitropoulou, 2012) Additional information they can get from
the links provided to the greek Wikipedia.
The interactive maps can help the students gain consciousness of the time and place the event occurred
(e.g. Exodus) and be able to realize the geographic location today (in the photo gallery they can even see how
that place looks today e.g. desert of Sina). Also by clicking with the mouse on a world map they can see e.g.
the countries (and their flags) where the death penalty is still applied). This is very important for the course of
Religion, especially when it concerns historical events or issues (e.g. Edict of Milan, world map of religions
today) as the students will realize that the historical events they are taught about are not fictitious and they do
not concern a remote past but they are related with present situation (e.g. Israel). The students with Google
earth can visit the cities e.g. that St. Paul visited during his journeys or the area that St Basil lived on a
present map (from Google maps). It is envisaged that they will connect the time he lived with the present
condition and gain consciousness of the time with a timeline cursor.
271
ISBN: 978-989-8533-12-8 © 2012 IADIS
Finally, the quiz exercises, which are located at the end of the lesson, contain various behavioristic type
exercises (true-false, multiple choice, word completion, correct order, word or phrase matching) that offer
the students the possibility of self-evaluation, because the students can either get immediate feedback of their
answers, or see again the exercises and the correct answers. Because, the closer the feedback and the correct
answer is to the answer given by the student, the better and longer is imprint in the student’s memory. At the
end of the quiz exercises, there is a final percentage of the correct answers given by the students. This is to
provide (a) to the teacher the possibility to check if the students achieved the aim of the lesson and
constructed their knowledge, and (b) to the students the chance to check if their new gained knowledge
responds to their views up-to-day, thus improving their meta-cognitive abilities.
Figure 1. Introductory presentation, interactive picture, photo gallery, interactive map, quiz
The teacher’s role is that of facilitator, guidance to the journey of learning; he provides the students the
suppliers, shows them the final destination (knowledge) but it is the students that will select how to use the
supplies and which way to follow to get there.
5. CONCLUSION
The interactive schoolbook of Religion, enriches their content with multiple ways and offer the students
multiple representation of the various information, resulting to their better imprint on the students’ memory.
Thus, the information is presented in a more attractive, direct, pleasant and understandable way. It is
expected that this interactive presentation of the learning content from the e-schoolbooks presents some
advantages, e.g. provide the students the possibility to participate actively during the teaching approach and
make correlations and comparisons with the content of other lessons of the same or other course thus,
supporting the interdisciplinary learning approach. We envisage that this kind of presenting the content of
Religion will create favorable conditions for the teaching approach of the course of Religion and it will be an
initiative for renewal of its teaching strategies (Μitropoulou, 2008, 2011). We are planning a research in the
secondary education schools of Greece to get feedback of the views of the teachers and the students when
these interactive books are used during the teaching of the Religion course, the results of which will be
presented at another conference next year.
ACKNOWLEDGEMENT
The members of the team for the development of the interactive e-schoolbooks in the course of religion are:
Nikolaos Argyropoulos, nickargiropoulos@gmail.com, Vasileios Vasileiou, billbas@sch.gr, Alexandros
Gkikas, alexgkikas@gmail.com, Alexandros Kapaniaris, kapaniaris@yahoo.gr, Alexandra Kolovou,
alexkol@sch.gr, Ioanna Komninou, kilm@otenet.gr, Nikolaos Marinopoulos,
n_marinopoulos@hotmail.com, Nikolaos Pavlou, nipavlou@otenet.gr, Georgios Papanikolaou,
elfinor3@yahoo.gr, Kostantinos Prentos, kprentos@sch.gr, Kosnstantina Tsitouridou, ktsitour@gmail.com.
272
REFERENCES
Government Paper, 2003. No 303/13.03.2003, v. Β’
Government Paper, 1998. No 406/05.05.1998, v. Β
Κapaniaris, Α., & Papadimitriou, Ε., 2012. Digital literacy in the new digital school. Ziti, Thessaloniki (in press)
Μitropoulou, V., 2012. Educational scenario with the use of ICT in the teaching of the course of Religion. Contributed
volume to honorary professor C. Vasilopoulos “Education in love and freedom”, Vanias, Thessaloniki pp. 197-220
Μitropoulou, V., 2011. The course of Religion in the Digital School. Ecclissiasticos Kyrikas, Larnaca, Cyprus, v. 17 pp.
71-84
Μitropoulou, V., 2008. Educational Software in the teaching of the course of Religion. Vanias, Thessaloniki
Mpampiniotis, G. 2002. Modern New Greek Dictionary, Leksikology Centre, Athens
Samara S. 2011. Teaching scenario with the integration of ICT and the Interactive White Board: Implementation in the
course of Social and Political Education. Proceedings of 2nd Educational Procedure of Introduction and Use of ICTs
in Education Approach. Patras, Greece pp. 739-750
Digital School, 2012. Ministry of Education, Athens. Accessed April 27, 2012 from: http://digitalschool.minedu.gov.gr/
Digital School – Synergasia, 2012. Ministry of Education, Athens. Accessed April 27, 2012 from:
http://synergasia.minedu.gov.gr/
273
ISBN: 978-989-8533-12-8 © 2012 IADIS
SEARCHING FOR BRIDGES BETWEEN FORMAL AND

INFORMAL LANGUAGE EDUCATION
Pavel Brebera and Jitka Hlouskova

University of Pardubice
Studentska 95
532 10 Pardubice
Czech Republic
ABSTRACT
Life in the contemporary society and ongoing globalisation processes result in growing demands on educators, including
language teachers in higher education. The frequently accentuated so-called postmethod approach to foreign language
teaching gives teachers a lot of freedom and flexibility but also creates a large space for various types of challenges and
dilemmas. These are related to issues ranging from the inevitability to perceive teaching in a broader educational context
to the necessity to internalise the lifelong learning concept, bringing along the categories of formal, non-formal and
informal learning. Besides, there is no doubt that language learning/teaching processes are to a great extent facilitated by
the rapid development of ICT. This paper is aimed at explaining the authors’ beliefs related to the role of the teacher in
curriculum development and the level of his/her control over the content of learning within the above specified context.
The central part of the paper relies on examples illustrating the teaching and research activities carried out at the
Language Centre of the University of Pardubice, Czech Republic, that are powered by the efforts to use elements of e-
learning for bridging the gaps between formal and informal language education.
KEYWORDS
Higher education, foreign language teaching, lifelong learning, e-learning, formal learning, informal learning
1. INTRODUCTION
The demands life in the contemporary society makes on humans are unparalleled from the point of view of
the history of mankind. They are a consequence of rapid development of modern technologies, primarily
information and communication, the increasing pace of globalisation tendencies and unprecedented migration
and travel worldwide for work or other reasons. For most people, this results in the necessity of learning at
least one foreign language. Simultaneously, the requirements on foreign language teaching in higher
education throughout Europe reflect the principal goal of the European education policy, namely preparation
of future experts and specialists for effective functioning on the job market. Therefore, the efforts of higher
education institutions are being driven by a well-justified plan to optimise conditions for the desirable shift
“from expert students to novice professionals” (Reid et al. 2011) also in the area of complex language
development.
As the language that plays the role of modern lingua franca is English, top quality and efficiency are
demanded in the area of English language teaching (ELT). Language teaching specialists can draw on
numerous positives in current approaches to methodology in terms of what some authors call the postmethod
stage (e.g. Kumaravadivelu 2008) which is characterised by a great extent of freedom and flexibility. There is
no single generally accepted or prescribed “right” method, and teachers are encouraged to be inspired by
whatever traditional or modern approaches in case they consider them appropriate, useful and efficient. This
type of freedom and flexibility presupposes a high level of teacher cognition, confidence and autonomy,
reflected experience and good decision-making skills. At the same time, it also requires a high degree of
teacher awareness of a broader educational context and of the importance of the concept of lifelong learning.
Establishing effective links between formal, non-formal and informal learning within the framework of
life-long learning represents one of the biggest challenges at all levels of educational systems. It is mainly
274
due to the fact that institutionalised formal learning, non-formal learning occurring in formal learning
environments but not formally recognised (e.g. workshops, interest based courses, conferences) and informal
learning of rather incidental or random nature (e.g. CEDEFOP 2001) are perceived as complementary but at
the same time, they bring about very different curricular implications. For example, Smith (2000) contrasts
all of the above mentioned categories in order to present the notion of curriculum as the boundary between
formal and informal education, thus initially questioning the relevance of curricular theory and practice
within the area of informal education but finally concluding about the necessity to conceptualise even the
area of informal learning in terms of the language of curriculum (e.g. informal programmes or projects
instead of formal courses at school). The importance of searching for “bridges” between formal, non-formal
and informal learning within the particular educational contexts is also obvious from the perspective of
addressing all domains of general aims of education, since according to Rabusicova (2008) formal learning
has the highest potential in the area of cognitive development, non-formal learning in the area of the
development of skills, and informal learning in the affective domain. Translated into the pedagogical
discourse of language education at the tertiary level, this triad might be interpreted in terms of the
complementarity of formal language courses at school, opportunities for developing language skills non-
formally in an authentic or simulated working context and informal opportunities to practice a foreign
language in a non-threatening context, thus building positive attitudes towards the language as a means of
communication.
Besides, one should not neglect the obvious fact that the processes of learning foreign languages are
nowadays to a large extent facilitated by means of ICT. The use of ICT for language teaching/learning
purposes undoubtedly represents a rapidly developing area, yet it might be still useful to categorise the
teacher roles according to a traditional criterion of the degree of control over the content of learning. At one
end of the continuum, we may observe teachers with a high degree of control over the content of learning
within formal Moodle courses, while at its other end teachers deliberately resign on their controlling role and
draw on the potential of informal learning within widely used social networks, such as for example
Facebook. However, searching for effective bridges between formal, non-formal and informal language
learning in the area of tertiary education presupposes overcoming the above mentioned unilateral views and
implies the need for a more balanced approach taking into the consideration the complexity of language
development (cognition, skills and attitudes) and the need for individual learning paths.
2. BRIDGING THE GAPS BETWEEN FORMAL AND INFORMAL

LANGUAGE LEARNING
2.1 Specifying the Context

With regard to the above mentioned issues, we will present several examples of efforts carried out at the
Language Centre (LC) of the University of Pardubice, Czech Republic. Many of the activities the LC staff
concentrate on are related to EU-funded projects. One of them, ELPiPL or European Language Portfolio on
Student’s Journey through Studies into Professional Life, was an international project involving secondary
and tertiary education institutions from six EU countries and running from 2009 till 2011. It was aimed
primarily at developing teaching methods and materials in cooperation with the world of work, enhancing
learner autonomy and promoting plurilingualism and lifelong learning. Another project, UNICOM –
Innovative and Integrated Concept of Language Training towards Quality Assurance, Excellence and
Internationalisation, has been running since January 2012 and will be completed by the end of 2014. Its
outcomes are expected not only to enhance the quality of learning primarily in the area of Languages for
Specific or Academic Purposes (LSP, LAP) in the formal higher-education setting but also to motivate the
students to increase the share of informal learning in their own education, emphasising the benefits it can
yield for life-long learning.
275
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.2 Examples of Bridges between Formal and Informal Learning in Higher

Education
2.2.1 ELPiPL Blogging
Using portfolio assignments generally represents a significant innovation of language teaching courses,
mainly in terms of the development of learner autonomy. Within the ELPiPL project, this aim was pursued
not only by means of the tasks from the printed portfolio brochures but also with the help of the social
networking platform NING. For this particular purpose, the NING platform was available only for authorised
users after an invitation by the NING administrator of the project. The blogging part of the ELPiPL project in
the Czech educational context proved that even lower-level students (A2) were able to formulate successfully
blog posts mainly of a reflective nature (e.g. personal stories “How I started learning foreign languages”,
“Why I find learning foreign languages important”) and that they demonstrated a high-level of peer support
by means of commenting positively on the accomplishments of the others, thus building positive attitudes
towards the use of social networking for one’s own language learning purposes.
2.2.2 Progress Test
An example of an effective bridge between formal and informal learning is Progress Test administered as
described below. Following this particular procedure, the test had a dual function: (a) an e-learning/testing
tool in the formal setting of an LSP course in higher education, b) a tool supporting learning from home or
another informal setting.
This test was used with a total of 48 students of English for IT. They had to take the test twice within four
months. The test was always active for a specified period of time and the students could choose when and
where to take it. They were asked to write a short comment on their results, including a note on the difference
in their two scores, and encouraged to include other comments. The purpose of this assignment was twofold.
First of all, the test was intended to provide the students with feedback on their progress in English; this
information was also important for the teacher. Secondly, the test results and student comments provided the
teacher with data for content analysis focused on finding answers to the following questions: (1) How many
students made progress? (2) What reasons do the students provide for their progress/decline? (3) Can an
assignment of this type have an influence on the students’ self-assessment and motivation?
As this paper does not provide enough space for a detailed data analysis, let us summarise the most
important results. This preliminary research will be further extended and the full results will be published
later. The answer to question (1) is displayed in figure 1 below.

Figure 1. Progress Test – improvement
A coding technique was used for the analysis of the students’ open comments, which provided data of
qualitative character. The most relevant answers to question (2) can be summarised as follows: The most
frequented reasons the students gave for their improvement was that Test 02 had seemed easier to them.
Some said they had been lucky, a few claimed they had been studying hard. Several students gave watching
films in English as a reason for their improvement, a small number said they had had better class attendance.
Although the students were not asked to comment specifically on their learning processes or motivation,
the analysis of their open comments yielded a definitively positive answer to question (3). Several students
said that having seen their scores, they wanted to improve their English. Some of them expressed
(dis)satisfaction with their results, which showed the tests and the assignment had given them an opportunity
for self-assessment and self-reflection.
276
2.2.3 Internet Forums

Another example of a bridge between formal and informal learning is a web-based assignment. The students
of a B1 course of English for IT were assigned the following task: Visit an internet forum on (an) operating
system(s) and write replies to two different posts. The teacher found inspiration for this type of assignment at
an interview she had carried out with an expert working in the IT business within her research incorporating
the analysis of the communicative needs in English for future graduates of the IT programme. Among other
tips, the interviewee advised to visit internet forums on specific topics to get acquainted with the language IT
experts use to communicate about issues related to their profession. In this context, internet forums can be
considered informal learning settings as they represent platforms on which not only experts from practice, but
also students exchange their expertise, experience and ideas in real-life communication outside the education
system. Apart from this, the students’ posts contributed to the bank of the most frequented errors that could
be analysed and worked with in classes.
2.2.4 Online Games as an Extension of Formal Language Training
Undoubtedly, the internet represents an enormous space for spontaneous formation of professionally-related
groups (see above the IT forums) but also of the communities whose members share the ultimate goal of
“mere” game-playing. As one of the components of formal language courses is training in presentation skills
in order to prepare the future professionals for the task of giving well-targeted and persuasive work-related
presentations, one of the model performances was given also by the teacher. This model presentation was
carried out with dual aims: demonstrating particular language and non-linguistic means for giving a business
presentation, and at the same time inviting the students to join an online game where they would practise
English with their teacher after the end of the formal language course and within the specific community of
players. In 2012, the unique context of the European football championship was taken advantage of, and the
students were invited to play the so called fantasy football together with their teacher.
3. CONCLUSION
The assignment and activities presented above support learning theory suggesting that learning is promoted
or enhanced when students are actively involved in the learning, when assignments reflect real-life contexts
and experiences and when critical thinking or deep learning is promoted through applied and reflective
activities (Smart and Cappel 2006). From this perspective, it is definitely worth to search for and analyse
potential bridges between formal and informal language learning systematically and on a long-term basis.
REFERENCES
European Centre for the Development of Vocational Training. 2011. Cedefop Glossary. [online] Available from:
http://www.cedefop.europa.eu/EN/about-cedefop/projects/validation-of-non-formal-and-informal-learning/european-
inventory-glossary.aspx. [accessed 21/05/2012].
Kumaravadivelu, B. 2006. Understanding Language Teaching: From Method to Postmethod. Lawrence Erlbaum
Associates, Mahwah, NJ, USA.
Rabusicova, M. Vzdelavani dospelych: ocekavani a efekty. [online]. In Pedagogicka evaluace 08 a Socialia 2008.
Keynote speech. 2008. Available from: http://www.phil.muni.cz/wupv/home/Documents/mpsv-projekt-
prezentace/prezentace-malenovice.ppt. [accessed 21/05/2012].
Reid, A. et al. 2011. From Expert Student to Novice Professional. Springer Science+Business Media, New York, USA.
Smart, K. L. and Cappel, J. J. Students‘ Perceptions of Online Learning: A Comparative Study [online]. In Journal of
Information Technology Education, 2006, vol. 5. ISSN: 1539-3585. Available from:
http://jite.org/documents/Vol5/v5p201-219Smart54.pdf. [accessed 21/05/2012].
Smith, M. K. Curriculum Theory and Practice [online]. In The Encyclopaedia of Informal Education. 2000. Available
from: http://www.infed.org/biblio/b-curric.htm. [accessed 21/05/2012].
277
ISBN: 978-989-8533-12-8 © 2012 IADIS
EXPLORING SPACES FOR LEARNING: USING

NARRATIVE MEDIATION PATH TO IMPROVE THE
ACADEMIC PERFORMANCE OF UNDERACHIEVING
UNDERGRADUATE STUDENTS
Dr Una Crowley and Dr Catherine Mahon

National University of Ireland Maynooth
ABSTRACT
Learning to learn has been identified as a key educational competence. Over the next two years, as part of the INSTALL1
project, NUI Maynooth is testing the effectiveness of an exploratory group technique, the Narrative Mediation Path
(NMP), which has been developed to promote reflective thinking skills. To date, interviews have been conducted with
200 first year students. Common themes emerging from these interviews regarding student engagement are discussed.
Notable was the number of concerns relating to coping with academic demands. From these 200 students, 20 students
will participate in the subsequent phases of the INSTALL project utilising the NMP.
KEYWORDS
Learning to Learn, Underachievement, Higher Education
1. INTRODUCTION
The EU2020 strategic objective of a more cohesive growth through knowledge based economy calls for
improved models to sustain the acquisition of the key competence, learning to learn, for disadvantaged and
underachieving Higher Education students. The National University of Ireland Maynooth (NUI Maynooth) in
collaboration with its European partners, INSTALL, is responding to this challenge by exploring innovative
solutions to address students’ constraints to acquiring and developing the learning to learn competence.
1.1 Learning to learn

The transition from second level instruction to university education involves students adapting to arguably a
different type of learning. Traditionally within the university domain greater emphasis has been placed on
self-initiated study. Consequently, for many students the challenge becomes one of learning how to learn.
The term learning to learn refers not to a unitary skill, but rather a range of skills identified as
contributing to optimal learning, including cognitive skills, motivation, organisation of time, and
self-regulation (see Amalathas, 2010, for further details). Within the literature, there is a particular focus on
the important role played by metacognitive skills in learning to learn. Findings have shown that university
students who score highly on measures of metacognitive skills tend to perform better on academic tasks (e.g.,
Van der Stel & Veenman, 2010). Reflection is one of the core metacognitive skills (Lew & Schmidt, 2011).
In terms of the INSTALL project, an exploratory procedure known as the Narrative Mediation Path
(NMP) has been developed to support the ability of students to reflect on their university experiences.
1
INSTALL, Innovative Solutions to Acquire Learning to Learn, is a European funded project (Erasmus Multilateral Projects, nº
517750-LLP-1-IT-ERASMUS-ESIN). Partners involved are: University of Naples, Italy, National School of Political and
Administrative Studies (NSPAS), Romania; University of Aarhus, Denmark; NUI Maynooth, Ireland and University of Seville, Spain.
Please note that this project has been funded with support from the European Commission. This paper reflects the views only of the
authors and the Commission cannot be held responsible for any use which may be made of the information contained therein.
278
Specifically, this tool was designed for use with students following semesters during which they have
struggled with academic activities. The NMP is a group narrative tool. Therefore, the project builds on the
more recent research in this area which has investigated the promotion of metacognitive skills through social
interactions with peers (e.g., De Backer, Van Keer & Valcke, 2012).
The current paper focuses on the research undertaken with students enrolled at NUI Maynooth and also
discusses the research that will be conducted throughout the coming months as part of the INSTALL project.
1.2 Background
NUI Maynooth has expanded rapidly with an increase of 21% between 2004/05 and 2011/12 in the total
number of students, to become one of the fastest growing universities in Ireland. Approximately one quarter
of the students come from non-traditional backgrounds. In addition, NUI Maynooth attracts more students
whose parents did not go to Higher Education than other universities (Irish University Study, 2009).
1.3 Statement of the Problem

Twenty-first century university students are expected to be flexible, digitally competent, self-regulated,
learners. To succeed, students must be ready to respond quickly to ever-changing intellectual and
technological environments. Evidence would suggest, however, that students entering university directly
from second level are ill-prepared, making the transition without the generic skills needed to cope at that
level (Hyland, 2011: 16). Professor Tom Collins chairman of the NCCA voiced his criticism as follows:
There is palpable concern in Higher Education regarding the capabilities and dispositions of
students entering [university] straight from second level. The manner in which the points
system rewards rote learning, instrumental learning and memorisation while simultaneously
discouraging exploration, self-directed learning and critical thinking means that even relatively
high achieving second-level students can struggle on entering third-level’ (Collins: 2010).
The selection of school leavers for third level education in Ireland is founded on a points system based on
an applicant’s results in six subjects in the Leaving Certificate examination. The Leaving Certificate is a high
stake examination, and because of this, the backwash effect on teaching, learning and the student experience
is considerable (ibid: 4). The race to optimise points greatly influences the basic level of knowledge,
cognitive skills and learning habits of students (Hyland 2011: 4). Concerns have been raised by policy
makers and academics that the Leaving Certificate rewards rote-learning rather than problem-solving, critical
thinking, or self-directed learning (ibid: 16). Such concerns about students not being adequately prepared for
Higher Education are not unique to Ireland. Similar concerns have been voiced in the US, UK and in other
EU countries (Hyland 2011, Gogus and Arikan, 2008: 279). Of course, there are also many other factors
which influence student underachievement and non-progression, including social, cultural, financial, and
institutional factors. The problem seems to be that most universities have not yet been able to translate what
[they] know about student lack of ‘preparedness’ ‘into forms of action that have led to substantial gains in
student persistence and graduation’ (Tinto, 2006: 5).
As part of the INSTALL project, NUI Maynooth is responding to this challenge by developing a tool, the
NMP, to help students acquire skills related to learning to learn.
2. METHODOLOGY
2.1 Phase One: Interviews with Students

Between September 2011 to May 2012, 200 face-to-face interviews took place with first year students who
visited the Academic Advisory Office (AAO) in NUI Maynooth. For each student, a written record was kept
to detail the reason for the visit. Analysis of the written records was undertaken by an independent coder to
identify key themes to emerge from the interviews.
279
ISBN: 978-989-8533-12-8 © 2012 IADIS
2.2 Phase Two: Implementation of the Narrative Mediation Path

Participation in the remaining two phases of the project requires fulfillment of a number of set criteria as
shown in Table 1. All students will be briefed about the project and informed consent will be obtained.
Table 1. Student selection criteria.
Category A Category B Category C
Academic Criteria Social and Economic Criteria Preferential Criteria
• Passed at least one • Ethnic minority. • Achieved one high
module by • Disability. grade in the first or
compensation2. • In receipt of a Local second semester of first
• Failed at least one Authority Grant. year of undergraduate
exam in January or • First generation degree programme.
May and repeated it in university student. • Scored at least 375
August (and then points in the Leaving
passed). Certificate.
• Self reporting – may
pass exams but do not
have a score above
50%.
Note. Students must satisfy at least one of the criteria from Category A or Category B.
There will be two cycles of the NMP programme. Initially, the first sample of 20 students will commence
the programme in October 2012. A second sample of 20 students will begin in February 2013. In the
intervening period between the two cycles, refinements will be made to the programme. Prior to the start of
the programme, students will answer open-ended questions related to their university experiences to obtain a
preliminary measure of their entry reflective skills. Subsequently, the written responses will be evaluated
according to the Reflective Functioning Scale devised by Fonagy, Target, Steele and Steele (1998).
The NMP programme will be presented to students across six meetings. Each meeting is guided by a
trained instructor. There are four modules to be covered as follows: (i) metaphorical; (ii) iconographic; (iii)
writing; (iv) bodily. A description of the contents of these modules can be found in Freda, Esposito, Martino
and Monteagudo (2012). The modules are completed in a set order, with the sequence of the four modules
arranged to encourage the cognitive, social, and emotional involvement of all the students. It is proposed that
such an involvement is made possible by using narrative stimuli (e.g., vignettes, written accounts) and
procedures designed to promote and develop reflective thinking skills.
To assess whether there has been any change in reflective skills, students will again provide written
responses to a set of open-ended questions. Responses will be scored using the Reflective Functioning Scale.
2.3 Phase Three: Tracking Academic Progress

Six months following completion of training, the academic performance of each student will be monitored by
examining the grades obtained by students for their chosen second year degree modules.
3. PHASE ONE RESULTS

The 200 written reports were reviewed to identify the main reason for the student’s visit. Fourteen categories
were established as summarised in Table 2.
2
Students may pass by compensation by passing two of their subjects, obtaining at least 35% in their third subject, and obtaining at least
40% on aggregate, provided that the two passed subjects are compatible in second year. Such students may only continue in second year
with the subjects in which he or she has obtained at least 40%, hence students cannot pass by compensation in a subject that is a
requirement in a second year programme.
280
Table 2. Categories employed to code the reports of the interviews conducted by the Academic Advisory Office.
Category Brief description
Onwards referral Student was recommended to contact the relevant
university administrative department (e.g., Admissions)
or academic department (e.g., Psychology), given the
specific nature of the query.
Dislike of course Student explicitly states a lack of enjoyment for the
chosen degree subjects.
Failure to meet course requirements A student who may not have submitted an essay or
assignment; who has not attended lectures or tutorials;
who has not achieved the required mark in a module.
Mental Health Issue Any mental health issue (e.g., depression, anxiety),
often accompanied by medical certification.
Physical Health Issue Any physical health issue (e.g., illness, treatment,
surgery, injury) often accompanied by medical
certification.
Financial Issues with grants, funding, debt.
Family Bereavement, family illness, unemployment, childcare.
Commuting Living too far away from the university which is
affecting attendance, study.
Finding course difficult Student explicitly states that the course is challenging.
Problems with studying Difficulties with writing essays, knowing what material
to study, compiling notes in lectures.
Time management Difficulties in planning revision, study time, reading
time.
Desire to change course Student expresses a wish to study a different subject.
Transfer to another university Student is considering leaving to pursue a programme
in a different third level institution.
Other Any other reasons which cannot be coded.
Figure 1 shows the number of students attending the AAO across the 14 categories. The most common
reason for students to contact the AAO was to change their course. However, approximately 35% of the 200
students recounted problems explicitly linked to their chosen courses. Twenty-six students were concerned
about a failure to meet course requirements. Other students needed assistance because they did not like their
course (22 students) or were finding the course difficult (17 students).
30
Number of students (n = 200)
25
20
15
10
0
er
l
fer
l
se
se
ily
lth
lth
nts
t
y
s
ra
cia
en
le m
u lt
tin
ot h
fam
ea
ea
fer
ur
ur
ns
me
em
an
fic
u
co
co
lh
ob
lh
mm
tra
re
fin
dif
ir e
ag
nta
ic a
e
pr
ge
lik
co
qu
an
se
an
dy
ys
me
d is
re
ur
em
stu
ch
ph
co
se
t im
ur
co
Figure 1. Reasons for first year students attending the Academic Advisory Office.
281
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. CONCLUSION
Student disengagement and academic underperformance is complex, although it is generally recognised as a
path that can be traced, with problems that can be resolved through enhanced student support (Thomas 2005:
63, Mann, 2008). Student support services, such as the AAO at NUI Maynooth, can help to directly enhance
academic supports for at risk students. Indeed, the initial phase of this research has helped to shed further
insight into some of the most widespread difficulties experienced by first year students attending NUI
Maynooth. A visible trend was that students expressed concerns specifically linked to their academic
performance, for instance, about not fulfilling course requirements or finding the course difficult. Our goal
now is to investigate potential ways to support these students. Helping students to acquire key skills such as
the learning to learn competence offers one such possibility. Through participation in the INSTALL project,
the aim is to test the effectiveness of the NMP in developing reflective skills. The ability to constructively
think back on past experiences and to recognise the potential for change can all be categorised as reflective
skills; skills that may be highly valuable for students attempting to successfully complete university.
ACKNOWLEDGEMENTS
This research is funded by the European Commission. We thank Nuala Murphy for help with data analysis.
REFERENCES
Amalathas, E. (2010). Learning to Learn in Further Education: A literature review of effective practice in England and
abroad. CFBT: London.
Collins, T. (2010). We’re missing the point on how to prepare for college. Retrieved August 2010 from
http://www.careerguidance.gov.ie
Conole, G. (2010). A holistic approach to designing for learning: A vision for the future. In Annual International CODE
Symposium, 18 February 2010, Chiba, Japan.
De Backer, L., Van Keer, H., & Valcke, M. (2012). Exploring the potential impact of reciprocal peer tutoring on higher
education students’ metacognitive knowledge and regulation. Instructional Science, 40, 559-588.
Fleming, T., & Finnegan, F. (2011). Cultural and Institutional Perspectives on Access and Retention in three Irish
Higher Education Institute. Access and Retention: Experiences of Non-Traditional Learners in Higher Education
(RANLHE). http://www.ranlhe.dsw.edu.pl/
Fonagy, P., Target, M., Steele, H., & Steele, M. (1998). Reflective-functioning manual version 5.0 for application to
adult attachment interviews. London: University College London.
Freda, M., Esposito, G., Martino, M. L., & Monteagudo, J. (2012). Narrative learning for disadvantaged students: a
model for intervention in HE. ESREA Conference / Life History and Biography Network / Odense, 1-4 March 2012
Gogus, A. & Arikan, H. (2008). Learning to Learn in Higher Education. IADIS International Conference and
Exploratory Learning in a Digital Age. (CELDA 2008).
HEA. (2006). Key Facts and Figures. Dublin: HEA.
HEA. (2010). A Study of Progression in Higher Education. A Report by the Higher Education Authority. Dublin: HEA.
HEA (2011). Sustainability Study: Aligning Participation, Quality and Funding in Irish Higher Education. Dublin: HEA.
Hyland, A. (2011). Entry to Higher Education in Ireland in the 21st Century. Discussion paper for the NCCA/HEA
Seminar. HEA: Dublin
Lew, D. N. M., & Schmidt, H. G. (2011). Writing to learn: Can reflection journals be used to promote self-reflection and
learning? Higher Education Research and Development, 30, 4, 519-532.
Lucas, B. & Greany, T. (2000). Learning to learn: setting the agenda for schools in the 21st century. Network Education
Press.
Mann, S. (2008). Study, Power and the University. Open University Press: Berkshire.
OECD (2010). Education at a Glance 2010: OECD Indicators. Paris: OECD.
Thomas, L., & Quinn, J. (2007). First Generation entry into Higher Education: An international study. Open University
Press: Berkshire.
Tinto, V. (2006). Research and practice into student retention. Journal of College Student Retention, 8, 1, 1-19.
Van der Stel, M., & Veenman, M. (2010). The development of metacognitive skilfulness: A longitudinal study. Learning
and Individual Differences, 20, 220-224.
282
DEVELOPING LEARNING MATERIALS USING AN

ONTOLOGY OF MATHEMATICAL LOGIC
Russell Boyatt and Mike Joy

Department of Computer Science, University of Warwick
Coventry, UK
ABSTRACT
Ontologies describe a body of knowledge and give formal structure to a domain by describing concepts and their
relationships. The construction of an ontology provides an opportunity to develop a shared understanding and a consistent
vocabulary to be used for a given activity. This paper describes the construction of an ontology for an area of
mathematical logic as part of an EU Lifelong Learning project and on-going work exploring how this ontology has been
used to inform the development of learning resources and appropriate structuring mechanisms. Finally, the paper
discusses future work developing enhancements to a virtual learning environment to allow better integration between
domain ontologies and on-line courses.
KEYWORDS
Ontology, mathematical logic, learning materials, Moodle
1. INTRODUCTION
The on-line delivery of learning materials is rapidly becoming an important tool in school and university
education programmes. Virtual learning environments afford new opportunities in course structures and
delivery formats. Pedagogical practice is undergoing a transition as new delivery mechanisms are integrated
into existing course structures. These learning technologies provide interesting opportunities to develop new
strategies for the delivery of learning materials to students. Many universities and schools now use virtual
learning environments (VLEs) such as Moodle (Dougiamas and Taylor, 2003) and Blackboard (Bradford et.
al, 2007) as a mechanism to collect together learning activities. However, the adaptive functionality in these
environments can only be exploited if sufficient information is known about domain to develop suitably
interesting learning paths.
In an era of increasing school and university class sizes, identifying learning paths for individual students
can be challenging due to time and resource constraints. The authors are part of the MALog project, an EU-
funded project that is working to develop open learning resources to improve mathematical logic knowledge
and skills. Mathematical logic is an umbrella term covering a branch of mathematics that includes topics
such as set theory, propositional logic, predicate logic and proof. These topics are seen as fundamental to
developing knowledge and skills used in many scientific, mathematical and engineering disciplines. Yet
developing resources suitable for delivery in the wide variety of learning environments and educational
systems found throughout Europe presents some difficulties. The problems include identifying key concepts
and their relationships, identifying the specific skills required, and recognising the differences in terminology
and educational approaches. Collecting and recording this information in a useful format is a first step to
developing better learning resources.
283
ISBN: 978-989-8533-12-8 © 2012 IADIS
Figure 1. Ontology classes representing set theory topics.

Building an ontology of a knowledge domain allows a shared representation of concepts and their
relationships to be explored. When the knowledge domain (in this case mathematical logic) is described with
an appropriate level of detail, the information can be useful enough to inform the structure and content of
appropriate learning resources. Furthermore, the information gathered can be used to develop learning paths
suitable for students with a range of abilities and educational backgrounds. The approach taken here has
been to develop an ontology of mathematical logic which identifies key mathematical concepts, mathematical
topic areas and prerequisite information.
The aim of this project is to use the information gathered in the ontology of mathematical logic to develop
a suitable course structure. Assessing existing knowledge and providing suitable remedial support is vital to
successful learning (Fensham, 1972). An individual learning path will be appropriate to the needs of the
student by accounting for their individual learning style and rate of progress. By identifying problematic
concepts, the adaptive functionality found in many VLEs can be exploited to produce learning paths where
material is automatically recommended to students based on their progress through learning materials. This
paper describes the construction of an ontology in the area of mathematical logic and describes on-going
work that uses this ontology to develop and structure educational resources appropriate for a range of
students.
2. ONTOLOGIES AND LEARNING

An ontology provides a mechanism to formally represent a body of knowledge. One of the earliest
definitions of ontologies (as used in information science) is the “specification of a conceptualization”
(Gruber, 1995). Ontologies have since developed into an important technology of the Semantic Web
(Berners-Lee, 2001), adding machine recognisable meta-data to information available on the World Wide
Web and providing new opportunities in data processing, sharing and reasoning. Technologies such as RDF
and the Web Ontology Language (OWL) have supported the development of the Semantic Web and
emphasised the importance of the ontology concept. Outside information science, the medical science
community have developed numerous examples of ontologies covering human, plant and animal biology
demonstrating how large vocabularies can be successfully developed, used and further explored.
In a learning context, an ontology can be used to formally describe a set of concepts (or topics) and the
relationships between these concepts. A domain ontology can be used to describe an agreed organisation of
concepts and a shared vocabulary which can be used to ensure consistent use of technical terms. An
ontology can provide students with a mediating artefact to aid understanding of the structure and vocabulary
of a domain. Henze et. al. (2004) demonstrate how ontologies can be used to personalise e-Learning material
and recent work by Laroussi (2012) describes how ontologies can be used to model user behaviour in
adaptive learning environments. The MALog project has been developing an ontology to provide a
description of the topics in the mathematical sub-field of mathematical logic. The ontology has been
constructed from a detailed survey of mathematical logic textbooks, Internet resources and other material to
identify specific concepts and topics. Key words and phrases have been identified and organised into
categories, first more general and then categories with increasing specificity. High level categories became
entities within the ontology class hierarchy (e.g. 'set theory', 'axiomatic set theory') and specific concepts
284
within those areas have been created as individuals within the ontology and classified under one or more
ontology classes. Figure 1 shows an example of this hierarchy. As part of the process, information about the
relationships between specific concepts has also been recorded within the ontology. Related topics are
linked, similar concepts from different sub-fields are identified and mathematical symbols are connected to
concepts. Production of the ontology is the result of a negotiation of meaning of the domain of mathematical
logic. Further information on the strategy used for developing the ontology is described in Boyatt and Joy
(2010).
3. ONTOLOGIES AND LEARNING MATERIAL DEVELOPMENT

The construction of the ontology is motivated by the desire to identify the mathematical logic topics and
relationships between these topics suitable for inclusion in a set of related learning materials. The ontology is
used as the semantic foundation giving an initial structure to the development of learning materials. The
initial version of the ontology identified over 500 concepts (as ontology individuals) and 60 categories and
sub-categories (recorded as ontology classes). The ontology information was then used to develop units of
material on mathematical logic topics. The categories provided learning material authors with initial
suggestions for how to structure units of material. The concepts (individuals) within each category (class)
immediately provides a list of concepts to be included within a unit of learning material. For example, the
‘NaiveSetTheory’ class included concepts such as ‘Set_Union’, ‘Set_Intersection and ‘Set_Membership’
suggesting a unit of material covering naïve set theory. For many topics it was easy to identify specific
concepts but for some topics this process exposed differences in educational systems for how concepts
should be organised and related. Explicitly enumerating mathematical concepts in the ontology also ensured
concepts were not missed or duplicated inappropriately in the learning materials. Additional information
stored such as related concepts and mathematical symbols could also be accessed by the material authors.
The MALog project is a collaborative project between several European countries requiring the
development of material in five different languages. Ensuring consistent use of mathematical terminology
has been aided considerably by the shared vocabulary and representation of knowledge stored in the
ontology. The shared artefact has ensured that learning material has been developed with consistent use of
terminology. Consistent use of terms has reduced the possibility of learners becoming confused with similar
mathematical concepts. Recording the reuse of terms has also allowed us to provide navigation aids to
learners by allowing them to identify all instances of specific concepts, e.g. allowing a learner to find all
instances of the concept 'Cardinality' within the material. This mechanism has been provided by developing
software to read the OWL ontology file and display information within Moodle listing concepts related to the
current activity.
Other mathematical ontologies such as GeoSkills ontology developed by the I2Geo project identified
specific competencies that learners can hold (Libbrecht et. al, 2008). These competencies connect together
different topics to represent a particular understanding that should be held by the learner. This approach has
been adopted for the mathematical logic ontology identifying competencies for three categories of learner:
high-school students, university undergraduate students and industry workers. Over 100 competencies have
been identified for each of these class of learners and are represented in the ontology in the 'Comptency'
class. For example, the 'Create_truth_table_for_NAND' relates the concepts (ontology individuals) for
'AND' and 'NOT'. For the learner material author, the competencies provide a structure to units of learning
materials and help build a basic structure of conceptual prerequisites. We are developing software that reads
the entities from the OWL ontology file (using the Apache Jena API) and creates activities within Moodle
that relate to each competency. Activities are ordered appropriately within the environment (using the
limited adaptive functionality available) to ensure that students encounter material about 'AND' and 'NOT'
before learning specifically about 'NAND'. The student using the ontology information can use the
information to identify their attainment level and the competencies they should understand. Quizzes that test
specific competencies are created ready for learning material authors to insert suitable questions that would
enable the next set of material within the environment.
285
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. FUTURE WORK
This paper describes work in progress developing and using an ontology of mathematical logic to produce
learning materials and structure this material in a manner appropriate for use by students in schools,
universities and industry. Undoubtedly the ontology of mathematical logic will continue to develop and be
enhanced as it is used to inform the development of further learning materials. We expect the ontology to be
enhanced with additional vocabulary, additional languages and cultural terms using strategies such as those
developed by Espinoza (2008) and new connections formed between topics. Nevertheless, the ontology has
reached a sufficient level of maturity to be useful to practitioners. The current approach is for the ontology to
inform the construction of courses within our virtual learning environment. Courses are mostly manually
constructed using the information gathered ontology editor view of the ontology (see Figure 1). On-going
work is to further develop plug-in functionality for the Moodle VLE to create a basic course structure. We
intend for activities in Moodle to be related to specific competency individuals from the ontology and
learning materials to be related to specific topic individuals.
5. CONCLUSION
This paper has described a strategy for developing learning materials and structuring the delivery of that
material using information collected in an ontology. While the work described is on-going, it shows the
advantages of collecting domain information with an ontology to agree consistent use of vocabulary across a
project involving multiple languages and providing a structure to develop learning resources appropriate for a
broad range of learning environments.
ACKNOWLEDGEMENT
This work is supported in part by funding from the EU Lifelong Learning MALog project.
REFERENCES
Berners-Lee, T., Hendler, J. and Lassila O. (2001). “The Semantic Web”, Scientific American, Vol. 284, No. 5, pages
34-43.
Boyatt, R. and Joy, M.S. (2010). “Developing an Ontology of Mathematical Logic”, Scientific Journal of the Technical
University of Civil Engineering Bucharest, 6. pp. 6-11.
Bradford, P. and Porciello, M. and Balkon, N. and Backus, D. (2007), “The Blackboard Learning System: The Be All and
End All in Educational Instruction?”, Vol. 35, No. 3, pp. 301-314.
Dougiamas, M. and Taylor, P. (2003), “Moodle: Using learning communities to create an open source course
management system”, Proceedings of world conference on educational multimedia, hypermedia and
telecommunications, Vol. 3.
Espinoza, M., Gómez-Pérez A. and Mena, E. (2008) “Enriching an Ontology with Multilingual Information”, The
Semantic Web: Research and Applications, Lecture Notes in Computer Science, Vol. 5021, p. 333-347.
Gruber, Thomas (1995). “Toward Principles for the Design of Ontologies Used for Knowledge Sharing”. International
Journal Human-Computer Studies Vol. 43, Issues 5-6, p. 907-928
Fensham, P.J., 1972, Prior knowledge - a source of negative factors for subsequent learning, Research in Science
Education, Vol. 2, No. 1, pp. 50-57.
Henze, N. and Dolog, P. and Nejdl, W. (2004). “Reasoning and ontologies for personalized e-learning in the semantic
web”, Educational Technology and Society, Vol. 7, No. 4. pp. 82-97.
Laoussi, Mona (2012), Ontology in Adaptive Learning Environment, Proceedings of Workshop on Learning Technology
for Education in Cloud 2012, Salamanca, Spain, pp. 167-177.
Libbrecht, P. and Desmoulins, C. and Mercat, C. and Laborde, C. and Dietrich, M. and Hendriks, M. (2008), “Cross-
curriculum search for Intergeo”, Intelligent Computer Mathematics, LNCS Vol. 5144, pp. 520-535.
286
PROFESSIONAL DEVELOPMENT: THEN AND NOW
Susan Bolt
Curtin University
ABSTRACT
Technological developments have altered pedagogies in classroom teaching but approaches to teacher professional
development have remained largely unchanged. The purpose of this paper is to describe an evolving learning process that
spans the last decade and draws from three different investigations into professional development. The author compares
and contrasts the key findings from two independent studies on teacher professional development. One study was situated
in a face-to-face action research context and the other in an online environment. The conclusions drawn about the nature
of effective professional development in these independent studies show that whether in online or face-to-face contexts
educational developers need to consider similar elements. A review of the literature indicated that further research was
required to enhance the development, implementation and evaluation of online professional development opportunities to
meet the in-time learning needs of teachers in sustainable ways. The author concludes by summing up the advantages and
disadvantages of moving towards online teacher professional development and considers possible applications.
KEYWORDS
Teacher professional development, informal learning, online learning
1. INTRODUCTION
Over the past decade technological developments have altered pedagogies in classroom teaching. Even so,
approaches to teacher professional development have remained largely unchanged and typically delivered in
a face-to-face mode (Lloyd & Duncan 2010). Problematically, face-to-face professional development has
often been delivered as one-off workshops off site, whereas best practice models recommend embedded
learning tailored to meet individual needs in the workplace (Bolt 2003, 2009; Guskey 2000; Lloyd et al.
2005; Zepeda 2012). Problems with formal professional development programs are well documented but
successful alternatives are less well publicized (Guskey 2000).
The purpose of this paper is to provide a quick overview of an evolving learning process which spans the
last decade and draws from three different investigations into professional development. The author first
investigated teacher professional development in the context of a funded action research Quality Teacher
Program in Western Australia (Bolt 2003, 2007). On the other side of the country Lloyd et al. (2005)
investigated teacher professional development in the use of information communication technologies.
Subsequently, the author investigated professional learning in the workplace in a study conducted in three
large dynamic organizations (Bolt 2009ab, 2010). The first two studies investigated teacher professional
development in separate, diverse contexts and resulted in the conceptualization of similar yet different
models of professional development. In an unrelated context, the author conducted the third study to
investigate the relationship between professional development and organizational change and development
by considering adult learning in the workplace as a result of formal programs, non-formal programs, informal
learning and incidental learning. Key findings indicated that adults in the workplace learned as much if not
more through informal learning, yet managers and human resource developers rarely planned for it (Bolt
2009ab, 2010).
This paper contributes to current knowledge and practice about professional development. By comparing
and contrasting findings from the three studies, the author considers ways in which formal and informal
learning could be blended through the use of information technology to enhance teacher professional
development. This paper may be helpful to academic developers, educational leaders and human resource
developers who are interested in knowing more about models of professional development suitable for either
online, blended or face-to-face scenarios.
287
ISBN: 978-989-8533-12-8 © 2012 IADIS
2. A TALE OF TWO STUDIES

The two studies undertaken independently in different locations, times and contexts arrived at remarkably
similar conclusions (Bolt 2003; Lloyd et al. 2005). The Bolt (2003) study investigated the significant action
learning experiences of teachers leading to professional growth. In the Bolt (2003) study ten case studies
written by teachers participating in action research projects funded through the Quality Teacher Program over
a one year period were analyzed and emergent themes identified. Although some of the participating teachers
used technology in these projects the professional development focus was on action research in diverse
situations. The Lloyd et al. (2005:1) study was commissioned by Queensland Society for Information
Technology in Education (QSITE) “to inform its development of a position statement on professional
development for teachers in the curricular use of ICT”. It was a small scale study in which conference
delegates were surveyed and a range of educational leaders were interviewed (Lloyd et al. 2005).
Interestingly, the resultant conceptualizations of professional development were similar (Bolt 2003, 2007;
Lloyd et al. 2005). In Table 1 the similarities and differences between the two studies are compared and
contrasted; then described in the following paragraphs.
Table 1. Comparison of key elements of two models of teacher professional development
Framework for improving teaching and learning Elements of effective professional development (Lloyd
through action learning (Bolt 2003) et al. 2005)
Context Context
Collaboration Community
Process (professional development + action learning) Theory and practice
Endurance Time
Professional growth (teaching and learning) Personal growth (ICT)
2.1 Context
Context is an important element of professional development in both face-to-face and online professional
development. In the Bolt (2003) study teachers appreciated opportunities to solve issues of concern to them,
with peers in school-based action research groups supported by organizational structures and infrastructure to
provide resources and time. By contrast, top-down approaches, teacher isolation, and lack of time were
identified as constraining forces. In the Lloyd et al. (2005) study effective professional development was
fostered in contexts in which learning was relevant, meaningful, practical and designed to meet immediate
and ongoing needs (Lloyd & Duncan-Howell 2010). Action research was an important element in both
studies and known to be an effective form of professional development (Bolt, 2003, 2007; Creswell 2008;
Lloyd and Duncan-Howell, 2010; Zepeda, 2012). To sum up, an enabling context is important regardless of
whether the professional development occurs in a linear program or an iterative online network.
2.2 Collaboration and Community

Whether in online, blended or face-to-face contexts there must be opportunities for participants to collaborate
within communities of practice. Lloyd and Duncan-Howell (2010:70) described community as
“collaborations during and following the professional development event and to ongoing connections and
collaborations within local and extended communities”. Bolt (2003:140) noted “collaboration is the conduit
through which professional growth flows and is fundamental to developing a learning organization … and is
vital to developing a learning community”. In both studies collaboration and community were interconnected
and involved action and reflection. In the Bolt (2003) study collaboration and community typically occurred
in school-based groups of teachers in face-to-face situations as they engaged with the iterative processes of
action research. In the Lloyd et al. (2005) study collaboration and community typically occurred online. In
either case, collaboration with others in communities led to participants’ engagement with professional
development as a process rather than as an event.
288
2.3 Theory and Practice

Professional development must provide participants with opportunities to learn and subsequently transfer that
learning into relevant workplace situations. In the Bolt (2003) study ‘theory’ was referred to as a broad range
of professional development inputs in relation to school-based action research projects. Lloyd and Duncan-
Howell (2010) describe theory and practice in ways consistent with action research – iterative, linked to
action and reflection, community, context and growth. Action research supports transfer of learning
(Caffarella, 2002).
2.4 Just-in-time and Sustainable-Over-Time

Time is one major constraining force acting against successful implementation of professional development
programs. Action research is timely, focusing on real life immediate problem-based issues. One of the key
success factors identified in the Bolt (2003) study was ‘endurance’ – supported by the preceding elements of
effective professional development participants sustained their engagement with school-based action research
projects over a one year period to achieve their goals and answer personally relevant questions. Teachers
directed their own learning and scheduled relevant associated action and reflection (Bolt, 2003). The
professional development described in Bolt (2003) is consistent with Lloyd and Duncan-Howell’s (2010:70)
“dual understandings of time” and depiction of effective professional development. Lloyd and Duncan-
Howell (2010) link ‘in-time’ learning with practice and ‘over-time’ to theory as a result of reflection.
2.5 Growth and Development

In the Bolt (2003:155) study, analysis of ten cases identified “professional growth was evidenced by change
at the classroom level in teachers’ pedagogy and student outcomes”. Lloyd and Duncan-Howell (2010) note
the importance of increased personal and corporate knowledge and the ability of online communities to
connect participants with ‘experts’ beyond their immediate situation.
Lloyd and Duncan-Howell (2010) refer to online professional development metaphorically, depicting it as
a Celtic knot rather than traditionally as a ‘journey’. Poignantly, they note that if professional development is
a journey then for teachers in the 21st Century “it is unlikely there is a destination” (Lloyd and Duncan-
Howell 2010:73). Recent rapid technological changes have shortened the shelf life of knowledge, resulting in
dynamic environments and increasing the need for lifelong learning, responsibly shared by individuals and
organizations (Burns 2002). Consequently, there is increased need to learn informally ‘on-the-go’.
3. THIRD STUDY SHOWS IMPORTANCE OF INFORMAL LEARNING

In the previous two studies participants were teachers and educational administrators. In the third study
participants were adult learners in the workplace many of whom were providers of training and professional
development. Data were collected through 70 semi-structured interviews and 210 responses to a 23 question
survey. Overall, participants in this study rated their personal informal learning as highly effective. By
contrast, managers and human resource developers typically planned formal programs without providing
structures to support informal learning (Bolt 2009). Four of the ten key recommendations from the Bolt
(2009:214) study were to “integrate informal learning into training and development programs and work
practices”, “enhance managers’ understandings of different types of learning and how they can most
effectively be used in different situations”, “include transfer of learning strategies” and “adopt a hybrid
approach that meets individual and organizational needs”. In this study participation in informal learning
included inductions, mentoring, networking, attending meetings, Internet searches, work shadowing, reading
and online learning (Bolt 2009: 136, 154).
Online learning environments can support informal learning by facilitating collaboration and reflective
practices, thus, enabling participants to construct knowledge and transfer learning through, for example;
journaling, conversing asynchronously and synchronously, sharing audio and video (Zepeda 2012; Zygouris-
Coe and Swan 2010). Lloyd and Duncan-Howell (2010) predict that in the future professional development
will occur as self-directed individuals fluidly move in and out of online communities of practice; learning
from each other just-in-time. So, if there is a destination – are we there yet?
289
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. CONCLUSION
A brief overview of literature on professional development indicates we are not there yet (Zepeda 2012).
Online resources have the potential to be used flexibly and widely to support teacher professional
development (Zygouris-Coe and Swan 2010). An advantage of online professional development is that it can
facilitate both formal and informal learning which can be accessed just-in-time and sustained over-time. It
allows teachers to form communities of practice and collaborate with people beyond their face-to-face
associations in both time and space. A significant point raised in this paper is that elements characteristic of
effective professional development are similar for face-to-face and online situations (Bolt 2003: Lloyd et al.
2005). However managers and developers in some large organizations do not plan for informal learning to
occur (Bolt 2009). Managers and developers face challenges due to rapid changes in technology and
participants’ ability to use it. Also, it is challenging to design, implement, maintain and assess the
effectiveness of online professional development programs (Zygouris-Coe and Swan 2010). Even so, the fast
paced dynamic environment of the 21st Century demands agile alternatives to traditional methods. As
workforce dynamics change and the proportion of technologically savvy staff increases there will be greater
demand for just-in-time flexible learning opportunities. As more teachers use technology to teach their
students, the need to provide them with online professional development will increase also. This paper
identifies the need for development of informal online professional development opportunities to support
knowledge creation and transfer of learning in a range of settings. Such developments would require
evaluation and dissemination of findings to the broader academic and educational community.
REFERENCES
Bolt, S. 2003. Teacher identification of significant action learning experiences leading to professional growth.
Unpublished Thesis, Curtin University of Technology, 2003.
Bolt, S. 2007. A framework for improving teaching and learning through action learning. Proceedings of the Fifth
Education in a Changing Environment Conference. Salford, England, pp. 275-282. Retrieved 12 July 2012 from
http://www.ece.salford.ac.uk/proceedings/papers/28_07.pdf
Bolt, S. 2009a. An Investigation of professional learning in dynamic environments. Unpublished Thesis, Curtin
University of Technology, 2009.
Bolt, S. 2009b. A contingency approach: Building the capacity of managers to facilitate professional learning in the
workplace. The International Journal of Knowledge, Culture and Change Management, Vol. 9, No. 5, pp.65-78.
Bolt, S. 2010. Synchronising formal and informal learning in the workplace to enhance professional development. In
Westover, J. 2010. Training the Next Generation of Knowledge Workers: Readings for Effective Secondary Education
and Workplace Learning Practices. Australia: Common Ground Publishing, pp. 187-201.
Burns, R. 2002. The Adult Learner at Work (Second Edition). Allen & Unwin: Sydney.
Caffarella, R.S. 2002. Planning programs for adult learner: A practical guide for educators, trainers and staff
developers. San Francisco: Jossey-Bass.
Creswell, J.W. 2008. Educational Research: Planning, conducting and evaluating quantitative and qualitative research.
Pearson Education Inc: New Jersey.
Guskey, T.R. 2000. Evaluating Professional Development. Corwin: California.
Lloyd, M., Cochrane, J. and Beames, S. 2005. Towards a model of effective professional development in ICT for
teachers. Commissioned report for Queensland Society for Information Technology in Education (QSITE). Retrieved
12 July 2012 from http://www.qsite.edu.au/publications/position/PD
Lloyd, M. and Cochrane, J. 2006. Celtic knots: Interweaving the elements of effective teacher professional development
in ICT. Australian Educational Computing, Vol. 21, No. 2, 16-19.
Lloyd, M. and Duncan-Howell, J. 2010. Changing the metaphor: The potential of online communities in teacher
professional development. In Olofsson, D. and Lindberg, J.O. 2010. Online Learning Communities and Teacher
Professional Development: Methods for Improved Educational Delivery. Information Science Reference: New York.
pp. 60-76.
Zepeda, S.J. 2012. Professional Development: What Works (Second Edition). Eye on Education : New York.
Zygouris-Coe, V.I. and Swan, B. 2010. Challenges of online teacher professional development communities: A state
wide case study in the United States. In Olofsson, D. and Lindberg, J.O. 2010. Online Learning Communities and
Teacher Professional Development: Methods for Improved Educational Delivery. Information Science Reference:
New York. pp. 114-133.
290
DEVELOPMENT OF VISUALIZATION OF LEARNING

OUTCOMES USING CURRICULUM MAPPING
Takashi Ikuta and Yasushi Gotoh

Niigata University
8050, Ikarashi 2no-cho, Niigata City, Niigata, Japan
ABSTRACT
Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS). The objective is
to have groups of teachers belonging to educational programs discuss whether visualized learning outcomes are
comprehensible. Discussions based on teachers' subjective judgments showed in general that visualized learning
outcomes express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level of
success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting and lessons
and assessments that correspond to weighting.
KEYWORDS
Visualization, Learning Outcome, Curriculum Mapping, Graduate attribute.
1. INTRODUCTION
Amid the increasingly fast-paced development of science and technology and the globalization of economic
activities, how to assure graduate attributes has become a matter of the utmost importance. As graduate
attribute frameworks, Japan has “graduate power” and debates about quality assurance in different fields on
the U.K. QAA model. Our purpose, in terms of such quality assurance, is to make the basic knowledge and
understanding which students must acquire as their grounding in a relevant discipline; the domain-specific
skills; the generic skills; and the learning methods and grade assessments needed to obtain such knowledge
and skills, reference points in forming educational curricula in different fields. At present, trials are underway
in the fields of management, languages, literature, law, biology, home economics, mechanical engineering
and mathematical science.
In parallel to this type of discussion about quality assurance in different fields, it will also be necessary
for teachers' groups to discuss individual educational programs. As Kawashima (2008) points out, teachers'
groups involved in educational programs must discuss learning outcomes with the aim of fostering human
resources. If a teacher is not able to perceive learning outcomes as being related to the actual learning
outcomes of her own students, it is highly likely that quality assurance in different fields will end up as an
abstract concept.
At the same time, there are a number of difficulties in the visualization of learning outcomes. In PDP
(Personal Development Planning) advocated in the U.S., students themselves perform reflection on the basis
of the results transcript provided by the university and the learning process record known as PDR (Personal
Development Records) compiled by the student. This is supposed to foster students' ability to plan their own
lifelong development, but it has also been reported that PDP is time-consuming and expensive and hinders a
proper grasp of learning outcomes (Benesse). What is needed is a system which enables students to visualize
learning outcomes for attainment targets.
Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS) on
the basis of these considerations. This system sets lower attainment targets in four educational target
domains, namely: knowledge and understanding, domain-specific skills, generic skills and attitude, and
visualizes these by displaying on a radar chart students' level of attainment for the corresponding learning
outcomes. By visualizing learning outcomes as a radar chart display, it becomes possible to discuss students'
291
ISBN: 978-989-8533-12-8 © 2012 IADIS
quality assurance from the point of view of generic skills and attitude, as well as from the point of view of
knowledge and understanding and domain-specific skills.
Ikuta and Gotoh (2011) proposed PDCA by teachers' groups on the basis of the relative importance of
educational target domains in educational programs. Teachers' groups have a bird's eye view of the
curriculum as a whole, but if quality assurance of graduate attributes is not to end up as an abstract concept
without any substance, they need to examine concrete learning outcomes visualized for each student. More
specifically, what they must do is verify whether visualized learning outcomes are comprehensible, i.e.
whether the radar charts displayed by the system reflect students' attributes and skills.
2. OBJECTIVE
The objective is to have groups of teachers belonging to educational programs discuss whether visualized
learning outcomes are comprehensible.
3. METHOD
Teaching staff from the life sciences, forest environment studies, and agricultural engineering programs and
teachers from the Institute of Education and Student Affairs took part in the study. The study started in
October 2010 and is scheduled to end in June 2012, i.e., it is still ongoing.
The visualization method for learning outcomes is the one shown by Ikuta and Gotoh (2011). First, the
educational targets domain was split into domain-specific academic knowledge, domain-specific skills,
generic skills and attitude, and attainment targets were set at the lower level of this educational target domain.
Next, a contribution ratio was assigned to the attainment targets in each subject forming an educational
program, For example, in subject A, the attainment target in knowledge and understanding was 50%,
domain-specific skills 30%, generic skills 10% and attitude 10%. In the educational target domain, several
attainment targets were set. For example, where there were contributions to several attainment targets in
knowledge and understanding, a further 50% was allocated to each attainment target. A curriculum map
drawn up in this way is called a “weighted curriculum map.” In providing such weighting, lessons and
assessments must be carried out by a corresponding method.
Learning outcome was the total score obtained by multiplying a student's grade assessment by this
contribution ratio and the number of credits. To give an example, if a student scored 80 points in subject A,
his score for knowledge and understanding was obtained by multiplying 80 (points) x 0.5 (50%) x 2 (number
of credits) to achieve a score of 80. This score was totalled for each attainment target to obtain the learning
outcome. Weighting was performed from October 2010 to March 2011.
To find out whether intuitive comprehension of learning outcomes visualized on the basis of actual
students' results was possible, students' performance data up to March 2012 (data at the end of the third year
of the life sciences program, and data at the end of the fourth year of the forest environments studies and
agricultural engineering programs) were used in order to draw up a radar chart. At the same time, materials
were also prepared for understanding a student's relative position on each attainment target.
Using these data, teachers belonging to the programs were interviewed and the results analyzed. The
analysis transformed speech data into textual data, split these data for each segment and then categorized
them. Although three educational programs were examined in this study, Niigata University offers 42
programs and in future the data must be supplemented and verified. For this reason MAXQDA10 was used
as the analysis software.
4. RESULTS
4.1 Visualized Learning Outcomes are Comprehensible

In general, visualized learning outcomes clearly express students' particular characteristics and are
comprehensible. In forest environment studies, the nature of the curriculum means that the subjects studied
292
by students are more or less identical and in many cases the shape of the radar chart display was similar. The
difference between learning outcomes for students with good grades and students with poor grades was also
immediately obvious from the size of the radar chart. (Figure.1)
Figure 1. An example of visualized learning outcomes

By looking at the shape of the radar chart for each student, it is possible to understand which specialized
domain subject students were learning and when. Students with an extremely low GPA have a distinctive
radar chart display and since they can grasp visually which attainment targets have not been reached teachers
are likely to find such radar charts useful when advising students.
In life sciences, there are four specialized domains: housing studies, clothing studies, dietary habits and
family resource management and since the subjects taken by students in each of these domains are virtually
identical, typical patterns were generally confirmed for all four.
It was proposed that using visualized learning outcomes in this way to carry out peer reviews among
students in the same domain might be useful in setting targets for the next stage.
4.2 Need for Pointers for Self-Assessment of Learning Outcomes

In order to enable students to self-assess their own learning outcomes, some teachers thought that they should
be provided with some kind of indicator or pointer. The idea is that for example, if one role model was drawn
up for forest environment studies and four role models for life sciences in the domains of housing studies,
clothing studies, dietary habits and family resource management, respectively, and if students could compare
themselves to these role models, self-assessment would become easier.
There are a number of methods concerning which pointers to use. Suggestions included: using actual data
from excellent students and virtual data; showing the minimum line; comparing one's own performance with
the standard performance by using the average value and the mode in subjects taken by that student; or
combining these methods, i.e. comparing one's own position to both the excellent line and the minimum line.
4.3 Need for Continuous Examination of Curriculum Map Weighting is

Needed
When the relative position of students for each attainment target was discussed, attention was drawn to the
fact that some attainment targets did not seem right. Some students who had a feeling that in terms of
attainment targets they should be placed higher up, found that they were not particularly highly placed.
When the weighted curriculum map was checked, it was found that only a small number of subjects had been
293
ISBN: 978-989-8533-12-8 © 2012 IADIS
weighted for that attainment target and students who scored poorly had done so because they had not taken a
weighted subject.
On this basis, it seemed clear that further debate was needed about whether attainment targets were too
fragmented or whether there were too few weighted subjects.
4.4 Need for Discussion about Whether Lesson Contents and Assessments
Correspond to Weighting
Where teachers' groups in life sciences programs weighted curriculum maps by giving, for example, 50% to
knowledge and understanding and 30% to domain-specific skills, they shared a recognition that lessons and
assessments must also be carried out by corresponding methods. As a result, compared to pre-weighting
(around March 2011), at present (March 2012), teachers are more aware of the relationship between
attainment targets and lessons and assessments and revision is possible. In addition, attention was also drawn
to the fact that teachers had greater awareness of the connection between other subjects making up the
educational program and their own subject.
5. DISCUSSION
Discussions based on teachers' subjective judgments showed in general that visualized learning outcomes
express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level
of success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting
and lessons and assessments that correspond to weighting. To identify the reliability of visualized learning
outcomes, some kind of direct assessment method such as learning portfolio or performance assessment will
be need.
Ikuta and Gotoh (2011) proposed that “If individual teachers revise the syllabus, extract attainment targets
and consider assessment methods and grade allocations in the educational program as a whole, they will be
able to discuss educational programs for fostering human resources in general not only from the point of
view of domain-specific knowledge, but also from the point of view of generic skills and attitude. Such
discussions will have wide-ranging potential application in assuring quality in higher education.”
Among teachers belonging to educational programs, there are disagreements even about individual
generic skills and discussions may finish in abstract definitions. We want individual teachers to say, “What
do these generic skills mean in concrete terms for students in my class? What lessons should I design to
foster such skills? How should I assess them?” For this reason, as in the present study, it is essential to
continue to examine how learning outcomes are being visualized for real, individual students and to provide
weighting to the whole curriculum by taking a panoramic view of it.
REFERENCES
Benesse Education Research & Development Center, 2008. Report of Higher Education Reform in United Kingdom. –
Eikoku ni okeru koutou kyouikukaikaku doukou chosa. Retrieved May28 ， 2012, from
http://benesse.jp/berd/center/open/report/oubei_19/index.html (in Japanese)
Ikuta,.T & Gotoh,Y. Visualization of Learning outcomes using Curriculum mapping. Proceedings of Cognition and
Exploratory learning in Digital Age 2011,323-324
Kawashima, T, 2008. International Trends in Outcome-Based Higher Education Reform and Their Implications for
Japanese Higher Education. Nagoya Higher Education Research, 8:173-191 (in Japanese)
The Quality Assurance Agency for Higher Education, 2001. Guidelines for Higher Education.Retrieved May28，2012,
from http://www.qaa.ac.uk/academicinfrastructure/progressfiles/default.asp
294
VIRTUAL REALITY AS A TOOL IN THE EDUCATION
Sandra Dutra Piovesan1, Liliana Maria Passerino1 and Adriana Soares Pereira2
1
Universidade Federal do Rio Grande do Sul
2
Universidade Federal de Santa Maria
ABSTRACT
The virtual reality is being more and more used in the education, enabling the student to find out, to explore and to build
his own knowledge. This paper presents an Educational Software for presence or distance education, for subjects of
Formal Language, where the student can manipulate virtually the target that must be explored, analyzed and studied.
With a simple interface, with easy comprehension and using, this paper presents an educational software where the
student manipulates the objects that will be studied in 3D, getting easier the study of concepts and theories about
Automatons, Regular Expressions and Minimization of Automatons interacting directly with the object in 3D. To the
design of the automatons, the software uses tolls in 3D, as the Blender and the VRML (Virtual Reality Modeling
Language) and to the publishing of a page on the internet it is integrated the Program Language PHP (Hypertext Pre
Processor). The results reached with the use of the developed software show the attributes that make the ideal Virtual
Reality for situations of research and learning taking the discipline as a reference of the classroom to the computer labs
and making more interesting to the student, making the learning easy.
KEYWORDS
Virtual Reality, Educational Software, Formal Languages, 3D, Blender, VRML.
1. INTRODUCTION
The constant evolution of the technology is taking the education to new ways, much more attractive to the
students, making possible the use of new tools, taking to an evolution on the teaching process. The Virtual
Reality takes an important place in this evolution.
Several are the definitions about the virtual reality, but in general, they refer to the an immersive and
interactive experience based on graphic images in 3D generated in real time by computer, in other words, it is
a simulation generated by a computer, about a real or just an imaginary world. Another definition, more
specific is: virtual reality is a computer interface that permits the user to interact in real time, in a
tridimensional space generated by a computer, using their feelings, through special devices (Kimer, 2012).
The user ca notice the virtual world, through a window built by the monitor screen or by projection screen or
it can be inserted in the real world through a helmet (HMD) or multi projections rooms (caves) and
interaction devices (Kimer, 2012).
The education can be seen as a discovery, exploration and observation process, besides the eternal
construction of the knowledge. With this, the specific characteristics of the virtual reality can transform it on
a mighty tool in service for everybody who seeks for an education evolution. Many things that until short
time ago were dreams, nowadays, with the current technological advances became reality.
With the virtual reality acting in the education we can discover, explore and build knowledge about places
and situations that we could never explore. The great potential of the virtual reality is exactly on these
possibilities, not only through classes or physical objects, but also through the virtual manipulation of the
target to be explored, analyzed and studied.
We can understand as Formal Language Theory and of the automatons, the study of the math models that
enable the specifications and the language recognizing (in the wide sense of the word), their classifications,
structures, priorities, characteristics and inter-relationships (Furtado, 2012). The importance of this Theory in
Computer Science is double: it supports as the other theoretical aspects of the Computer Science, as it
justifies several computer applications such as language processing, pattern recognizing, modeling systems
(Furtado, 2012).
295
ISBN: 978-989-8533-12-8 © 2012 IADIS
The proposed educational software has as a purpose to help students and teachers to have a more
complete and interesting view about the automatons and about the changes occurred on their minimization. It
also has as a purpose to make the subject more interesting to the students, making the learning simpler. We
intend to make the student more participative, making the learning interactive. Trough the software
developed it’s also possible to evaluate the learning of the students on the distance modality, making easier
the teacher’s work who receives in his registered electronic address the result of the exercises done by the
students on the 3D software.
2. VIRTUAL REALITY
Virtual reality, augmented reality and their variations represent computer interface techniques that take into
account the tridimensional space. In this space, the user acts in a multi sensorial way, exploring aspects of
this space through the viewing, hearing and tact. According to the available technology it’s also possible to
explore the smell and the taste. Body perceptions, like cold, heat and pressure, are included in the tact,
through the skin (Kimer, 2011).
Virtual reality is characterized by three basic ideas: (Pinho, 2004)
• Immersion: the user has the real sensation of being inside the virtual world of the computer. Devices
that make this sensation: digital helmets and digital cave.
• Interaction: the user manipulates virtual objects. Devices that make this sensation: digital gloves.
• Involvement: exploring of a virtual environment, it’s as if the user took part of the virtual world and
he can interfere directly in result of the application, the user can navigate on the virtual environment
in a passive or active way.
2.1 Virtual Reality in the Education

The technological revolution has been permitting the use of new approaches in the teaching-learning process.
One of the conductive technologies to the building of innovative tools for the education is the Virtual reality,
which offers tridimensional computer environments with advanced forms of interaction that can provide
more motivation to the learning process.
A very short time ago, we could consider that the great potential of VR use was in small groups placed in
large urban centers and in teaching and researching institutions. However, the integration VR-VRML
democratized its access, expanding more and more its potential and using fields (BARILLI et al, 2012).
With the help of resources of some modeling and animation programs as the Blender 3D, for example, the
VR use can help students in the comprehension and assimilation of concepts, coming up as a valid alternative
to get good results.Other benefits are observed with the use of Virtual Reality in the education.
According to Clark (2006) the Virtual Reality can be used to make the learning more interesting and fun
with the purpose of improving the motivation and attention, decreasing costs when using the objective and
the real environment no matter how expensive the simulation is. It also makes possible that situations that
were impossible to explored in the real world can be done, for example: exploring a planet like Mars,
traveling inside the human body, doing submarines explorations or inside caves, visiting very small places to
be seen (molecules) or very expensive or very far away, or yet because this place is in the past (historical
places).
3. FORMAL LANGUAGES
The Formal Languages were developed in 1950 with a purpose of developing theories related to the natural
languages. (Menezes, 2005) But soon it was checked that the formal languages were ideal to the languages
study in the Computer Science area. In the languages study two kinds of problems to be solved and treated
were found, the syntactic and the semantic. The syntax treats about the grammatical checking of the
programs, the free properties of the language, as the meaning of a program.
The study of the Regular Languages is done starting from three formalisms (Menezes, 2012):
296
• Finite automaton: it’s the recognizer formalism, being a set of states of a finite system;
• Regular expression: it’s the denotational formalism, it defines how to build the words of the
language, it’s a basic set of language operation;
• Regular grammar: it’s the generator formalism that corresponds to the production rules.
A finite Automaton is a system of sequential finite states that represent a model of Computer Science,
widely used in the Formal and Compiler Languages, being used to studies of Formal Languages theory in
Computer Science. A Finite Automaton can be deterministic, non-deterministic or with empty movements
(Menezes, 2012).
4. SOFTWARE DEVELOPMENT METHODOLOGY

The educational software developed is based on the Virtual Reality oriented to the Formal Language subject,
having as a target the study of Automatons Minimization. To the mounting of this software a Blender
modeler was used, enabling the building of images in 3D of the automatons.
The Blender modeler permits to explore its files to the VRML 2.0 language that is a language used to the
descriptions of virtual worlds enabling this way that corrections on the images be done using the command
lines.
It’s also necessary that the Plug-in Cortona be installed on the computer where the software will be used.
The Plug-in opens the navigator with controls that permit that the user to visualize the images by different
angles, enabling thus, the interaction of the student with the objects to be studied.
The Blender modeler used to the building of the Automatons images don’t permit that letters and numbers
be exported, so it’s necessary that the letters and numbers, as for example: q1, q2, a, b, and 0 be placed on the
images through the VRML Language.
The software is available through a webpage on the internet, where the students have access to the initial
page.
Through the initial page the users will be able to access the Menu of the page that shows the study option,
it presents to the student the possibility of downloading the Plug-in Cortona and it also leaves available some
links for reading.
The software will be available through pages built using the PHP program language. The student will be
able to choose to answer questions about regular expressions before performing the Automatons’
Minimization.
The automatons’ minimization is done on a screen where the user visualizes the Automatons’ design in
3D and he must insert some state that is necessary ant to use the table and the lists to find the equivalent
states that can be removed. When the user finds the equivalent states, he will be able to visualize the
minimized Automaton in 3D. If the student doesn’t get the right equivalent states, the page will advertize him
about the error and the student will be able to repeat the exercise.
The page presents exercises with multiple choices, where the student is going to answer the questions and
check if the chosen answers were correct, this way the student will be able to test his theoretical knowledge.
The page advertises the student when the options done in the questions of multiple choices are the correct
ones and the student will have the option of repeating these wrong questions.
The page presents a Material for Reading that is destined to explain doubts about Regular Expressions
and Automatons’ Minimizations. This page can also be used by educators in their presence or virtual classes.
The page also presents a help material that explains how to use the pages with the Minimization exercises.
This help is available through the Menu of Minimization.
If the material for reading doesn’t solve all the doubts, some links that were used in the work are also
available in a web page. The answers of the questions of Minimization are also available in the page of the
subject, and they can be accessed through the Help page. This page is to help teachers and students, if some
problems about solving these exercises happen.
Using the system above with a package of distribution apache with PHP provider that owns the
phpMyAdmin toll, a bank of data was created to store the login and the students’ password to do the
evaluation. The student’s login must be the email that he desires to receive his evaluation, and the teacher
will receive in an email that is already defined in the program. To facilitate the use, the bank will be exported
and it must be imported by the teacher who wants to use the system.
297
ISBN: 978-989-8533-12-8 © 2012 IADIS
Then it has done a checking to check if the answers are correct. When all the exercises are done the
student will send the results and the grade automatically to the email address registered in the beginning like
a login and to the teacher’s email, so this way the evaluation is done automatically.
5. CONCLUSION
The educational software helped to take the Formal Language Subject from classrooms to the computer labs,
making the teaching/learning process more interesting and pleasant to the students, facilitating the teacher’s
work during the evaluation performance too.
The Virtual Reality presents an opportunity of learning with a real situation, but artificially created,
facilitating the visualization and the interaction sensation with the study focus. When we can’t have the real
experiences, the Virtual Reality is irreplaceable. The simulation in the VR also permits us to be in hard and
dangerous situations, which aren’t usually accessible in the real world. Furthermore, the VR permits to take
to the students complex themes of hard learning and sometimes impossible to show.
The software presented was used during a year in the Formal Language Subject, at the X University,
where it was found a better appropriation of the concepts and a bigger facility in the activities performance
with automatons’ minimization, resulting thus in an improvement in the students’ evaluation who used this
software with VR.
REFERENCES
Barrili, E., C., V., C.; Ebecken, N., F., F.; Cunha, G., G.; 2012. The technology of virtual reality resource for formation in
public health in the distance: an application for the learning of anthropometric procedures. In: Scielo. Disponível em
< http://www.scielo.br/pdf/csc/v16s1/a57v16s1.pdf>, 2012.
Clark, Donald. 2006 Motivation in e-learning. Disponível em: <http://www.epic.co.uk> Março, 2012.
Furtado, O. J. V.; 2012. Linguagens Formais e Compiladores. Universidade Federal de Santa Catarina, Centro
Tecnológico Departamento de Informática e de Estatística. Diponível em <
http://www.ime.usp.br/~jef/tc_gramaticas.pdf >, 2012.
Kirner, C. Realidade Virtual e Aumentada, Acesso em Março 2012, Disponível em
<http://www.realidadevirtual.com.br>.
Kirner, C.; Kirner, T., G.; 2011. Evoluça~e Tendências da Realidade Virtual e Aumentada. In: Pré-Simposio SRV, 2011.
Menezes, Paulo Blauth, 2005.Linguagens Formais. Editora Sagra Luzzatto.
Pinho, Marcio Serolli. Realidade Virtual. PUC, (2004).
Rangel, Jose Lucas. 2012. Material Didático Relativo à Disciplina de Linguagens Formais. Disponível em www-
di.inf.puc-rio.br/~rangel/lf.html.
Rangel, Jose Lucas. 2012. Material Didático Relativo à Disciplina de Linguagens Formais. Disponível em www-
di.inf.puc-rio.br/~rangel/lf.html.
Von Schweber, L. & Von Schweber, E; 1995.Cover story: realidade virtual, PC Magazine Brasil, pp. 50-73, v. 5, n. 6,
junho, 1995.
298
USING DIGITAL STORYTELLING TO IMPROVE

LITERACY SKILLS
Helena Menezes
Instituto de Educação, Universidade de Lisboa
ABSTRACT
The paper shows the importance of Storybird, an online platform, for developing writing and storytelling among young
learners of a foreign language.
KEYWORDS
Digital literacy, storytelling, creative writing, foreign language
1. INTRODUCTION
Story telling is one of the most ancient forms of human expression. In an increasingly complex and
media-saturated world there are a vast range of digital tools that allow students to express through digital
narratives. Digital storytelling emerged over the last few years as a powerful teaching and learning tool that
engages both teachers and their students, (Robin, 2008). Teachers must invest more time/effort in getting
both content producers and consumers to develop their storytelling and narrative skills.
There are several definitions for ‘Digital Storytelling’ however we identify with the one from the Digital
Storytelling Association that defends that “Digital Storytelling is the modern expression of the ancient art of
storytelling. Throughout history, storytelling has been used to share knowledge, wisdom, and values. Stories
have taken many different forms. Stories have been adapted to each successive medium that has emerged,
from the circle of the campfire to the silver screen, and now the computer screen.” (Leslie Rule,2010)
Digital stories usually contain computer based images, text, recorded audio narration, video clips and
music. They vary in length but in education the most used last between two and ten minutes. Topics range
from a wide area, covering almost everything thinkable.
2. STORYBIRD
Storybird is an extremely engaging collaborative storywriting website that embodies three ideas – creating,
reading, and sharing. It is also a collaborative storytelling tool that allows students to focus more on the
content of their writing rather than drawing pictures. Students are provided with the pictures - free collections
of art. They just have to add the words to write stories. Once the art is chosen, students are able to build their
story by dragging and dropping pictures and creating/writing a story to match the pictures chosen. Stories can
enclose a variety of genders – poetry, mysteries, tales, among others.
For teachers it is very easy to use because Storybird requires minimal teacher preparation and allows
them to easily create individual user accounts for students. With that, teachers can view all story books that
students are making. Storybird also has online safety for young students built into it. Storybird can be used
collaboratively with, either with another student in class or school, or with students from different schools in
the region or even from another country.
Storybird can also be used as a way to collaborate between parent and child. Several experiments were
made in the USA in order to bridge home and school by having a child and their parents writing pages in a
book. Afterwards schools have a book share day where parents come to school and read the story with their
children to the rest of the class.
299
ISBN: 978-989-8533-12-8 © 2012 IADIS
Storybird is also excellent for many writing assignments (feature built in the platform) helping even the
most hesitant writers to bring out of their shell as this can be considered an engaging activity.
3. APPLICATION IN EDUCATIONAL CONTEXT

Storybird was introduced and demonstrated in a Language Teacher Master class in the Polythecnic of Castelo
Branco, Portugal. It arose enthusiasm among the student teachers and so they decided to design activities to
be used in schools with their own students. As there were teachers from different levels of teaching, activities
were designed for kindergarten students up to secondary students. I´m going to describe the implementation
and outcomes of some of those activities below:
1. The first one was implemented in two Primary Schools and intended to pair up students from two
classes in different schools in the region. They had to write a story together. This was very important because
each student brought his/her own style to the story while working together. First they chose a theme. Then
they were given a vast array of pictures that could accompany that theme. They had to choose the pictures for
the story and then drag and drop each picture onto the pages. Afterwards they discussed orally about the story
and then wrote it. The fact that the work could stay unpublished until it was finished allowed it to be worked
on and improved over several class periods. Each class worked on the stories at their own pace and rhythm
and once the stories were finished, students read/tell the story via Skype to the other class. Afterwards the
story was published online and printed.
Writing and reading for an audience encouraged and developed literacy skills. Storybird also helped the
reluctant and shy students to write on their own. The use of artwork allowed students to develop deep
reflection and higher-order thinking.
2. Progressive story project
This activity was developed in a language class in a Basic School. The class was divided into groups. One
group of students started the story and then the next day, the rest of the students read what the first group
wrote, then the second group added to it and so on until the end of the story. All the students participated
actively. Students loved seeing how the story that they started went on and ended.
In this case it was not only a great writing opportunity, but a great reading one as well. Creativity was
high. All the students wanted to surprise the group that was coming afterwards.
3. In a Kindergarten, the experiment was done orally, using the art available. Children were asked to tell
the story. When they did not agree, they negotiated another development for it. With these wordless picture
books Storybird promoted early writing and early story telling creatively.
With these young students it was also used to develop sequencing skills and understanding the parts of
the story (reading/writing skills).
There were many other activities developed by the students. These were only some examples.
4. CONCLUSION
Engaging in activities which students see as having a concrete and practical outcome, such as writing
electronic books, allows the students to be creators of something unique, of which they have ownership. They
become comfortable with the act of creation: turning nothing into something.
It also facilitates the transition from teacher-centred, class-based learning to one in which the pupil begins
to acquire individual responsibility.
Because the work is carried out and published online, new, previously inconceivable possibilities of
peer- and self-assessment also emerge in which students start learning via interaction with others.
The art pictures were so good that the students in the activities described above, were inspired to write
and read the colleagues’ stories. The more they read the more they wanted to write in order to read more
stories to the classmates. This was very important because students were able to understand concepts and
ideas better. The sharing of their stories made them comment, negotiate and discuss, giving them the tools to
live in society.
Storybird promoted imagination, literacy, and self-confidence.
300
REFERENCES
http://digitalstorytelling.coe.uh.edu/language_arts.html, accessed on 28th January 2012
http://en.wikibooks.org/wiki/Instructional_Technology/Digital_Storytelling, accessed on 28th January 2012
Burgess, J. 2006. Hearing Ordinary Voices: Cultural Studies, Vernacular Creativity and Digital Storytelling. Continuum:
Journal of Media & Cultural Studies 20(2):pp. 201-214.
Nielsen, P. 2005. Digital storytelling as life writing, Speculation and Innovation (SPIN) conference, Queensland
University of Technology, Brisbane, Australia.
Robin B. 2008. Digital Storytelling: A Powerful Technology Tool for the21st Century Classroom. The College of
Education and Human Ecology, The Ohio State University
301
SOME ASPECTS OF MATHEMATICAL MODEL OF

COLLABORATIVE LEARNING
Yasuyuki Nakamura, Koichi Yasutake* and Osamu Yamakawa**

Graduate School of Information Science, Nagoya University
A4-2(780) Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan
*Graduate School of Social Sciences, Hiroshima University
*1-2-1 Kagamiyama Higashihiroshima, Hiroshima 739-8525, Japan
**Center for Arts and Sciences, Fukui Prefectural University
**4-1-1 Matsuoka-Kenjyojima, Eiheiji-cho,Fukui, 910-1195, Japan
ABSTRACT
There are some mathematical learning models of collaborative learning, with which we can learn how students obtain
knowledge and we expect to design effective education. We put together those models and classify into three categories;
model by differential equations, so-called Ising spin and a stochastic process equation. Some of the models do not contain
interactions among students and we discussed the possibility of improvement of the models for collaborative learning.
KEYWORDS
Mathematical learning model, Collaborative learning, Differential equation, Ising spin, Stochastic process
1. INTRODUCTION
In recent years, the theoretical study of teaching-learning processes has attracted the attention of an
increasing number of scientists. Early studies on those processes have been conducted by psychologists and
sociologists (Piaget, 1929; Vygotsky, 1978). Although the topic is so difficult and complex that there are still
many open questions, the study of cognitive processes has developed into an active area of multidisciplinary
investigation as a number of physicists have got interested in research areas such as economy, social science,
biology and so on. Since Hake reported that the performance of the students can be enhanced using a
teaching approach involving collaborative group work, in contrast to the traditional non-interactive lectures
(Hake, 1998), the processes of learning and understanding physics and mathematics have become the focus
of cognitive research. In order to precede the research, mathematical model of teaching-learning process have
been proposed and studied.
The aim to study teaching-learning process by the use of mathematical model is to investigate the
influence of the structure of the group works on the achievement of the students and to design effective
education. In this reflection paper, we put together mathematical models of learning process and summarize
some aspects of those models.
2. MATHEMATICAL LEARNING MODEL

There are some mathematical models of teaching-learning process and we classify these models into three
categories in the following subsections.
2.1 Differential Equation Modeling

Pritchard et al proposed models of teaching-learning process by the use of differential equation (Pritchard,
2008). The models are based on various theories of learning: tabula rasa, constructivist, and tutoring. They
305
ISBN: 978-989-8533-12-8 © 2012 IADIS
predict the improvement of the post-test as a function of the pre-test score depending on the type of
instruction. One of the models is the connectedness model combined with tabula rasa and constructivist. The
connectedness parameter β measures the degree to which the rate of learning is proportional to prior
knowledge; dUT / dt = −βα KTUT − (1− β )α ′UT . UT and KT are fraction of unknown and known in a
test domain T respectively and they are dependent of the amount of instruction t . α and α ′ are control
parameters. The first term of the right hand side (rhs) is motivated by the constructivist view that students
learn new knowledge by constructing an association between it and some prior knowledge. The second term
of rhs is motivated by the tabula rasa theory of learning.
Although the model itself is quite simple and an exact solution can be obtained analytically, it fits existing
data by sharply determining a parameter. However, these models do not describe a collaborative learning
among students.
2.2 Ising Spin Modeling

Bordogna and Albano proposed a mathematical model of teaching-learning process in a classroom by the use
of a constructive approach (Bordogna and Albano, 2001). In the model, interactions between students and
teachers are described with a set of equations similar to those that describe magnetism in materials. They
used so-called generalized Ising spin σ j that is the knowledge of the student j and is defined as a dynamic
variable such as −1 ≤ σ j ≤ 1. Students play the role of spins and their knowledge of a subject is similar to
the orientation of the spin. A teacher behaves like an external magnetic field trying to align the student’s
knowledge to the right direction. The model is described as follows. The cognitive impact of the teacher on
the student j is j = PjT (1− σ j σ T ) ,
CI TS where σT and PjT are the knowledge of the teacher and his/her
ability to persuade the student j, respectively. Within groups of N students, the cognitive impact of the
j =∑
N
student-student interaction is given by CI SS [Pij (1− σ iσ j ) − Sij (1+ σ iσ j )]sign(σ i / σ T ) ,
i=1,i≠ j
where Pij and Sij are mutual persuasiveness and support of student i to j respectively. The knowledge σ j
changes according to the total cognitive impact stochastically by the use of Metropolis algorithm.
The simulation results of the model is consistent with well-established empirical results, such as the
higher achievements reached by working in collaborative groups and the influence of the structure of the
group on the achievements of the individuals. Yasutake et al introduced network structures formed by
students into the model and investigated some effects of the network structures (Yasutake, 2011).
2.3 Stochastic Process Modeling

Nitta developed a phenomenological theory of peer instruction (Nitta, 2010). His model describes rather a
short-time process of learning, in contrast to the models presented in the above subsections that describe
long-term learning gain. He modeled the transition of the number of students answering correctly for
multiple-choice questions (MCQ) after discussions among students by the use of a master equation. By
denoting the normalized number of students choosing the answer a for the MCQ q before discussion and
after discussion as ρ1 (q;a) and ρ2 (q;a) , respectively, ρ2 (q;a) is described as
ρ2 (q;c) = ρ1 (q;c) − ∑d(≠c) Tdc (q)ρ1 (q;c) + ∑d(≠c) Tcd (q)ρ1 (q;d) , where c and d represent the correct
answer and distractors, respectively and Tab (q) is the transition matrix that represents the normalized
transition rate of students from answer b to a on MCQ q. Then the master equation was simplified
analytically and he showed that the number of correct answers after peer discussion is approximately given
by a simple function of the number of correct answers before discussion as ρ2 = ρ1 + ρ1 (1− ρ1 ) .
The theoretical curve agrees with data obtained from lectures implementing the peer instruction. In
addition, it was shown that a differential equation derived from the model corresponds to the simple
connected model of Pritchard et al. However details of the peer instruction is not included in the model.
306
3. REFLECTIONS
Although Ising spin model describes the mutual interaction among students and teachers, differential
equation model and stochastic process model do not include elements of collaborative learning. In this
section, we would like to suggest the possibility to extend those models in order to take into account
collaboration among students.
There is a simple extension in the differential equation model described in section 2.1. The unknown
fraction of a student i in a test domain Ui should decrease with the help of other wiser students and could
rather increase because of confusion caused by discussion with less wise students. One possibility to
implement the collaboration is to add a term −α ′′∑ (K j − Ki ) , where Ki is a known fraction of a
j≠i
student i and the subscript T for test domain is omitted here.
It is not straightforward to improve the stochastic process model described in section 2.3. Collaboration
should emerge as a result in elements of the transition matrix Tab (q) . Therefore we have to model each
student’s behavior intended to choose an answer for MCQ before and after discussion and make up the result
statistically to construct the transition matrix. This extension requires further consideration.
4. SUMMARY
We briefly summarized three kinds of models for teaching-learning process; differential equation modeling,
Ising spin modeling and stochastic process modeling. Mathematical modeling approach to design effective
teaching-learning environments has just started and the study hovers at a level of applying models to each
individual case. Further investigation and more general model would be expected.
ACKNOWLEDGEMENT
This work was partially supported by KAKENHI (Grant-in-Aid for Scientific Research) (24650557) from
Japan Society for the Promotion of Science (JSPS).
REFERENCES
Bordogna, C. M. and Albano, E. V., 2001, TheoreticalDescriptionofTeaching-LearningProcesses:
AMultidisciplinaryApproach, Phys. Rev. Lett., Vol. 87, No. 11, pp. 118701-1-4.
Hake, R. R., 1998, Interactive-engagement vs. traditional methods: A six-thousand-student survey of mechanics test data
for introductory physics courses, Am. J. Phys., Vol. 66, No. 1, pp. 64-74.
Nitta, H., 2010, Mathematical theory of peer-instruction dynamics, Phys. Rev. ST Phys. Educ. Res., Vol. 6, pp. 020105-1-
4.
Piaget, J., 1929, The Childs Conception of World, Harcourt, Brace, New York.
Pritchard, D. H. et al, 2008, Mathematical learning models that depend on prior knowledge and instructional strategies,
Phys. Rev. ST Phys. Educ. Res., 4, 010109-1-8.
Vygotsky, L. S. et al, 1978, Mind in Society: Development of Higher Psychological Processes, edited by and translated
by Cole, M et al, Harvard University Press, Cambridge, Massachusetts.
Yasutake, K. et al, 2011, A Simulation Analysis on Learning Effects through Interactions in Networked Communities (in
Japanese), Transactions of Japanese Society for Information and Systems in Education, Vol. 28, No. 1, pp. 50-60.
307
ISBN: 978-989-8533-12-8 © 2012 IADIS
COLLABORATIVE QUIZ GAME DEVELOPED WITH EPIK
Bruno Sampaio, Carmen Morgado and Fernanda Barbosa

CITI/Dept. de Informática, FCT, Universidade Nova de Lisboa, Quinta da Torre, Caparica, Portugal
ABSTRACT
Games are currently being used by an increasing number of people, of different ages and in different contexts. Usually,
they provide fun ways of interaction, collaboration, and competition among players. These aspects may be very important
in an educational context, since many research studies confirm that games have many positive effects on students when
used as learning activities. Therefore, we present in this paper a Collaborative Quiz game concept, which promotes
collaboration between students and motivates them for learning. This game will be created using a graphical development
environment currently under development, denominated Epik.
KEYWORDS
Collaboration, Interaction, Educational Games
1. INTRODUCTION
Educational games, compared to other types of learning materials, can transmit new concepts to players in a
much funnier, interactive and dynamic way. Their use in education can contribute to an increase of students’
motivation and skills acquisition (Sindre 2009). When accompanied by means of collaboration they also
allow the development of skills like communication, cooperation and coordination, all relevant in teamwork
and other daily tasks (Zea 2009).
Despite these advantages, games are not yet widely used in education, mainly due to difficulties involved
in their development and distribution (Torrente 2009). However, in recent studies (Torrente 2009 and
Moreno-Ger 2008), ways to solve some of these problems were already identified, involving the creation of
tools that enable the development of educational games by teachers, and the use of Learning Management
Systems (LMS) as a means of delivering them.
Therefore, we decided to create a graphical development environment for collaborative and competitive
games development, denoted Epik (Edutainment by Playing and Interacting with Knowledge) (Sampaio
2012). This environment will support Moodle (Modular Object-Oriented Dynamic Learning Environment)
integration, allowing teachers to deliver the games as Moodle activities and to reuse learning materials
produced in a Moodle course. Epik will allow the development of collaborative quiz games, mainly because
quizzes are one of the most used learning activities. However, they usually don’t allow collaboration and
competition among students, which will be the main difference between the usual quizzes and Epik games.
In the following section we’ll characterize a collaborative quiz game to be developed with Epik and then
we'll present some conclusions.
2. COLLABORATIVE QUIZ GAME

We will use Epik to create a collaborative quiz game with several multiple choice questions related to a
certain topic. These questions will be associated with different scores and some types of help will be
available for players, a bit like in traditional quiz games. The helps and scores shall serve as a way to foster
collaboration and competition among the players.
The game shall be played by a group of students who may be in the same physical location or
geographically distributed. This game will be organized in different scenarios, each composed of an equal
number of questions for each group member. To progress in the game it will be necessary that all group
308
members respond correctly to the questions from the current scenario (coordination). If any member of the
group is in doubt about a question, he or she could ask for help. Helps may involve interaction with other
group members (communication and cooperation) and may be used a limited number of times throughout the
game. The types of help available will be: (i) learning materials consultation - such as images, audio,
videos, or files that contain the information needed for quick reference; (ii) request hints from other
players - there will be a list of hints associated to each question, from which a team member can choose the
most suitable to help his colleague; and (iii) incorrect answers removal by a player - a team member that
already answered correctly to that question can help is colleague by removing some of the incorrect answers
(also known as 50/50).
The scores associated to each question may vary depending on different factors such as time to answer,
number of failed attempts and helps used. When a player correctly answers a question, he or she will
immediately receive the currently associated score. If the answer is incorrect, the question score will be
reduced in a percentage of its total value. The first player to answer all questions in a scenario correctly will
receive a bonus with a specified value, which will be a very important aspect to encourage competition
among the team players. Although, players must also collaborate with each other to progress in the game by
using helps. By helping a colleague, the player receives a percentage of that question score, while his or her
colleague receives the question score without suffering any penalties. However, what will determine if the
help was successful will be the time the player takes to answer the question correctly after receiving help. If
this time is greater than the given time to answer, the question score will be reduced by half and the player
that helped will get nothing.
When the game ends, the team will be ranked based on its final score and other teams scores. This
ranking is controlled by Epik and will act as a way to encourage competition between different teams. All
these information elements will contribute as a form of awareness, with regard to the state and actions
performed by the players.
Throughout the game it will always be presented to each player the team score (sum of each player total
score), all of the other players’ scores and their respective avatars. For each player, the scores will be
displayed and listed by the following categories: correct answers, incorrect answers, collaboration and total.
Moreover, all the information about team actions (such as, the quickest player to respond on each
scenario, the number of attempts and helps each one required, the individual player score and the team score)
will be logged in Epik and sent to Moodle. This information can help teachers with the students’ evaluation.
Figure 1 shows the example of a game scenario flow, which is related with programming teaching and
has as main objective the demonstration of the abstract data type List behavior. This game may be an activity
evaluated in a lab class or a homework activity, for example. In all the scenarios, the players’ scores and
theirs avatars are displayed on the right side and for each of them their information is highlighted on their
own screen. At game start the scenario on Figure 1a is displayed for each player, where some concepts and a
video are presented to them, in order to explain the List concepts. By clicking on the button at the bottom
right of this scenario, each player goes into the second scenario (Figure 1b), where each one of them needs to
answer a set of questions (the same for all players). If a player asks for a hint about the upper right question,
the system selects another player to help him (Figures 1b and 1c - Anakin asks for a hint and Irina helps him).
Team Score: 0 Team Score: 48.5

The List Anakin
Correct Answers: 0 The list's size is:
If a node with number 6 is added it will:
Anakin
Correct Answers: 10
List or sequence is an abstract data type that implements a sequence of values, where Be ignored because 6 already exists.
Incorrect Answers: 0 10 Incorrect Answers: 1.5
the same value may occur more than once. Each instance of a value in the list is Replace the node with number 6.
Collaboration: 0 9 Collaboration: 0
usually called an item, entry, or element of the list; if the same value occurs multiple Be added to the tail.
times, each occurrence is considered a distinct item. Score: 0 19 Points: 8.5
Be added in sorted order between 6 and 7.
0
Lists have the following properties: Margarete Points: 5 Helps: Margarete
- The size of list: indicates how many elements there are in the list.
Correct Answers: 0 Points: 5 Helps: Correct Answers: 15
- Each element in the list has an index, which is a value between 1 and the list size.
- The head of the list: the first element in the list (index 1). Incorrect Answers: 0 Incorrect Answers: 0
- The tail of the list: The last element in the list (index size of the list). Collaboration: 0 Collaboration: 0
Player
Score: 0 Score: 15
clicks here
The List
Irina Irina
Correct Answers: 0 Correct Answers: 15
Incorrect Answers: 0 Incorrect Answers: 0
Collaboration: 0 Collaboration: 0
Score: 0 If node with number 3 is removed: Score: 15
If a new node with number 4 is inserted on
position 7 it will: All nodes are lost except 1 and 2.
Marco Marco
Correct Answers: 0 Be ignored because 4 already exists. Node with 2 will be linked to 4. Correct Answers: 10
Player Incorrect Answers: 0 Be added between nodes with values 7 and 8. Two lists will be created. Incorrect Answers: 0
clicks here Replace the node on position 7. Node 3 remains there but marked as hidden.
Score: 0 Be added between nodes with values 8 and 9. Points: 5 Helps: Score: 10
Points: 5 Helps:
Continue Helps: 5 Helps: 5
(a) Presenting a concept to players. (b) Anakin asks for a hint on the upper right question.
309
ISBN: 978-989-8533-12-8 © 2012 IADIS
If a node with number 6 is added it will:

Team Score: 48.5 If a node with number 6 is added it will:
Team Score: 48.5
Be ignored because 6 already exists. Anakin Be ignored because 6 already exists. Anakin
The list's size is: Replace the node with number 6. Correct Answers: 10 The list's size is: Replace the node with number 6. Correct Answers: 10
10 Be added to the tail. Incorrect Answers: 1.5 10 Be added to the tail. Incorrect Answers: 1.5
9 Be added in sorted order between 6 and 7. Collaboration: 0 9 Be added in sorted order between 6 and 7. Collaboration: 0
19 Points: 5 Helps: Points: 8.5 19 Points: 5 Helps: Score: 8.5

0 0
Values may be repeated. Send Margarete Hint: Values may be repeated. Timer: 10s Margarete
Points: 5 Helps: Correct Answers: 15 Points: 5 Helps: Correct Answers: 15
Collaboration: 0 Hint and Timer Collaboration: 0
Hints here Score: 15 here Score: 15
The List The List
Irina Irina
Correct Answers: 15 Correct Answers: 15
If node with number 3 is removed: Score: 15 If node with number 3 is removed: Score: 15
If a new node with number 4 is inserted on If a new node with number 4 is inserted on
position 7 it will: All nodes are lost except 1 and 2. position 7 it will: All nodes are lost except 1 and 2.
Marco Marco
Be ignored because 4 already exists. Node with 2 will be linked to 4. Correct Answers: 10 Be ignored because 4 already exists. Node with 2 will be linked to 4. Correct Answers: 10
Be added between nodes with values 7 and 8. Two lists will be created. Incorrect Answers: 0 Be added between nodes with values 7 and 8. Two lists will be created. Incorrect Answers: 0
Replace the node on position 7. Node 3 remains there but marked as hidden. Replace the node on position 7. Node 3 remains there but marked as hidden.
Be added between nodes with values 8 and 9. Points: 5 Helps: Score: 10 Be added between nodes with values 8 and 9. Points: 5 Helps: Score: 10
Points: 5 Helps: Points: 5 Helps:

Helps: 5 Helps: 4
(c) Irina chooses and sends a hint to Anakin. (d) The hint and the timer are displayed to Anakin.
Figure 1. Scenarios’ flow of a collaborative quiz game.

After selecting and sending the hint, it will appear on his colleague screen, beneath the area with the
question, with the timer next to it (Figure 1d).
All mentioned aspects such as the questions, the topic, the scores, the times to answer, the rewards, the
penalties and the actions to log, could be defined by the teacher while developing the game with Epik.
Moreover, the development of these games will be a simple task for the teacher, as their integration in
Moodle.
3. CONCLUSION
With collaborative quiz games we can develop and improve students skills related to the: (i) learning topic,
since they must answer related topic questions, using the information provided in the game, or possibly the
help provided by others. Besides that they can also help other students by choosing adequate hints or by
removing incorrect answers. In both cases the students can acquire and improve their knowledge about the
game topic; (ii) teamwork, in order to finish the game, students must collaborate, which improves their
teamwork capacity.
The competition component introduced by the game scores also motivates the collaboration between
students and improves their motivation to learn. In addition, the teams’ ranks stored by Epik also motivate
students to play again and to get better results.
ACKNOWLEDGEMENT
The authors would like to thank the FCT/MCTES, Pest-OE/EEI/UI0527/2011 Centro de Informática e
Tecnologias da Informação (CITI/FCT/UNL) 2011-2012, for the support given to this work.
REFERENCES
Moreno-Ger, P. et al, 2008. Educational Game Design for Online Education. Computers in Human Behavior, Vol. 24,
No. 6, pp 2530–2540.
Sampaio, B. 2012. Epik, viewed 21 September, 2012, <http://asc.di.fct.unl.pt/~cpm/epik/>.
Sindre, G. et al, 2009. Experimental Validation of the Learning Effect for a Pedagogical Game on Computer
Fundamentals. Education, IEEE Transactions On, Vol. 52, No. 1, pp 10–18.
Torrente, J. et al, 2009. Integration and Deployment of Educational Games in E-learning Environments: The Learning
Object Model Meets Educational Gaming. Educational Technology & Society, Vol. 12, No. 4, pp 359–371.
Zea, N. et al, 2009. Design of Educational Multiplayer Videogames: A Vision from Collaborative Learning. Advances in
Engineering Software, Vol. 40, No. 12, pp 1251–1260.
310
A SURVEY OF STUDENT ATTITUDES ON THE USE OF

SOCIAL NETWORKING TO BUILD LEARNING
COMMUNITIES
Timothy Arndt
Dept. of Computer and Information Science, Cleveland State University. Cleveland, OH, USA
ABSTRACT
Post-secondary students are increasingly receiving instruction by distance learning. This mode of learning can result in
the students learning in isolation, with bad results in learning outcomes. The same type of isolation can occur for part-
time students and those who are working while taking classes. In such circumstances, we believe that it would be
beneficial for instructors to facilitate the formation of small learning communities by distance learning mechanisms. One
mechanism that can be exploited for this purpose is social networking sites, given their popularity among the student
population. We are currently developing innovative approaches to building learning communities for distance learning
students using social networks. As a preliminary step in this research, we have conducted a survey of our target student
population. In this reflection paper, we present the results of our survey and our reflections on the how the results will
guide our future work.
KEYWORDS
Social networking, learning communities, peer support.
1. INTRODUCTION
In order to support students in distance learning as well as part-time students, we are committed to building a
system for developing learning communities among such students. Research has shown that a shared learning
experience is vital for student success [1, 2, 3]. Our intuition is that social networking sites can be leveraged
for such purposes due to their widespread acceptance among the target students [4, 5, 6]. In order to affirm
our intuition, we performed a survey of the target students. The results of the survey and our comments are
presented in the following section.
2. STUDENT SURVEY AND COMMENTS

In order to justify our research in using for distance learning with social networking , a survey of the target
audience – students in a traditional Masters degree program in Computer Science was carried out. The survey
was meant to gauge their interest in using social networking as part of their study routine, as well as
preferences for such use. 15 students replied to the survey. The comments on the results follow the survey
results shown in Table 1.
Table 1. Survey of Students
1. In general, which type of class organization would you prefer?
Traditional, face-to-face. 80% Distance learning. 7% No Preference. 13%
2. Do you use social networking services (e.g. Facebook, Twitter, Google+, LinkedIn, etc.)?
Yes, often. 67% Yes, seldom. 33% No, never. 0%

3. In preparing for exams, understanding homework, etc. what is your predominant study mode?
I study by myself. I study in a group with other students
87% from the class. 13%
311
ISBN: 978-989-8533-12-8 © 2012 IADIS
4. If you answered "I study by myself" to the previous question, would you study with other students in a group if there was a convenient
way to do so online?
Yes. 69% No. 31%
5. If you answered "I study in a group with other students from the class" to question 3., estimate the typical size of the study group.
1 other student. 25% 2 other students. 50% 3 or more students. 25%
6. If you answered "I study in a group with other students from the class" to question 3., do you also typically interact with members of
the study group using a social networking service?
Yes. 50% No. 25% Not sure. 25%
7. Do you currently use social networking services as part of your study routine for classes (e.g. exchanging information about a class
with another student in the class via Facebook)?
8. Would you be interested in using social networking services as part of a study routine for classes in future?
9. As a platform for using social networking services as part of your study routine for classes, would you prefer an existing system (e.g.
Facebook, Google+ or Twitter) or a service created specifically for that purpose?
Existing service. 27% New, specialized service. 53% Not sure. 20%
10. If using an existing social networking service as part of your study routine for classes, which would you prefer?
Existing account on the service. New account on the service dedicated to No preference.
50% classes. 43% 7%
11. If you answered "Existing account on the service" to the question above, would you be willing to grant an application access to your
data on the service in order to better allow class interaction to be supported?
12. If an experimental system for using social networking services for study was available, would you be willing to use it?
The students who completed the survey were all enrolled in traditional (not distance learning type)
Master’s degree-level Computer Science classes. This is reflected in the response to question 1, where the
respondents overwhelmingly prefer traditional, face-to-face classes. The respondents also overwhelmingly
study by themselves, however, in responses that are promising for this research, they all use social
networking services, and they are interested in studying in a group online, if there was a convenient way to
do so. Most students who study in groups do so in groups of 2 or 3, and a fair number of those who do group
study using social networking services to interact. The respondents demonstrate a certain amount of
reluctance to mix personal and study use of social networking as only 27% would prefer an existing service
over a new specialized service and only 50% would want to use an existing account of a social networking
service rather than a dedicated one. The remaining questions are promising, in that students do not currently
integrate social networking services in their study routines, but would be interested in doing so if a system
was available, at least on an experimental basis.
REFERENCES
1. Johnson, R.D., S. Hornik, and E. Salas, “An Empirical Examination Of Factors Contributing To The Creation Of
Successful E-Learning Environments”, International Journal of Human-Computer Studies 66 (2008):356-369.
2. Gunawardena, C.N., “Social Presence Theory And Implications for Interaction and Collaborative Learning in
Computer Conferences”, International Journal of Educational Telecommunications 1 (1995): 147–166.
3. Richardson, J. C. and K. Swan, “Examining Social Presence in Online Courses in Relation to Students’ Perceived
Learning and Satisfaction”, Journal of Asynchronous Learning Networks 7 (2003):68–88.
4. Haythornethwait, C. and M. de Laat, “Social Networks and Learning Networks: Using Social Network Perspectives to
Understand Social Learning”, Proceedings 7th International Conference on Networked Learning, 2010, pp. 183-190.
5. Chatti, M.A., M. Jarke, and D. Frosch-Wilke, “The Future of E-Learning: A Shift to Knowledge Networking and
Social Software”, International Journal of Knowledge and Learning 3 (2007):404-420.
6. Baird D.E. and M. Fisher, “Neomillennial User Experience Design Strategies: Utilizing Social Networking Media To
Support “Always On” Learning Styles”, Journal of Educational Technology Systems 34 (2005-2006):5-32.
312

Conferencia Educación, Cognición, TIC, S PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Conferencia Educación, Cognición, TIC, S PDF

Uploaded by

Copyright:

Available Formats

IADIS International Conference on Cognition and Exploratory Learning in

Digital Age 2012

K1 - THE THREE LEARNING SCIENCES (BIOLOGICAL, ARTIFICIAL, HUMAN) by Professor Norbert M.

- Pedro Isaías, Universidade Aberta (Portuguese Open University), Portugal

- Demetrios G Sampson, University of Piraeus and CERTH, Greece

Ah-choo Koo, Multimedia Univerisity, Malaysia

REDEFINING PUBLIC POLICY EDUCATION THROUGH

Ming Hui Shoon and Yam San Chee

2. TECHNOLOGY’S ROLE IN 21ST CENTURY PUBLIC POLICY

2.1 Literature Review

2.2 Game-Based Learning through Statecraft X

Figure 3. An in-game menu where players modify immigration policy.

2.3 A Curriculum for Digital Empowerment

2.5 Results and Discussion

E-LEARNING 3.0 = E-LEARNING 2.0 + WEB 3.0?

2. HOW WEB 3.0 TECHNOLOGIES CAN BE INCORPORATED IN THE

2.1 Evolution of the Web

2.2 Evolution of e-Learning

2.3 Relationship between Web Evolution and E-Learning Evolution

3. DO WE NEED A LEARNING THEORY FOR THE WEB 3.0?

Table 3. Web 3.0 technologies supported by basic principles of connectivism.

Web 3.0 technologies used in e-Learning 3.0 Basic principles of connectivism

4. CHALLENGES AND ISSUES RELATED TO E-LEARNING 3.0

LEARNING IN OR WITH GAMES?

Jan Hense1 and Heinz Mandl2

2. THE PROMISE OF DIGITAL LEARNING GAMES

3. LEARNING IN CONVENTIONAL ENTERTAINMENT GAMES

3.1 A Learning and Instruction Theory Perspective

3.2 An Emotion Theory Perspective

3.3 A Motivation Theory Perspective

4. QUALITY CRITERIA FOR DIGITAL LEARNING GAMES

VIRTUAL SUMMER SCHOOL IN SECOND LIFE:

Mikhail Fominykh1, Ekaterina Prasolova-Førland1 and Monica Divitini2

In this paper, we explore alternative approaches to technology-enhanced learning, community building,

2.1 Collaborative Educational Visualizations and Role-Plays

Figure 1. Student visualization project Awareness Lab

2.2 Virtual Events

Figure 2. Virtual seminar at NTNU

2.3 Method and Data Collection

3. SUMMARY OF THE STUDY RESULTS

3.1 Collect Phase

3.1.1 Brainstorming the Topic to be visualized

3.2 Relate Phase

3.3 Create Phase

3.3.1 Collaborative Construction

3.4 Donate Phase

5. CONCLUSIONS AND FUTURE WORK

FIVE COMPONENTS TO CONSIDER FOR BYOT/BYOD

Amy S. Ackerman and Melissa L. Krupp

2. PROBLEM AND FRAMEWORK

2.2 Definition of Terms

2.3 Previous Work

3. FIVE COMPONENTS TO CONSIDER WITH BYOT

3.1 Security (Component One)

3.2 Stakeholders need to be vested (Component Two)

3.2.2 Partnering with Stakeholders Recommendations

3.3 Policies (Component Three)

3.4 Professional Development (Component Four)

3.5 Financial Plan for Sustainability (Component Five)

4. SUMMARY AND CONCLUSIONS

4.1 Suggestions of what to expect in the Future

4.2 Limitations of Paper

HOW PRIOR KNOWLEDGE AND COLOUR CONTRAST