Review of research metholdogies, hints and tips for PhD seminar at Aarhus University, Denmark. January 2009
1 of 49
Download to read offline
More Related Content
Social Software Research For Aahrus
1. Overview of Research
Methodologies for Social
Software Research
Aarhus University, Denmark January 21, 2009
Terry Anderson, Ph.D.
Canada Research Chair in Distance EducaAon
2. The context of Social SoFware
ImplementaAon
• DisrupAve (Christensen, 2008) simpler, not
wanted by main stream customers
• Rapid gains in funcAonality
• Cheaper
• AdapAve
3. Studying Technological Based Models
of Change
• Technological determinism (Marx, Zuboff,
Kurzweil)
• Affordances and Social ConstrucAvist (Bjiker,
Stewart and Williams, Dalsgaard
• DomesAficaAon/AppropriaAon –(Silverston,
Ling)
7. • “those who are seeking the strict way of truth
should not trouble themselves about any
object concerning which they cannot have a
certainty equal to arithmeAc or geometrical
demonstraAon”
– (Rene Descartes)
• Inordinate support and faith in randomized
controlled studies
8. QuanAtaAve 1 –
CMC Content Analysis
• Anderson, Garrison, Rourke 1997‐2003
– hcp://communiAesofinquiry.com ‐ 9 papers reviewing results
focusing on reliable , quanAtaAve analysis
– IdenAfied ways to measure teaching, social and cogniAve
‘presence’
– Most reliable methods are beyond current Ame constraints of
busy teachers
– QuesAons of validity
– Serves as basic research as grounding for AI methods of the
future
– Serves as qualitaAve heurisAc for teachers and course designers
11. Is DE Becer than Classroom InstrucAon?
Project 1: 2000 – 2004
• QuesAon: How does distance educaAon compare
to classroom instrucAon? (inclusive dates
1985‐2002)
• Total number of effect sizes: k = 232
• Measures: Achievement, Amtudes and RetenAon
(opposite of drop‐out)
• Divided into Asynchronous and Synchronous DE
11
12. Primary findings
• DE and CI are essenAally equal (g+ ≈ 0.0 to low
average effect) on all measures
• Effect size distribuAons are heterogeneous; some
DE >> CI, some DE << CI
• Generally poor methodological quality
• Pedagogical study features account for more
variaAon than media study features (Clark, 1994)
• InteracAve DE an important variable*
*Lou, Y., Bernard, R.M., & Abrami, P.C. (2006). Media and pedagogy in undergraduate distance
education: A theory-based meta-analysis of empirical literature. Educational Technology
Research & Development, 54(2), 141-176.
12
17. Anderson’s Equivalency Theorem
(2003)
Moore (1989) disAncAons are:
Three types of interacAon
o student‐student interacAon
o student‐teacher interacAon
o Student‐content interacAon
Anderson (2003) hypotheses state:
High levels of one out of 3 interacAons will produce
saAsfying educaAonal experience
Increasing saAsfacAon through teacher and learner
interacAon interacAon may not be as Ame or cost‐effecAve
as student‐content interacAve learning sequences
17
18. Do the three types of interacAon
differ? Moore’s disAncAons
Achievement and Attitude Outcomes
Achievement Attitudes
Interaction
Categories k g+adj. k g+adj.
10 0.342 6 0.358
Student-Student
44 0.254 30 0.052
Student-Teacher
20 0.339 8 0.136
Student-Content
44
74 0.291 0.090
Total
2.437 6.892*
Between-class
Moore’s distinctions seem to apply for achievement (equal importance), but not for
attitudes (however, samples are low for SS and SC)
18
19. Does strengthening interacAon improve achievement
and amtudes? Anderson’s hypotheses
Achievement and Attitude Outcomes
Achievement Attitudes
Interaction
Strength k g+adj. SE k g+adj. SE
Low Strength 30 0.163 0.043 21 0.071 0.042
Med Strength 29 0.418 0.044 18 0.170 0.043
High Strength 15 0.305 0.062 5 -0.173 0.091
Total 74 0.291 0.027 44 0.090 0.029
(Q) Between-class 17.582* 12.060*
Anderson’s first hypothesis about achievement appears to be supported
Anderson’s second hypothesis about satisfaction (attitude) appears to be
supported, but only to an extent (i.e., only 5 studies in High Category)
19
21. Because social soFware use is so
emergent need for
• Demographic studies
– Who is using what for doing what?
– How are they using mulA‐use tools?
VisualizaAon of acAvity and relaAonships
•
Network analysis
•
ConnecAon between context and type of use
•
Data mining
•
22. QuanAtaAve Summary
• Can be useful especially when fine tuning well
established pracAce
• Provides incremental gains in knowledge, not
revoluAonary ones
• The need to “control” context oFen makes results of
licle value to pracAcing professionals
• In Ames of rapid change too early quanAtaAve
tesAng may mask beneficial posiAve capacity
• Will we ever be able to afford blind reviewed,
random assignment studies?
23. Paradigm 2
QualitaAve Paradigm
• Many different varieAes
• Generally answer the quesAon ‘why’ rather
then ‘what’ or ‘how much’?
• Presents special challenges in distributed
contexts due to distance between parAcipants
and researchers
• Currently most common type of DE research
(Rourke and Szabo, 2002)
24. QualitaAve study of Social SoFware
• CriAcally important:
– In early stages of adopAon
– To track effects of user competence and efficacy
– As contexts are personalized
– as tools are appropriated by users for enArely
different tasks than those intended by developers
27. QualitaAve example 2
• Mann, S. (2003) A personal inquiry into an experience of
adult learning on‐line. Instruc8onal Science 31
• Conclusions:
– The need to facilitate the presentaAon of learner and teacher
idenAAes in such a way that takes account of the loss of the normal
channel
– The need to make explicit the development of operaAng norms and
convenAons
– reduced communicaAve media there is the potenAal for greater
misunderstanding
– The need to consider ways in which the developing learning
community can be open to the other of uncertainty, ambiguity and
difference
28. 3rd Paradigm
CriAcal Research
• Asks who gains in power?
• David Noble’s criAque of ‘digital diploma Mills’
most prominent Canadian example
• Are profits generated from user generated
content exploitaAve?
• ConfronAng the “net changes everything”
mantra of many social soFware proponents.
• Who is being excluded from social soFware
30. • Why does Facebook own all the content that
we supply?
• Does the power of the net further marginalize
the non connected?
• Who benefits from voluntary disclosure?
• Why did the One Laptop Per Child fail?
31. QuanAtaAve vs. QualitaAve
Paradigm Wars Rekindled
• Current research “more resembles the pendulum
swings characterisAc of art or fashion, rather than
the progressive improvements characterisAc of
science and technology” (p. 16).
• Slavin (2002) in EducaAonal Researcher
• SoluAon to embrace “evidence based learning”
• Projected to increase from 5% to 75% of US Gov.
funding by 2007 for “research that addresses causal
quesAons and uses random assignments ….” Slavin, 2002
p. 15
33. But what type of research has most
effect on pracAce?
– Kennedy (1999) - teachers rate relevance and
value of results from each of major
paradigms.
– No consistent results – teachers are not a
homogeneous group of consumers but they
do find research of value
– “The studies that teachers found to be most
persuasive, most relevant, and most
influential to their thinking were all studies
that addressed the relationship between
teaching and learning.”
34. But what type of research has most
effect on PracAce?
– “The findings from this study cast doubt on
virtually every argument for the superiority of
any particular research genre, whether the
criterion for superiority is persuasiveness,
relevance, or ability to influence practitioners’
thinking.” Kennedy, (1999)
35. 4th Paradigm
Design‐Based Research
• Related to engineering and architectural
research
• Focuses on the design, construcAon,
implementaAon and adopAon of a learning
iniAaAve in an authenAc context
• Related to ‘Development Research’
• Closest educators have to a “home grown”
research methodology
37. Design‐based research
• Methodology developed by educators for
educators
• Developed from American pragmaAsm – Dewey
(Anderson, 2005)
• Recent Theme Issues:
The Journal of the Instruc1onal Sciences, (13, 1, 2004),
–
Educa1onal Researcher (32, 1, 2003) and
–
Educa8onal Psychologist (39, 4, 2004)
–
See bibliography at
–
hKp://cider.athabascau.ca/CIDERSIGs/
DesignBasedSIG/
• My ar8cle at www.cjlt.ca/abstracts.html
39. • “design‐based research enables the creaAon
and study of learning condiAons that are
presumed producAve but are not well
understood in pracAce, and the generaAon of
findings oFen overlooked or obscured when
focusing exclusively on the summaAve effects
of an intervenAon” Wang & Hannafin, 2003
42. Design Based research and the Science of
Complexity
• Complexity theory studies the emergence of
order in mulAfaceted, changing and previously
unordered contexts
• This emerging order becomes the focus of
iterate intervenAons and evaluaAons
• Order emerges at the “edge of chaos” in
response to rapid change, and failure of
previous organizaAon models
44. • Need to study usability, scalability and
innovaAon adopAon within bureaucraAc
systems
• Allow knowledge tools to evolve in natural
context through supporAve nourishment of
staff
47. Personal Academic use of social
soFware
• Do create a blog to document your research
journey
• Follow blogs of those you admire
• Self archive and only give your copyright away
under unusual circumstances
• Support open access journals ie
www.irrodl.org
49. Conclusion
• EducaAon research is grossly under‐resourced to
meet the magnitude of opportunity and demand
• Paradigm wars are unproducAve
• Design‐based research offers a promising new
research design model
• The semanAc web offers promise of greatly
enhanced collaboraAon and research
disseminaAon