Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo

J.D. Fletcher, Institute of Defense Analyses and Sigmond Tobias, SUNY

•Download as PPTX, PDF•
•
SeriousGamesAssoc
SeriousGamesAssoc

"On Tour in the Garden of Empirical Analysis" The authors report findings, both solid and speculative, after six years of an ongoing effort to collect, organize, and understand all available empirical research on the use of games to create environments in which people learn.

1 of 32
On Play in the Garden of Empirical Analysis Serious Play Conference 20-22 August 2013 Redmond, WA J. D. Fletcher Institute for Defense Analyses Sigmund Tobias State University of New York at Albany fletcher@ida.org
Synthetic Environments: E.g., Simulations & Games
On Simulations and Games: Similarities Simulations Games Synthetic environments Synthetic environments Missions Missions Successful mission completion despite competition Successful mission completion despite competition Highly interactive Highly interactive Governed by rules of engagement Governed by rules of engagement Selective realism Selective realism
On Simulations and Games: Differences Simulations Games Will sacrifice entertainment in favor of reality Will sacrifice reality in favor of entertainment Scenario/realistic tasks Storyline/fictional quests Emphasis on task completion Emphasis on competition and levels Not necessarily interactive Necessarily interactive Focus on (rule) accuracy/detailed Focus on (rule) clarity/stylized Not all simulations are games All games are simulations
Synthetic Environments: Advantages • Safety • Economy • Visibility • Time Control
1909 Antoinette Simulator 1910 Saunders Teacher 1930 Link Trainer ?? Today
Top Gun: An (Unintentional) Warfare Experiment Air-to-AirCombatExchangeRatio EnemyLosses/U.S.Losses 2:1 4:1 6:1 8:1 10:1 12:1 1965-1968 1969 1970-1973 USN Top Gun School Forme d No Air-to- Air Action USN 2.4:1 USAF 2.4:1 USAF 2.4:1 USN 12:1
Where Do Synthetic Environments Fit In? LearningContent Learning Objectives Facts Simple Concepts Adaptive Procedures Abstract Concepts Remember Understand Apply Analyze Evaluate Create Simple Procedures (Framework courtesy of Anderson & Krathwohl, 2001) Learning environments we have known how to create since the 1960s with drill and practice. Learning environments we are learning how to create with authentic, situated, syntheti c environments.
Assessment
Level Description Evaluation (Did we do things right?) 1 Surveys Impressions and opinions? 2 Outcomes Were the objectives achieved? (Did we do the right things?) 3 Transfer Did the instruction improve workplace performance? 4 Benefits Is the enterprise more effective? Effectiveness: What Are We Looking For? Kirkpatrick’s Four Levels of Evaluation What About Costs, Cost/Effectiveness, Cost/Benefits Return on Investment, etc.?
What We Need: A Wish List • Agreement on the operational definition of an instructional game (!) • Examination of variables that are exclusive to games compared to robust competitors • Measures of pre-test and post test learning • Comparison with a control group receiving robust instruction intended to produce the same instructional objective • Achievement measured quantitatively and in the same way for both treatment and control groups
An Argument for Using Games in Learning We Know: • Games can be compulsively motivating and immersing, • People will voluntarily persist in playing games longer than they will engage in non-game learning, • If the game is instructionally relevant, this engagement will increase time on (learning) tasks, • Increased engaged time on learning tasks will yield increased learning. Therefore: • People will either learn more from games than from other instructional environments or learn the same amount at lower cost.
NB: Effect Sizes (Practical Significance) A descriptive (not inferential) statistic used to estimate the magnitude of an effect (e.g., experimental treatment). It may be calculated as: Cohen’s d = Mean Group 1 – Mean of Group 2 “Pooled” Standard Deviation d < 0.20 Negligible 0.20 to 0.39 Small 0.40 to 0.59 Moderate 0.60 to 0.79 Large d > 0.80 Very Large
People Do Learn from Games (1) For Instance (Perception): Who What Effect Size Boot, et al. 2008 Tracking speed 1.83 Visual STM 2.04 Task-switching 1.18 Castel, et al. 2005 Reaction time detect 0.88 Reaction time visual search 1.12 Chisholm, et al. 2010 Response speed 1.31 Speed despite distractor 1.78 Nouchi, et al. 2012 Exec functioning 1.62 & 1.31 Processing speed 1.12 & 1.34
People Do Learn from Games (2) For Instance (Attitude & Affect): Who What Effect Size Bachen, et al. 2012 Global Empathy 0.25 Interest in More Learning 0.47 Baldwin, et al. 2010 Self Esteem 0.42 Vos, et al. 2011 Perceived Competence 0.38 Interest 1.44 Effort 1.13
People Do Learn from Games (3) For Instance (Subject Matter): Who What Effect Size Gremmen & Potters, 1997 College Economics 0.76 Follow up 0.78 Kebritichi 2010 High School Math 0.39 Ravenscroft & Matheson, 2002 Grade 1&2 Math 1.70 Segers & Verhoeven 2005 Kindergarten Phonics – 1st grade follow-up 0.43 Suh, & Kim 2010 ESL Listening 0.53 ESL Speaking 0.23 Reading 0.31
Alex Wind’s Table http://www.alexanderpwind.com/all_games_studies.shtml NB: It’s /all_games_studies.shtml
• Randel et al. (1992) in 68 studies: – 56% of the studies found no difference between simulations, games, and conventional instruction, – 32% of the studies favored simulations/games – 12 of 14 studies found more interest in simulations/games than classroom instruction – Greater retention for games/simulations, even in some studies showing no immediate differences. – Increasing computer capacity yields more powerful sophisticated simulations and games Previous Research Review
• Israeli pilots playing Space Fortress II (modified to simulate cockpit complexity ) performed better in actual flight than non gamers (Gopher, Weil, & Bareket, 1994) • But assigning trainees to an off-the-shelf game (Apache Strike Force) had no transfer effects (Hart & Battiste, 1992) A Paradox
Conduct Cognitive Task Analysis to Identify the Cognitive Processes Required by the Game and Task(s) to be learned. – It is not just physical similarity between games and tasks that lead to learning and transfer. It is the overlap in cognitive processes engaged by both. Recommendation 1
Provide Guidance a) Pictorial Support. Reduces cognitive demands of game (cf The Multimedia Principle) b) Encourage Participants to Reflect About Correct Answers Not to reflect about wrong answers c) Use Unsupported Discovery Learning with Caution Kirschner, Sweller, & Clark (2006) cite “unambiguous evidence that minimal guidance during instruction is significantly less effective and efficient than guidance specifically designed to support the cognitive processing necessary for learning” (p. 76). Recommendation 2
Use First Person (“I” “You”)in Player–Game Dialogue. –Some findings suggest this practice will increase learning and transfer –Unknown whether using player names improves learning Recommendation 3
Use Animated Agents in Interactions with Players Some conflicting findings, especially for higher order learning Since agents have never been shown to reduce learning they might as well be used Use Human, Not Synthetic Voices Seems to increase learning and transfer Easy to find good speakers or actors to record speech Recommendations 4 & 5
–Maximize User Interactivity • Increases learner involvement in game (Flow) and post test scores –Reduce Cognitive Load • Sweller, Mayer, and others have shown importance of attending to cognitive load (e.g., present graphics with text) Recommendations 6 & 7
–Maximize Motivation • Games should have appropriate challenge, arouse curiosity, & include fantasy elements appealing to users (Yerkes-Dodson Inverted U; Vygotsky’s ZPD) • Assure that motivation is related to game success and avoid seductive details that reduce learning Recommendation 8
–Increase Pro-social Content & Reduce Aggression (per Gentile, 2009) • Games can increase pro-social attitudes • Games can increase aggression • Hence minimize aggressive game behavior and maximize pro-social content increased Recommendation 9
–Revise Games and Task Analyses • Off the shelf games that appear similar to task may not emphasize same cognitive processes – Integrate Games with Instructional and Task Objectives • Research indicates that games not integrated into curriculum/task lead to learning game strategy but not more general learning Recommendations 10 & 11
–Keep Abreast of Research Findings • Knowledge in area is exploding & recommendations may be revised in light of newer findings. –Use Teams to Develop Instructional Games. • No one person has skills/knowledge in cognitive task analysis, game design, computer expertise, familiarity with research findings Recommendations 12 & 13
…There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things... Niccolo Machiavelli (1469 – 1527), De Principatibus, 1513 About Change …
There is no sun without shadow, and it is essential to know the night … The struggle itself toward the heights is enough to fill a man’s heart. One must imagine Sisyphus happy. -- Albert Camus It’s all rock and roll to me. -- M. Jagger & K. Richards And Finally
Questions? Comments? Objections? Complaints? Thank you!
Questions? Comments? Objections? Complaints? Thank you!

More Related Content

J.D. Fletcher, Institute of Defense Analyses and Sigmond Tobias, SUNY

  • 1. On Play in the Garden of Empirical Analysis Serious Play Conference 20-22 August 2013 Redmond, WA J. D. Fletcher Institute for Defense Analyses Sigmund Tobias State University of New York at Albany fletcher@ida.org
  • 3. On Simulations and Games: Similarities Simulations Games Synthetic environments Synthetic environments Missions Missions Successful mission completion despite competition Successful mission completion despite competition Highly interactive Highly interactive Governed by rules of engagement Governed by rules of engagement Selective realism Selective realism
  • 4. On Simulations and Games: Differences Simulations Games Will sacrifice entertainment in favor of reality Will sacrifice reality in favor of entertainment Scenario/realistic tasks Storyline/fictional quests Emphasis on task completion Emphasis on competition and levels Not necessarily interactive Necessarily interactive Focus on (rule) accuracy/detailed Focus on (rule) clarity/stylized Not all simulations are games All games are simulations
  • 5. Synthetic Environments: Advantages • Safety • Economy • Visibility • Time Control
  • 6. 1909 Antoinette Simulator 1910 Saunders Teacher 1930 Link Trainer ?? Today
  • 7. Top Gun: An (Unintentional) Warfare Experiment Air-to-AirCombatExchangeRatio EnemyLosses/U.S.Losses 2:1 4:1 6:1 8:1 10:1 12:1 1965-1968 1969 1970-1973 USN Top Gun School Forme d No Air-to- Air Action USN 2.4:1 USAF 2.4:1 USAF 2.4:1 USN 12:1
  • 8. Where Do Synthetic Environments Fit In? LearningContent Learning Objectives Facts Simple Concepts Adaptive Procedures Abstract Concepts Remember Understand Apply Analyze Evaluate Create Simple Procedures (Framework courtesy of Anderson & Krathwohl, 2001) Learning environments we have known how to create since the 1960s with drill and practice. Learning environments we are learning how to create with authentic, situated, syntheti c environments.
  • 10. Level Description Evaluation (Did we do things right?) 1 Surveys Impressions and opinions? 2 Outcomes Were the objectives achieved? (Did we do the right things?) 3 Transfer Did the instruction improve workplace performance? 4 Benefits Is the enterprise more effective? Effectiveness: What Are We Looking For? Kirkpatrick’s Four Levels of Evaluation What About Costs, Cost/Effectiveness, Cost/Benefits Return on Investment, etc.?
  • 11. What We Need: A Wish List • Agreement on the operational definition of an instructional game (!) • Examination of variables that are exclusive to games compared to robust competitors • Measures of pre-test and post test learning • Comparison with a control group receiving robust instruction intended to produce the same instructional objective • Achievement measured quantitatively and in the same way for both treatment and control groups
  • 12. An Argument for Using Games in Learning We Know: • Games can be compulsively motivating and immersing, • People will voluntarily persist in playing games longer than they will engage in non-game learning, • If the game is instructionally relevant, this engagement will increase time on (learning) tasks, • Increased engaged time on learning tasks will yield increased learning. Therefore: • People will either learn more from games than from other instructional environments or learn the same amount at lower cost.
  • 13. NB: Effect Sizes (Practical Significance) A descriptive (not inferential) statistic used to estimate the magnitude of an effect (e.g., experimental treatment). It may be calculated as: Cohen’s d = Mean Group 1 – Mean of Group 2 “Pooled” Standard Deviation d < 0.20 Negligible 0.20 to 0.39 Small 0.40 to 0.59 Moderate 0.60 to 0.79 Large d > 0.80 Very Large
  • 14. People Do Learn from Games (1) For Instance (Perception): Who What Effect Size Boot, et al. 2008 Tracking speed 1.83 Visual STM 2.04 Task-switching 1.18 Castel, et al. 2005 Reaction time detect 0.88 Reaction time visual search 1.12 Chisholm, et al. 2010 Response speed 1.31 Speed despite distractor 1.78 Nouchi, et al. 2012 Exec functioning 1.62 & 1.31 Processing speed 1.12 & 1.34
  • 15. People Do Learn from Games (2) For Instance (Attitude & Affect): Who What Effect Size Bachen, et al. 2012 Global Empathy 0.25 Interest in More Learning 0.47 Baldwin, et al. 2010 Self Esteem 0.42 Vos, et al. 2011 Perceived Competence 0.38 Interest 1.44 Effort 1.13
  • 16. People Do Learn from Games (3) For Instance (Subject Matter): Who What Effect Size Gremmen & Potters, 1997 College Economics 0.76 Follow up 0.78 Kebritichi 2010 High School Math 0.39 Ravenscroft & Matheson, 2002 Grade 1&2 Math 1.70 Segers & Verhoeven 2005 Kindergarten Phonics – 1st grade follow-up 0.43 Suh, & Kim 2010 ESL Listening 0.53 ESL Speaking 0.23 Reading 0.31
  • 18. • Randel et al. (1992) in 68 studies: – 56% of the studies found no difference between simulations, games, and conventional instruction, – 32% of the studies favored simulations/games – 12 of 14 studies found more interest in simulations/games than classroom instruction – Greater retention for games/simulations, even in some studies showing no immediate differences. – Increasing computer capacity yields more powerful sophisticated simulations and games Previous Research Review
  • 19. • Israeli pilots playing Space Fortress II (modified to simulate cockpit complexity ) performed better in actual flight than non gamers (Gopher, Weil, & Bareket, 1994) • But assigning trainees to an off-the-shelf game (Apache Strike Force) had no transfer effects (Hart & Battiste, 1992) A Paradox
  • 20. Conduct Cognitive Task Analysis to Identify the Cognitive Processes Required by the Game and Task(s) to be learned. – It is not just physical similarity between games and tasks that lead to learning and transfer. It is the overlap in cognitive processes engaged by both. Recommendation 1
  • 21. Provide Guidance a) Pictorial Support. Reduces cognitive demands of game (cf The Multimedia Principle) b) Encourage Participants to Reflect About Correct Answers Not to reflect about wrong answers c) Use Unsupported Discovery Learning with Caution Kirschner, Sweller, & Clark (2006) cite “unambiguous evidence that minimal guidance during instruction is significantly less effective and efficient than guidance specifically designed to support the cognitive processing necessary for learning” (p. 76). Recommendation 2
  • 22. Use First Person (“I” “You”)in Player–Game Dialogue. –Some findings suggest this practice will increase learning and transfer –Unknown whether using player names improves learning Recommendation 3
  • 23. Use Animated Agents in Interactions with Players Some conflicting findings, especially for higher order learning Since agents have never been shown to reduce learning they might as well be used Use Human, Not Synthetic Voices Seems to increase learning and transfer Easy to find good speakers or actors to record speech Recommendations 4 & 5
  • 24. –Maximize User Interactivity • Increases learner involvement in game (Flow) and post test scores –Reduce Cognitive Load • Sweller, Mayer, and others have shown importance of attending to cognitive load (e.g., present graphics with text) Recommendations 6 & 7
  • 25. –Maximize Motivation • Games should have appropriate challenge, arouse curiosity, & include fantasy elements appealing to users (Yerkes-Dodson Inverted U; Vygotsky’s ZPD) • Assure that motivation is related to game success and avoid seductive details that reduce learning Recommendation 8
  • 26. –Increase Pro-social Content & Reduce Aggression (per Gentile, 2009) • Games can increase pro-social attitudes • Games can increase aggression • Hence minimize aggressive game behavior and maximize pro-social content increased Recommendation 9
  • 27. –Revise Games and Task Analyses • Off the shelf games that appear similar to task may not emphasize same cognitive processes – Integrate Games with Instructional and Task Objectives • Research indicates that games not integrated into curriculum/task lead to learning game strategy but not more general learning Recommendations 10 & 11
  • 28. –Keep Abreast of Research Findings • Knowledge in area is exploding & recommendations may be revised in light of newer findings. –Use Teams to Develop Instructional Games. • No one person has skills/knowledge in cognitive task analysis, game design, computer expertise, familiarity with research findings Recommendations 12 & 13
  • 29. …There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things... Niccolo Machiavelli (1469 – 1527), De Principatibus, 1513 About Change …
  • 30. There is no sun without shadow, and it is essential to know the night … The struggle itself toward the heights is enough to fill a man’s heart. One must imagine Sisyphus happy. -- Albert Camus It’s all rock and roll to me. -- M. Jagger & K. Richards And Finally