Working at the cutting edge of live performance, an emerging generation of artists is employing digital technologies to create distinctive forms of interactive, distributed, and often deeply subjective theatrical performance. The work of these artists is not only fundamentally transforming the experience of theater, it is also reshaping the nature of human interaction with computers. In this book, Steve Benford and Gabriella Giannachi offer a new theoretical framework for understanding these experiences--which they term mixed reality performances--and document a series of landmark performances and installations that mix the real and the virtual, live performance and interactivity. Benford and Giannachi draw on a number of works that have been developed at the University of Nottingham's Mixed Reality Laboratory, describing collaborations with artists (most notably the group Blast Theory) that have gradually evolved a distinctive interdisciplinary approach to combining practice with research. They offer detailed and extended accounts of these works from different perspectives, including interviews with the artists and Mixed Reality Laboratory researchers. The authors develop an overarching theory to guide the study and design of mixed reality performances based on the approach of interleaved trajectories through hybrid structures of space, time, interfaces, and roles. Combinations of canonical, participant, and historic trajectories show how such performances establish complex configurations of real and virtual, local and global, factual and fictional, and personal and social.
Cited By
- Sivertsen C, Salimbeni G, Løvlie A, Benford S and Zhu J Machine Learning Processes As Sources of Ambiguity: Insights from AI Art Proceedings of the CHI Conference on Human Factors in Computing Systems, (1-14)
- Yates M and Løvlie A (2023). Subtle Sound Design: Designing for Experience Blend in a Historic House Museum, Journal on Computing and Cultural Heritage , 17:1, (1-15), Online publication date: 29-Feb-2024.
- Benford S, Manninen K, Martindale S, Hazzard A, Martinez Avila J, Tennent P, Spence J, Castle-Green T, Brundell P, Barnard P and Darzentas D (2023). Infrastructures for Virtual Volunteering at Online Music Festivals, Proceedings of the ACM on Human-Computer Interaction, 7:CSCW1, (1-26), Online publication date: 14-Apr-2023.
- Greenhalgh C, Hazzard A, Benford S, Cliffe L and Kelly E (2022). Crafting Trajectories of Smart Phone Use at the Opera, ACM Transactions on Computer-Human Interaction, 29:6, (1-37), Online publication date: 31-Dec-2022.
- Ioannidis P and Løvlie A Designing for Design-after-Design in a Museum Installation Nordic Human-Computer Interaction Conference, (1-11)
- Wu Q, Li Y, She Y, Liu F, Luo Y and Yang X Bridging Virtual and Reality in Mobile Augmented Reality Applications to Promote Immersive Experience Proceedings of the 2022 International Conference on Advanced Visual Interfaces, (1-5)
- Rueben M, Horrocks M, Martinez J, Cormier M, LaLone N, Fraune M and Toups Dugas P “I See You!”: A Design Framework for Interface Cues about Agent Visual Perception from a Thematic Analysis of Videogames Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, (1-22)
- Rostami A and McMillan D The Normal Natural Troubles of Virtual Reality in Mixed-Reality Performances Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, (1-22)
- Benford S, Sundnes Løvlie A, Ryding K, Rajkowska P, Bodiaj E, Paris Darzentas D, Cameron H, Spence J, Egede J and Spanjevic B Sensitive Pictures Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, (1-16)
- Weijdom J Performative prototyping in collaborative mixed reality environments: an embodied design method for ideation and development in virtual reality. Proceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction, (1-13)
- Vayanou M, Katifori A and Ioannidis Y (2021). Perspective Sharing in Culture Group Games, Proceedings of the ACM on Human-Computer Interaction, 5:CHI PLAY, (1-24), Online publication date: 5-Oct-2021.
- Vishkaie R (2021). Interaction design inspirations from children's drawings in a mixed-reality museum exhibition, Interactions, 28:5, (50-55), Online publication date: 1-Sep-2021.
- Salovaara A, Nelimarkka M, Kuikkaniemi K and Jacucci G Augmenting the Performer–Audience Live Participation in Professional Event Productions Proceedings of the 24th International Academic Mindtrek Conference, (186-196)
- Yan S, Ge S, Wang J and Shen X Performing with Me: Enhancing Audience-Performer Interaction in An Immersive Virtual Play Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, (1-6)
- Alaoui S and Matos J RCO : Investigating Social and Technological Constraints through Interactive Dance Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (1-13)
- Tennent P, Höök K, Benford S, Tsaknaki V, Ståhl A, Dauden Roquet C, Windlin C, Sanches P, Marshall J, Li C, Martinez Avila J, Alfaras M, Umair M and Zhou F Articulating Soma Experiences using Trajectories Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (1-16)
- Alharthi S, LaLone N, Sharma H, Dolgov I and Toups Dugas P An Activity Theory Analysis of Search & Rescue Collective Sensemaking and Planning Practices Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (1-20)
- Tholander J, Rossitto C, Rostami A, Ishiguro Y, Miyaki T and Rekimoto J Design in Action: Unpacking the Artists’ Role in Performance-Led Research Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, (1-13)
- Bhardwaj A, Chauhan N and Shah R Synthesized 3D models with smartphone based MR to modify the PreBuilt environment Proceedings of the 2nd ACM International Conference on Multimedia in Asia, (1-3)
- Yan S, Yan X and Shen X Exploring Social Interactions for Live Performance in Virtual Reality SIGGRAPH Asia 2020 Posters, (1-2)
- Velt R, Benford S and Reeves S (2020). Translations and Boundaries in the Gap Between HCI Theory and Design Practice, ACM Transactions on Computer-Human Interaction, 27:4, (1-28), Online publication date: 25-Sep-2020.
- Benabdallah G Sybil Companion Publication of the 2020 ACM Designing Interactive Systems Conference, (275-279)
- Lange V, van Beuzekom M, Hansma M, Jeurens J, van den Oever W, Regterschot M, Treffers J, van Turnhout K, Ibrahim Sezen T, Iurgel I and Bakker R Blending into the White Box of the Art Museum Proceedings of the Halfway to the Future Symposium 2019, (1-10)
- Isbister K Toward ‘Suprahuman’ Technology Proceedings of the Halfway to the Future Symposium 2019, (1-4)
- Bluff A and Johnston A Devising Interactive Theatre Proceedings of the 2019 on Designing Interactive Systems Conference, (279-289)
- Taylor R, Williamson J, Spence J, Wood M, Hook J and Chen K Research in the Wild via Performance Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video, (279-285)
- Speicher M, Hall B and Nebeling M What is Mixed Reality? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, (1-15)
- Jacobs R, Schnädelbach H, Jäger N, Leal S, Shackford R, Benford S and Patel R The Performative Mirror Space Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, (1-14)
- Barnard D fanSHEN’s Looking for Love: A Case Study in How Theatrical and Performative Practices Inform Interactive Digital Narratives Interactive Storytelling, (401-407)
- Alavesa P, Pakanen M, Ojala T, Opoku Asare K, Ojala K, Lehto M and Kukka H (2018). Ludic Markers for Player-Player Observation in Location-Based Mobile Games, Simulation and Gaming, 49:6, (700-717), Online publication date: 1-Dec-2018.
- Rossitto C, Rostami A, Tholander J, McMillan D, Barkhuus L, Fischione C and Turchet L Musicians' initial encounters with a smart guitar Proceedings of the 10th Nordic Conference on Human-Computer Interaction, (13-24)
- Rostami A, Rossitto C and Waern A Frictional Realities Proceedings of the 2018 ACM International Conference on Interactive Experiences for TV and Online Video, (15-27)
- Benford S, Koleva B, Preston W, Angus A, Thorn E and Glover K Customizing Hybrid Products Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, (1-12)
- Barati B, Giaccardi E and Karana E The Making of Performativity in Designing [with] Smart Material Composites Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, (1-11)
- Moulder V, Boschman L, Wakkary R, Neustaedter C and Kobayashi H HCI Interventions for Science Communication Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, (1-9)
- Alharthi S, Sharma H, Sunka S, Dolgov I and Toups Dugas P Designing Future Disaster Response Team Wearables from a Grounding in Practice Proceedings of the Technology, Mind, and Society, (1-6)
- Rostami A, Bexell E and Stanisic S The Shared Individual Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, (511-516)
- Patel R, Schnädelbach H and Koleva B Come and Play Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, (376-385)
- Rostami A, Rossitto C, McMillan D, Spence J, Taylor R, Hook J, Williamson J and Barkhuus L (2017). Glimpses of the future, Interactions, 25:1, (46-51), Online publication date: 21-Dec-2017.
- Yan S, Ding G, Li H, Sun N, Guan Z, Wu Y, Zhang L and Huang T (2017). Exploring Audience Response in Performing Arts with a Brain-Adaptive Digital Performance System, ACM Transactions on Interactive Intelligent Systems, 7:4, (1-28), Online publication date: 19-Dec-2017.
- Sharma H, Alharthi S, Dolgov I and Toups Dugas P A Framework Supporting Selecting Space to Make Place in Spatial Mixed Reality Play Proceedings of the Annual Symposium on Computer-Human Interaction in Play, (83-100)
- Shakeri H, Singhal S, Pan R, Neustaedter C and Tang A Escaping Together Proceedings of the Annual Symposium on Computer-Human Interaction in Play, (115-128)
- Chamberlain A, Bødker M, Hazzard A, McGookin D, De Roure D, Willcox P and Papangelis K (2017). Audio Technology and Mobile Human Computer Interaction, International Journal of Mobile Human Computer Interaction, 9:4, (25-40), Online publication date: 1-Oct-2017.
- Kerne A, Lupfer N, Linder R, Qu Y, Valdez A, Jain A, Keith K, Carrasco M, Vanegas J and Billingsley A Strategies of Free-Form Web Curation Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition, (380-392)
- Pan R, Lo H and Neustaedter C Collaboration, Awareness, and Communication in Real-Life Escape Rooms Proceedings of the 2017 Conference on Designing Interactive Systems, (1353-1364)
- Velt R, Benford S and Reeves S A Survey of the Trajectories Conceptual Framework Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (2091-2105)
- Rossitto C, Normark M and Barkhuus L Interactive Performance as a Means of Civic Dialogue Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (4850-4862)
- Tekin B and Reeves S Ways of Spectating Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (1558-1570)
- Taylor R, Spence J, Walker B, Nissen B and Wright P Performing Research Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (4825-4837)
- Rostami A Performing with Technology Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction, (721-723)
- Ausmeier N and Celik T Stencil Cutouts for Virtual Reality Inputs Proceedings of the 9th International Conference on Computer and Automation Engineering, (31-35)
- Romualdo S Going undercover Proceedings of the 20th International Academic Mindtrek Conference, (312-321)
- Kors M, Ferri G, van der Spek E, Ketel C and Schouten B A Breathtaking Journey. On the Design of an Empathy-Arousing Mixed-Reality Game Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, (91-104)
- Sharma H, Toups Dugas P, Dolgov I, Kerne A and Jain A Evaluating Display Modalities Using a Mixed Reality Game Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, (65-77)
- Isbister K (2016). Connecting through play, Interactions, 23:4, (26-33), Online publication date: 28-Jun-2016.
- Flint T, Turner P and Banach A The Ensoulment of Virtual Space Minecraft as a Tool for Engaging With a Sculpture Park Proceedings of the The 15th International Conference on Interaction Design and Children, (680-683)
- Jacobs R, Benford S, Luger E and Howarth C The Prediction Machine Proceedings of the 2016 ACM Conference on Designing Interactive Systems, (497-508)
- Benford S, Hazzard A, Chamberlain A, Glover K, Greenhalgh C, Xu L, Hoare M and Darzentas D Accountable Artefacts Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, (1163-1175)
- Webb A, Wang C, Kerne A and Cesar P Distributed Liveness Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, (432-437)
- Sharma H, Toups Dugas P, Jain A and Kerne A Designing to Split Attention in a Mixed Reality Game Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, (691-696)
- Sosa A, Stanton R, Perez S, Keyes-Garcia C, Gonzalez S and Toups Dugas P Imperfect Robot Control in a Mixed Reality Game to Teach Hybrid Human-Robot Team Coordination Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, (697-702)
- Yule D, MacKay B and Reilly D Operation Citadel Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play, (285-294)
- Benford S, Greenhalgh C, Anderson B, Jacobs R, Golembewski M, Jirotka M, Stahl B, Timmermans J, Giannachi G, Adams M, Farr J, Tandavanitj N and Jennings K (2015). The Ethical Implications of HCI's Turn to the Cultural, ACM Transactions on Computer-Human Interaction, 22:5, (1-37), Online publication date: 8-Sep-2015.
- Taylor R, Bowers J, Nissen B, Wood G, Chaudhry Q, Wright P, Bruce L, Glynn S, Mallinson H and Bearpark R Making Magic Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, (313-322)
- Jacobs R, Benford S and Luger E Behind The Scenes at HCI's Turn to the Arts Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, (567-578)
- Darzentas D, Brown M, Flintham M and Benford S The Data Driven Lives of Wargaming Miniatures Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, (2427-2436)
- JafariNaimi N and Meyers E Collective Intelligence or Group Think? Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, (1872-1881)
- Lundgren S, Fischer J, Reeves S and Torgersson O Designing Mobile Experiences for Collocated Interaction Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, (496-507)
- Kan A, Ploderer B and Gibbs M Mixed reality stories Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design, (115-118)
- Cerratto-Pargman T, Rossitto C and Barkhuus L Understanding audience participation in an interactive theater performance Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, (608-617)
- Bowers J, Taylor R, Hook J, Freeman D, Bramley C and Newell C HCI Proceedings of the 2014 companion publication on Designing interactive systems, (203-206)
- Freeth B, Bowers J and Hogg B Musical meshworks Proceedings of the 2014 conference on Designing interactive systems, (219-228)
- Kan A, Gibbs M and Ploderer B Being chased by zombies! Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, (207-216)
- van Troyer A Enhancing site-specific theatre experience with remote partners in sleep no more Proceedings of the 2013 ACM international workshop on Immersive media experiences, (17-20)
- Beira J, Carvalho R and Kox S Mixed reality immersive design Proceedings of the 2013 ACM international workshop on Immersive media experiences, (45-50)
- Benford S, Greenhalgh C, Giannachi G, Walker B, Marshall J and Rodden T (2013). Uncomfortable user experience, Communications of the ACM, 56:9, (66-73), Online publication date: 1-Sep-2013.
- Benford S, Greenhalgh C, Crabtree A, Flintham M, Walker B, Marshall J, Koleva B, Rennick Egglestone S, Giannachi G, Adams M, Tandavanitj N and Row Farr J (2013). Performance-Led Research in the Wild, ACM Transactions on Computer-Human Interaction, 20:3, (1-22), Online publication date: 1-Jul-2013.
- Bonner J and Peebles D Rhetorical considerations for innovative approaches to performance and audience engagement Proceedings of the 2013 Inputs-Outputs Conference: An Interdisciplinary Conference on Engagement in HCI and Performance, (1-4)
- Fosh L, Benford S, Reeves S, Koleva B and Brundell P see me, feel me, touch me, hear me Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (149-158)
- Taylor R, Schofield G, Hook J, Ladha K, Bowers J and Wright P Crafting interactive systems CHI '13 Extended Abstracts on Human Factors in Computing Systems, (3223-3226)
- Edmonds E, Benford S, Bilda Z, Fantauzzacoffin J, Malina R and Vinet H Digital arts CHI '13 Extended Abstracts on Human Factors in Computing Systems, (2439-2446)
- Spence J, Frohlich D and Andrews S Performative experience design CHI '13 Extended Abstracts on Human Factors in Computing Systems, (2049-2058)
- Fatah gen Schieck A and Moutinho A ArCHI Proceeding of the 16th International Academic MindTrek Conference, (39-45)
- Höök K and Löwgren J (2012). Strong concepts, ACM Transactions on Computer-Human Interaction, 19:3, (1-18), Online publication date: 1-Oct-2012.
- Benford S, Greenhalgh C, Giannachi G, Walker B, Marshall J and Rodden T Uncomfortable interactions Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (2005-2014)
- Benford S and Giannachi G (2012). Interaction as performance, Interactions, 19:3, (38-43), Online publication date: 1-May-2012.
- Massimi M, Odom W, Banks R and Kirk D Matters of life and death Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (987-996)
Index Terms
- Performing Mixed Reality
Recommendations
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Compelling experiences in mixed reality interactive storytelling
ACE '04: Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technologyEntertainment systems promise to be a significant application for Mixed Reality. Recently, a growing number of Mixed Reality applications have included interaction with virtual actors and storytelling. However, AI-based Interactive Storytelling ...
Multi-user mixed reality system `Gulliver's World': a case study on collaborative edutainment at the intersection of material and virtual worlds
This case study is about `Gulliver's World', a multi-user mixed reality environment that functions simultaneously as interactive edutainment platform and learning environment as well as flexible infrastructure for the expansion of mixed reality ...