Keywords

1 Introduction

Samuel Beckett was one of the great innovators of technology in theater. He created works for all media, including radio, television, film, as well as experimenting with technological innovations on the live stage. He was interested in the possibilities that new technologies open for creative expression. If Beckett had continued to work into this century, he would likely be curious about new digital technologies that did not exist during the analog age. A major artistic mission of the practice-based research trilogy underpinning this paper was to extend the idea of Beckett’s theater into these new digital realms, affording audiences and artists extended capabilities in networking, immersion, interactivity, telepresence, and the new medium of volumetric video (VV)Footnote 1 for mixed reality (MR). As new technologies emerge, they compel new experiments in art-making and storytelling, and revisiting Beckett’s texts asks what these plays will mean in the 21st century. As such, the epistemological significance of remaking Beckett’s Play for virtual reality (VR) and augmented reality (AR) consists of a will to transmit his texts to forthcoming generations of art-going publics, who will increasingly access content via MR technologies, thereby responding to a cultural need for his works to be intergenerationally reactivated.

Virtual Play and Augmented Play are the second and third parts of a three-year practice-as-research trilogy, wherein Samuel Beckett’s groundbreaking theatrical text, entitled Play (1963), was reimagined and reinterpreted for digital culture, as 1) a webcast (Intermedial Play), 2) a VR drama (Virtual Play), and 3) an AR drama (Augmented Play). This project reinterprets Beckett’s classic modernist text in a way that is engaging for 21st-century audiences, by capturing actors using cutting-edge digital media recording techniques and then visualizing their corresponding characters using advanced VR and AR head-mounted displays (HMDs). The actors were recorded using a VV technique [23] which involved capturing action simultaneously on multiple video cameras “strategically placed, in an arc of about 150\(^{\circ }\), in a compromise between scene coverage and image overlap” [15]. Actors were recorded against a green-screen backdrop, which simplified segmentation (or chroma-keying) processes at the post-production stage.

For traditional video, viewers can only watch the captured content as framed on a 2D display in a way that was decided by the director and the cinematographer; for traditional theater and performance, there is the visceral and contingent experience of being present in the space of the live performance. MR technologies are distinct from both theater and film, but exhibit characteristics that overlap with existing modalities of content creation and consumption. MR technologies enable viewers to potentially interact with and explore content in ways that were not possible via traditional performance modes. Developments in VV capture techniques and MR display technologies create a mediatized content paradigm that can be, on the one hand, utilized by the content creators, and on the other, consumed openly by the audiences. However, this raises questions concerning the intent and realization for the dramatic arts, where one can discuss the relationship between the narrative, interaction, and perception.

This paper analyzes the results of a series of subject-matter expert (SME) interviews, which include the opinions of academics, researchers and practicing artists who specialize in film, theater, and performance studies. The goal was to reflect on the relationship between emergent MR technologies and discuss opportunities and challenges for the domain. Specifically, we identify topics of novel, immersive, interactive imaging devices, cutting-edge live-action capture techniques, and the potential that they hold for creative artists as a means of expression, and for audiences as a means of engaging content. The qualitative data analysis of the SME interviews showed that specific improvements can be made to our MR experiences that can potentially benefit the broader domain.

2 Background and Related Work

The philosophies of mixed realities have been of interest and influence for creative technologists for many years. However, it is only recently that advanced AR and VR technology has become more readily accessible for broader research, commercial content creation, and widespread consumption. New developments in the area now focus on the creation of accessible content and the techniques MR designers and artists can apply in communicating their creative ideas.

2.1 Mixed Reality and Volumetric Video

The ubiquitous nature of digital technology in the everyday lives of many means that human-computer interactions take place more frequently, where “collective human perception meets the machines’ view of pervasive computing” [20]. The idea that the digital world is constantly intervening in the real via technology is one of the driving philosophies of mixed reality, where technology channels data from the physical world into the virtual and vice versa. MR can be defined as the merging of the real world with the virtual, where both physical and digital objects exist in real time [13]. Therefore, MR can refer to a continuum that spans between real environments and virtual environments, encompassing both augmented reality and virtual reality. New developments in MR have presented users with options of engaging content in VR, where the user is placed within an immersive virtual environment (IVE), and AR, where virtual objects are superimposed on to the real-world view, using HMDs and mobile ‘window on the world’ technologies that employ spatial registers to anchor digital objects to the real world. Often, content is created for these platforms via proprietary and open-source cross-platform game engine technologies.

Capitalizing on developments made in 3D capture and reconstruction, VV technology enables content creators to reconstruct live action in 3D. This is done in dedicated studios where multiple cameras are placed around the edges of the studio looking inwards, although the setup may differ depending on limitations and requirements, using cameras ranging from 12 [19] to 106 [6]. In general, the 3D object in the scene is segmented via either chroma-keying [19], depth-keying [22], or their combination [6]. For Virtual Play and Augmented Play, the method proposed by Pagés et al. [19] was applied due to the low number of cameras required, which would be more feasible for many content creators.

2.2 Mixed Reality and Volumetric Video in Creative Practice

Following the emergence of consumer VR, one of the early innovators was journalism. This trend was led by the New York Times, “which distributed over a million cardboard viewers to its print subscribers and created a high-end, VR-specific smartphone application to distribute Times-related VR experiences” [2, p. 204]. Many of these experiences were filmed using 360 video technology, which has a fixed viewpoint and only affords three degrees of freedom (3DoF).

One of the pioneers of combining VV in VR and journalism was Nonny de la Peña, who created several VR experiences that allowed her audiences to immerse themselves in realistic six degrees of freedom (6DoF) VR. De la Peña is a “trained journalist who was drawn to VR for its immediacy and empathy-encouraging qualities” [2, p. 209]. There are several other examples of content creators who are attempting to write original fictional narratives for VV VR storytelling and to define a grammar thereof; see for example, Awake: Episode One [24], where the narrative drops the viewer into a series of scenes that are ambiguously situated somewhere between the protagonist’s reality and unconscious - memory, hallucination, or both. The dominant narrative strategy for such experiences consists of placing the spectator into the middle of a scene, and some action ensues; however, at no point is the audience directly involved in the dialogue or action or addressed by the characters. The role of the viewer is reduced back into the same voyeuristic role of traditional media practice; there is no attempt to elicit the new interactive potentialities for narrative in VR. As of yet, there is very little investigation into combining interactive narrative techniques with VV, and affording the viewer significant agency in the unfolding of the narrative event. However, with the emergence of new MR and VV technologies, users are now permitted 6DoF, which affords a more naturalistic sense of presence.

As a new form of visual media, VV has started to be used more frequently in MR applications, including: remote communication and collaboration schemes [18], live-action reenactments for educational museum-guide applications for cultural heritage [17], and an AR version of the aforementioned Play by Samuel Beckett [16]. Play lends itself to MR because of Beckett’s deep engagement of the notion of play. In the original script, the sequence of the actors speaking is determined by a moving spotlight, which Beckett calls the “inquisitor” [3, p. 318]; they speak when the light is on them and fall silent when the light is off. Play is a game of interaction between the light operator and the actor, mediated by the lighting technology of the time [15]. In the presented MR versions the spotlight is aligned with the user’s gaze, and they are afforded the power to activate the characters into speaking. Thus, the user embodies the interrogator, a privilege originally withheld for the director or light operator. By directly involving the user in the cause-and-effect dynamics of shining a light upon the actors, the reinterpretation attempts to elicit the interactive opportunities afforded by the MR medium. Thus, Play offers a glimpse of the new opportunities for MR to break away from the predominant three-act narrative, as well as the fly-on-the-wall audience paradigm.

2.3 Narrative and Interactivity

The will to transpose old media processes and idiosyncrasies onto digital media is a tendency that formulates the theory of “remediation”, which documents and theorizes the prolongation of old media characteristics in new media [4]. Historically speaking, this is evident in the progression of live-action storytelling from theater to film, where 1) directors initially filmed using the proscenium format, and 2) actors made broad gesticulation as if communicating with audiences in the back row of an auditorium, rather than a camera positioned close by. However, as the film medium became more established, a grammar gradually developed that allowed filmmakers and audiences to understand that film was not synonymous with theater, and that the working processes and finished artifacts were different. The theater-to-film transition shows us that it takes time for creative practitioners to exploit the potentialities of new interdisciplinary art forms, and those potentialities are subject to the indeterminacy of technological developments. This raises the fundamental question concerning the preference of MR audiences: is it better to observe narrative as a fly-on-the-wall, choosing a viewpoint without having any effect on the outcome of the story, or to embody a character who participates in the plot and impacts the narrative? The former paradigm, which appeals to empathy and emotions through a tightly regimented and efficient system of narrative disclosure, is the domain of Hollywood; the latter, concerning exploration, immersion, deep engagement, and repetition, is that of gaming.

The reinterpretation of Beckett’s Play progresses a new “storyworlding” grammar [2, p. 225]. Beckett’s original script had to be linear for it to work in the proscenium format; however, in some sort of prophetic act anticipating the destiny of storytelling, Beckett designed his script as circular and, therefore, potentially infinite. Furthermore, in a note entitled “Repeat”, he says: “The repeat may be an exact replica of the first statement or it may present an element of variation” [3, p. 320], and in his own adaptation for radio, he allowed the actors to speak in a random order while ensuring that the sequential integrity of individual monologues was maintained [8, pp. 125–154]. In both Augmented Play and Virtual Play, this concept was pushed to its limits by putting control of the randomness in the hands of the audience. To explore the establishment of a storyworlding grammar, SME interviews were proposed to raise, present, and explore multiple issues of current interest and contention at the intersection of MR with film, theater, and performance.

In traditional productions, the stories being told are compelling because they often furtively reflect the audiences’ previously lived experiences. AR and VR can be used in this context, because MR narratives are often delivered on a first-person experiential basis, and the innately immersive nature of the technology enhances these narrative experiences even further. However, this alone does not create engaging content, just as not everything that is performed on stage or captured for film is storytelling. Therefore, performance-audience resonance is required to impart a sense of inclusion within a narrative that is laid out explicitly for the role of the viewer. Although MR technology has seen many new developments in recent years, past theories of VR and storytelling that closely tie the roles of imagination, interaction, and immersion still hold today [5]. For example, to understand a story via MR the viewer must use their imagination to fill in any gaps perceived in the presented materials. A similar phenomenon was observed, revealing that the way in which an actors’ representations of an imaginary narrative was judged by an audience was closely connected with the subjective attribution of the origin of their movements, attribution based purely upon the values, beliefs, and memories of the observer [10]. Creators of cultural content can, therefore, never remove subjective interpretation.

When multimodal sensory narratives are experienced in MR, the perceptions of the audience require some grounding in the real world; this is a central principle of Gestalt psychology – pattern recognition and previous experiences combine to engage the imagination [12]. When creating content for MR, new experiences must be created that exhibit some commonalities with the audience’s previous experiences. Therefore, we sought to find which experiential practices had the most impact, and what was useful for achieving artistic goals and intentions. To test these ideas, data was collected and analyzed to learn and develop a heuristic for creating MR interpretations of existing works, exploring the technicalities of the processes undertaken to stage performances via AR and VR platforms, and improving experiences of MR practitioners and audiences in the future. The main goal of the presented work was to learn from previous practices, to discover what worked and what did not, and to use these data to inform future creative projects in the MR creative sector in an ongoing, iterative design process.

In terms of cultural heritage and human-computer interaction, this raises some questions: How aware are domain experts of the disruptive impact of MR on artistic practice and audience engagement? How can we establish a storyworlding grammar that allows makers to more easily create engaging VV-based MR content, and audiences to more enthusiastically engage with it? How can we make MR content that aligns better with the time-honored paradigm of storytelling as a communal (non-solipsistic) experience?

3 Interview Methodology

Data collection took place in the Republic of Ireland during December 2019 and January 2020. Participants were recruited via email, with potential respondents identified via a snowball method to identify an appropriately qualified peer user group from the UK and Ireland. Sessions of one hour were conducted face-to-face at locations and times that suited the individual. The session began with an introduction to the research, then a fully labeled 5-point Likert scale questionnaire was used to gather demographic and user-type identifier data. Following this, each participant experienced five minutes of both Virtual Play and Augmented Play to familiarize themselves with the material and platforms before being interviewed. For each session, a Dell Alienware Area 51 laptop (Intel i9k-9900K CPU, NVIDIA GeForce RTX 2080, and 64 GB DDR4 Memory) and an HTC Vive (a VR headset developed by HTC and Valve Corporation) was set up for Virtual Play, and a Magic Leap One was used for Augmented Play. The order of presentation was counterbalanced to control for ordering effects, and visible observation (GoPro Hero7) was started at this time. Participants were recorded throughout and observational notes logged.

An in-depth interview on topics relating to the use of MR in creative practice was then conducted. The interview followed a semi-structured design for continuity and repeatability, and employed an interview-laddering technique [9] to systematically explore the same core themes with each participant, whilst allowing them to elaborate on their chosen topics of interest. The interviews probed: the SMEs’ previous knowledge and experience of state-of-the-art MR technology; the use of MR technology in their respective domains; current user requirements for technology applied in creative practice; the challenges of using new MR technology; and the receptiveness of the domain to MR technology. The interview section of the experiment took on average 20 min (M = 21:30; SD = 5:29). Finally, participants were debriefed and given the opportunity to ask further questions.

The interviews were transcribed and coded using a grounded theory approach. The data were first categorized into the core rungs of our laddering technique and first-level core categories were then derived from topics of interest raised in the interviews. In this way, our interview analysis was structured to provide a focused view that matched our initial hypothesis.

4 Results and Discussion

The study consisted of 13 interviews with experienced practitioners and scholars of film, theater, performance, and literature, 6 male and 7 female. The average age of the cohort was 46.92 (SD = 14.42), and the sum total of professional experience was 221 years, with a mean number of years’ experience of 18.42 (SD = 13.28). The current interviewee occupation profiles consisted of academics (n = 9), artistic directors (n = 3), and performer (n = 1). Their work experiences included titles of Professor, Head of School, Postdoctoral Research Fellow, industry expert, and theater director. All participants had undertaken important roles in areas of: Modern Irish Literature or Beckett Studies; artistic, pedagogic, and administrative leadership; movement practice in performance; feminism, gender, and women in performance; representations of 3D space in literature; and history and theory of theater, cinema, and digital media.

As a manipulation check, participants self-identified their aptitude for new or novel technologies as being “Fair” to “Good” (M =  3.65, SD = 0.82). To further classify the participant pool as experts within their respective domain, a user-cube [14] was constructed, see Fig. 1. The participants rated their knowledge and experiences of MR technology as “Moderately Familiar” (M = 2.92, SD = 1.21), and their expertise in relation to the study of avant-garde, modern literature, and modernism, as “Above Average” (M = 4.08, SD = 0.80). In summary, the cohort self-identified as being sufficiently informed, technically competent users, who could be considered as having some familiarity with MR in practice, and above-average expertise within their domain.

Fig. 1.
figure 1

User-cube identifying participant user-types (blue line representing the linear average and numbers representing the number of participants). (Color figure online)

Full size image

To understand our cohort’s opinions on MR in creative practice, interview data were analyzed applying inductive reasoning to generate codes relating to our core themes, see Fig. 2. The collected data were reviewed and repeated ideas and topics were extracted and categorized. This paper focuses on discussions around practitioners as content creators, and audiences as content consumers.

4.1 Previous Knowledge and Experiences of MR

A wide range of MR works had been previously developed and experienced by the cohort. This provided insight into how their involvement with MR influenced their opinions on future contexts of use and their expectations of the technology in practice. They commented on the genealogical reach of MR within the domain by referencing early AR and 3D performance experiments in Beckett’s Waiting for Godot by George Coates Performance Works. Other early references were of an academic nature, such as the Annual Conference of Cyberspace ca. 1994/95, a period when the term “cyberspace” was first coined; as one SME explained, “There were a lot of artists working, at the time, using early virtual reality technologies”.

The cohort’s experiences of MR provoked comments on innovative and exploratory forms of artistic practice; for example, they supposed that MR was highly influential in helping artists to think differently about how new fiction is structured. Equally, it was perceived as a niche, experimental, and a counter-mainstream form of content creation and consumption, struggling to be integrated into conventional performance spaces: “It’s its own thing. Like gaming, it’s completely different”. Participants speculated that MR would afford artists the ability to provide new types of imaginary worlds, reproduce specific historical scenarios, apply modern perspectives (e.g. feminism or postcolonialism) to narratives and allow viewers to reinterpret these events, and create new empathy-driven experiences that allow audiences to step into the shoes of another person and gain an understanding of their perspective.

Bridging both practice and audience perspectives, our participants were familiar with earlier performative VR works that concerned philosophies of “cyberspace” and the associated anxieties surrounding the nature of being. It was generally believed that these anxieties were somewhat quelled today: “We’re now living in the virtual”; citing Videodrome, one participant jested, “Your reality is already half... hallucination” [7]. Historically, it was believed that scholars and practitioners were somewhat uncomfortable using new MR technologies, but in the current epoch people live much more of their lives in the virtual domain – “Whether it be in social media or banking, people aren’t quite alarmed by that... it’s just another extension of our existence”. As such, it was felt that MR is now more readily accepted for creating layers of reality within artistic productions, and could be a disruptive technology for contemporary performances. Generally, it was believed that the new “theater” needs to explore digital technology to reconfigure the stage as a societal laboratory, where creatives can practice new techniques and better understand what it means to be in the digital space, and understand “...how stories or identities are recomposed in that space”.

Fig. 2.
figure 2

Interview analysis of MR in film, theater, or performance practices.

Full size image

Participants were aware of the growing interest in AR and VR. However, it was reported that their access to MR experiences was primarily obtained through organized events in institutes of higher education, as a pedagogical tool, and cultural heritage sites or art museums/galleries, where it was used to represent historical narratives as installation art or bespoke applications. Notable public performances included works in which the mise-en-scène of virtual landscapes and dance choreographies played on new interactions between the reality of performance and the virtuality of digital worlds. It was generally agreed that these types of experiences were immersive, conducive to a feeling of presence, and offered a new viewing platform beyond the conventional formats. Participants also believed VR was better suited to installation art, where immersing a viewer in a virtual space could facilitate surreal/avant-garde theater and new perspectives on performance.

Participants highlighted that MR technology would have to develop significantly to enable a collective audience to experience a singular production. Although technically feasible with location-based VR [15], this would be expensive/risky in terms of the established auditorium-based business model. The repeated failures of 3D cinema – a medium introduced, revised, and revisited since the 1960s – was given as an example of this phenomenon. One participant historicized the importance of technical evolution and shared experiences:

“When sound was introduced to the cinema, within a year every single cinema had to have sound in it, and you couldn’t make a film that was silent any longer. It hasn’t been like that with 3D, which is basically the precursor to what you are doing [with MR].” – (PW67)

Shared CAVE experiences, where communal IVEs are simultaneously occupied by the performers and audience, were also discussed. These types of environments highlighted the power of creating new, shared virtual landscapes, where the experience was unimpeded by the physicality of the real world. These practices provided our SMEs with memorable, positive experiences that motivated them to explore MR in their respective domains, as one participant explained: “I felt like it had elevated beyond being an experiment in the art form; it was a very beautiful thing”. While MR was described as “[Just] another technology to be employed... as simple as using lighting on the stage”, it was also asserted that theater has always been virtual, and audiences are comfortable with that. Although a true disconnect from the physical world is not possible, the SMEs expected to be able to realize their imaginings and allow audiences to explore something new that would ordinarily be inaccessible. Moreover, the space between audiences and performers would also be transformed “By interactivity... between the user, performer, or creator”, facilitating creations that are neither classical film nor stage performance. As per any disruptive technology, or a new mode of transposing concepts, “From creative moment to audience moment...”, the discipline must expand its knowledge and expertise in a new way.

Some SMEs were particularly critical of current explorations of 360 video in stage production, while others described it as an interesting way of capturing a fundamentally ephemeral art form, one that is notoriously difficult to capture successfully on conventional video. One participant declared: “I would look at a lot of theater on video in my previous job and it was always horrendous; even the most wonderful shows were a torture to sit through”. 360 recordings are attempting to create a sense of seeing the show live, and audiences are afforded the added advantage of being on stage among the performers. While it was predicted that original work made for 360 film would be genuinely impactful and interesting to the domain, it was also felt that 360 technology was lacking the communal experience of a classical performance, because each spectator is in their own world: “One of the great joys of theater and performance is that communal experience of being in an audience with other beating hearts”.

4.2 Using MR in Current Practice

A major domain-specific, practice-related issue was the problem of novelty, and the uncertainty this creates in content creation and audience consumption. It was agreed that MR created a fundamentally different author–audience paradigm; for example, SMEs acknowledged: “There will be a period of audience learning” and “More of these problems will be solved as more and more people start to use it”. Some were hesitant to align MR technology with existing performance formats, because the technology is neither film nor theater, which “...raises the question as to whether it is a completely new medium”. Thus, MR introduces a new set of conditions; “It’s an opportunity all by itself”. In terms of the commercializing, “You would have to imagine something quite different... You are a long way off the three-act narrative”. However, others were accepting of MR as an inevitable evolution of the domain, while remaining ambivalent about the benefits: “You can gain a lot... by allowing the viewer more agency; but you can lose a lot, because you still have the same problems [as film]: it’s a one-off performance.” Therefore, everything hinges on a quality take in the film studio as the actor is not permitted to refine their performance, resurrecting the live-versus-recorded debate. A Beckett director suggested that some spontaneous presence was lost, because the actors acted individually in the capture studio: “It really is between the three of them that the play is formed as a trialogue”. However, another SME highlighted how the technological idiosyncrasies open new opportunities to challenge the audience engagement paradigm, and how to frame the art: “The future of VR is in content that’s generated specifically for it and doesn’t try to be either, and that acknowledges new opportunities that the new medium presents”. SMEs generally acknowledged the variety and freshness that the interactivity provides to the audience: “Theoretically, it’s the same performance, but you can change your method of engaging with it”.

The film contingent problematized the difficulties that the MR viewer paradigm creates for storytelling, compared with the “mainstream”. Controlling the viewer’s gaze through framing and editing is crucial for narrative progression in complex or fast-moving plots, so “The question of gaze and where the user is directing their attention becomes a really important consideration”. Audiences are not used to having the freedom to look around. The unedited choose-where-you-look viewer paradigm complicates the traditional narrative model, highlighting the need to establish a new grammar: “Because of the open-endedness... there needs to be ways to guide people through, so if you don’t see everything you still understand what’s happening”. They concluded that slow, experimental, non-narrative stories are “Better suited to VR than fast-paced plots”, because they are conducive to taking time, exploring and observing the surroundings. These observations accord with the aforementioned assertion of MR’s suitability for installation and video art, where one could, for example, “Get to be in the shoes of someone... and see what it’s like, for a few minutes”.

The SMEs were excited by “...the possibility for directors to have real bodies... and virtual bodies together”. They enthusiastically articulated creative ideas for experimenting with MR technologies along the lines of telepresence, interconnectivity, and the disembodied image. They suggested the technology could be useful for “Reanimating dead actors”, expediting or supporting rehearsal processes, or solving “The problem of all performers not being available at the same time”. These ideas support some of the anticipated future uses of the technology that are enthusiastically advocated by computer science researchers.

Practicing SMEs questioned the reliability of MR technology in performance: “You hope that the technology is going to work at the moment of the performance... I’m afraid of technological failure”. Some SMEs advocated the dissemination benefits: “[MR] could help theater to boost its potential to immerse audiences in various sorts of realities”, and help to solve the problem of universal access, for “[The] people who can’t travel, or are not able-bodied”. However, the converse view was also cautioned, as viewing theater in isolation could cause “A potential loss of communal experience”. Generally, it was acknowledged that under the weight of new technologies “[The] performance changes... from whatever classical times have proceeded. It’s not necessarily bad, it’s just different”. Where traditional works hinge on a passive empathy paradigm, MR offers a new active mode of empathizing, by putting the viewer in the place of the ‘other’ – potentially a group experiencing racial, gender, or religious oppression. MR was seen as “An opportunity to help build bridges and bring people together... in a world that is getting fragmented”. SMEs also identified the potential to use the technology in “Site-specific theater and very immersive work, where it’s often one-on-one between you and a performer”. Specifically, performances by Anú Productions [1] and Punch Drunk [21] were described as championing storytelling paradigms ideally suited to these media, with one SME stating: “I think there are interesting opportunities there, where it creates another dimension within an existing world.”

4.3 User Requirements

Our SMEs expected that current and next-generation practitioners and audiences will gain the most from MR, and it will play a crucial role in keeping stakeholders informed on contemporary media creation and consumption practices. Although it was acknowledged that current creative methodologies are openly engaging with 3D technologies, computer-generated models, and virtual environments, the role of MR and its integration into current practice was thought to impact upon several fundamental stakeholders in the wider domain, including:

“People who are interested in going to the theater, anybody who loves performance and theater, theater-makers, the actors, directors, designers, people who love tech, people who are performance and technology experts, students, obviously, and practitioners.” (TQ44)

Discussions on the potential disruptions to practice explored the role of the producer, who must acquire new skills to yield professional MR content, as one participant highlighted: “It won’t be someone who thinks purely in theatrical or cinematic modes”. Thus, producers will have to think differently to understand the idiosyncrasies of immersion, how audiences will consume new content, and how this process will affect performance practices. While this role would suit directors/producers who have “A very distinct idea about what they are looking for in a performance”, it may constrain some producers. It will provide a new platform for artists to express their imaginings and engage audiences as they envision in a more pure and controlled way – “The filters seem to be different and more manageable, from an artist’s point of view”. Effectively integrating MR into practice requires a rethinking of the relationship between screenwriting and directing to realize scripts with 6DoF. By maintaining elements of the existing “film-making perspective” and developing experimental practice, MR could push the boundaries of the current mainstream model. This would provide a new MR platform for content creators with “More experimental urges”, like Jean-Luc Godard, to make full use of 6DoF formats – “To deconstruct aspects of style and our perception” – in very different ways to conventional Hollywood studios.

The requirements of MR in practice were compared to that of film production pipelines – a collaborative endeavor, where certain roles are critical and others peripheral. Where performances occur within 3D space, the “standard director model” would be challenged by scenographic considerations – “How they [the audience] are orientated and located [will be] really important”. It was thought that this factor would be particularly disruptive, because the capture-to-consumption process would have to prioritize a more spatial “world-building” perspective over the dominant temporal one. For larger productions, a production designer would have to design exactly how the virtual world should look.

Participants repeatedly stated that MR was currently suited to avant-garde gallery installation (which already has an established grammar) above mainstream performance spaces, so platform-specific challenges would have to be addressed. It was also stressed that avant-garde cinema does not attract large, regular paying audiences; as such, work is carefully programmed by arthouse cinemas or film festivals. As embodied performances often take place in a spacious area in front of an audience, the constrained nature of the field of view in MR was potentially problematic for some practicing SMEs. In response, it was suggested that the producers of VV performances must carefully consider the space and the range of views available to the audience in an IVE; specifically, in AR, an audience “Could move through a space and discover” new perspectives. It would be more akin to reading the text, or “actually walking” through a physical scene, engaging with content, and interacting with the environment. MR has the potential to provide multimodal experiences for the audience, where they can “feel” the physicality of the mise-en-scène and move around it:

“They could feel the walls and hear the dripping, a physical thing that they are really experiencing and hearing Beckett’s words at the same time... If you give the freedom to the body, you give the meaning to the body and it moves and crouches and stands. You create a choreography of the body while there are also experiences of the visual” – (UQ80).

MR technology will disrupt the traditional way of telling stories, as the audience has “The power to create the story” and complete it; for example, in Virtual Play and Augmented Play, the user becomes the “light”, provoking the character’s speech or reducing them to silence. This interactive narrative model points towards new ways creatives can approach storytelling, as the viewer becomes a type of director – not the author, but an experiencer with some authority. One SME observed: “We compel people to do something or not do something; to me, this is the most disrupting thing”.

4.4 The Challenges of Using MR

The interviewees expressed that the role of MR within the domain brings many unique challenges to cultural industries. The advantages and disadvantages that were described by the SME cohort largely related to practitioners and audiences, or as one SME explained: “The ontology of the virtual is still incredibly interesting, and philosophically problematic, as to what it is and how we exist within these virtual domains”.

The SMEs reaffirmed that there will be specific roles affected by the introduction of MR. Academic SMEs expressed interest in supporting MR production skills within the classroom. For other participants, this extended to related fields like motion capture, animation, and innovations in TV production, where there is a responsibility for institutes to keep up to date with new technologies as they emerge. For Virtual Play and Augmented Play, the work involved in the post-production stage (involving in the reconstruction, programming, and world-building within the game engine environment) far outweighed that of the brief period of filming real-life actors, at the beginning of the project, and one SME asked: “How useful is this in actually creating new theater, and evolving the art of acting?” Practicing SMEs also indicated that directors were now entering more dramaturgical roles. They believed directors were increasingly expected to draw out and distill the essence of a script, to allow for a more interactive performance that prioritizes the audience perspective. In this regard, “Directorial creative input would diminish hugely... It diminishes because what you are doing is putting the elements together, and then handing it over”.

Although deeply interested in the future of actors, one pedagogue explained that apart from some work “...with the actors in the motion capture lab”, there are few options that explicitly focus on the use of emergent technologies in practice. SMEs speculated on potential advantages and disadvantages resulting from the VV capture process. It would place increased demands on acting styles, as performances become “Even more intimate than even a big-screen closeup”. From previous experiences and watching the MR performances of Play, SMEs commented that the opportunity to get up close and personal with the actor, without the weight of social constructs around bodily intimacy, meant that the nuance of the performance (like micro facial movements that indicate certain emotions) could be scrutinized more closely, and performers would become pressured by this level of dissection. “It’s hard to say that this would negatively impact [actors] any more than film, or... existing hierarchies”; actors would more likely engage the challenge with zealous enthusiasm. It was also believed that experienced theater workers (e.g. technicians) will always develop new skills in line with technological evolution: “They’ll be fine as long as they skill up”, or, as another participant noted: “I don’t want to take the position that technological advancement is necessarily going to destroy somebody’s way of life. It’s a disruption... but a lot of times people find ways of coping”. Practitioners must, therefore, become more focused on idiosyncratic elements of the platform and how they relate to the work being created. “It should be simple, but effective... you have to think a lot more about the design and making sure it is palatable and really clear - concise!”. In the planning stage of an MR project the creators will have to focus their creativity on world-building – “You are going to be storyboarding for a lot longer, taking longer to establish the world”. MR was believed to be a very powerful way to show plays conceived in design-led practices that emphasize immersion and visual elements, akin to “Bringing a painting alive”. Therefore, much excitement was expressed about its potential to engender new types of performance and extend existing visual practices.

One SME commented that VR installations in art galleries “[They] always seem to be an odd experience”, because VR is a solo experience and if there are other people entering the same physical space, surrounding the user, it generates a vulnerability or anxiety. Additionally, by entering the experience on a temporal continuum, visitors determine the beginning and end: “You decide yourself when you are going to bow out”. This practice influences how practitioners formulate their work, translating linear thoughts into non-linear practice. Some SMEs held that the works of Beckett, such as Play, were “Significant, major artwork[s]” and audiences should be allowed to experience them as the creator envisaged; the MR experiences are not necessarily better, as the audience may not have the capacity to reconstruct the complex narrative themselves. Most of the cohort’s experiences were led by chance – “You look left, you look right; it’s just an alternative subjective moment when you make that choice”. This begs the question: “Do you enhance the Beckettian experience of that work or do you in some ways diminish it?”. In a conventional performance the narrative is delivered in a linear fashion, like “A play function, where it just gives you the entire play, straight through”, but these MR versions show how audiences can explore performances from multiple perspectives, attaining new levels of fascination with each viewing.

The cohort cogitated on the performance space where MR could be consumed collectively by audiences. They suggested that MR technology would have to evolve beyond the current individualistic mode of consumption to a more communal one, where mediated, interconnected audiences are engaged collectively. In naturalistic theater, a suspension of disbelief is often established by imagining a so-called “fourth wall” between a stage and an audience. The cohort emphasized that “At a live performance, the audience has a certain responsibility... When it’s a conventional theater, the lights go down and the audience remains quiet”; whereas, in some performances, the audience accepts a contract of interacting, following prompts within the performance, or consenting to be moved around. Engaging with MR content was described as comparatively easy and accessible, as there are clear and explicit expectations of the user. It was, therefore, supposed that MR will contribute to removing these spatial thresholds: “You aren’t in your own space anymore... that’s transformative”. VR performances can bring audiences into new spaces where they may not have previously been. Although AR was considered exciting on a technical level, the cohort quickly became accustomed to digital objects occupying their world; conversely, the sense of immersion afforded by VR gave the SME audience a deep sense of being “Transported to another place”. They felt that, in VR, “The more interesting the material, the more you’ll stop thinking about what you are wearing, and the wires don’t stop you from moving”. This led them to think about how to move audiences around an IVE without causing collisions, whereas they could easily imagine creative compositions for more mobile audiences in AR, because it mixes the imaginary with the real, without overburdening/isolating singular senses. Therefore, in relation to the audience–performer dialectic, most of the SMEs preferred AR because they could more easily imagine successful collective experiences.

Moving forward, SMEs suggested that the technology would have to provide an interactive space for the observer and performer, “Where performers are moving and interacting physically amongst themselves” and the audience. This prompted the question: Does that place the audience in the performance? In MR performances to date, the perspective of the audience is almost always directed and limited, and the viewer is often seen pivoting in a circle or looking for a macro-perspective over the entire scene. This may contradict the potential of walking around a performance space; however, if the performers were to start moving around the MR space, then the possibility to navigate around them presents itself, and “That starts to change your role... and your position within it, as an audience member, which opens up all kinds of interesting things”.

4.5 Receptiveness of the Community

The cohort commented upon the receptiveness of the domain to new technology, including their openness to using novel technology in practice and the technologies that they were currently presenting with. The SMEs generally agreed that MR technology would have a positive impact on multiple domains within the cultural sector, particularly as the technology itself would be an attraction. While they generally advocated the use of technology, our SMEs were wary of the potential “suffocation” of theater, as “A unique form of art that relies on the corporeal presence of the actors”, through excessive technological interventions. Although technology was sometimes seen as a detractor, AR and VR were described as “kind of fun”, so while the novelty factor lasts, the technology was considered to be attractive. It was suggested that during this period of novelty, MR performances would see a boost to their profile and viewership. Furthermore, future advances in networked technology were identified as potentially providing MR audiences with distributed shared live experiences. The cohort predicted that more interesting work would emerge, particularly MR as an art form in and of itself, which does not impose a technological innovation upon other art forms. When the form of the artwork is a defining feature of what it is about, it raises philosophical questions about its nature. It was generally believed that for MR to be integrated into practice, the domain would have to be receptive and should use it as a resource to fulfill its intentions. But it may also provoke artists to think differently, as one participant suggested: “What is the thought that the virtual is thinking or inviting us to think? What happens inside the virtual... unto itself? Where does art collide with technology?”.

The use of VV to represent characters within MR was described as a potentially disruptive intervention, comparable to that of video, due to the apparent “liveness” of the experience. Therefore, the space between live and mediated experiences will potentially become more problematic, as the mediated experience may precede the live. This was expressed as being particularly disruptive to the very nature of what theater is, as something “Here and now, and, all of a sudden... not here and not now”; the temporal and spatial configuration of the stage could be wildly disrupted. The possibility for technology to help develop a sense of intimacy and connection with the performer was described as a desirable way to engage with cultural content. As such, it was noted that MR technology has drastically improved audience engagements within the cultural heritage domain. However, AR HMDs were not thought of as being as advanced as VR, because they are not as freely available, and, as one participant pointed out: “[AR] feels nit-picky; it’s cool! But it’s not quite there yet”. Furthermore, it was noted that areas of the emergent MR market are still in a state of flux and hard to predict, and different devices vie for prominent status and market share. As one SME explained:

“If for example, Facebook’s Oculus becomes the most popular, I can imagine a thousand news ways advertisements can be pumped straight into my brain. But if Microsoft’s HoloLens becomes more popular... it really depends on the existing business models of the company...” – (GD38)

The cohort had predominantly experienced AR via “Window on the World” methods, as practiced on mobile phones and tablets. One SME described the mobile phone as a “companion; it’s part of us”, so the experience of using an AR HMD was “really disturbing”. This conflict between existing human-computer interaction experiences and user expectations caused “uncanny” sensations. Using AR was described as “disruptive” to the overall experience, because the passage from watching Play in an IVE to the intrusion of physical reality, in AR, was “immersive breaking”. The AR visual was described as “Glitchy”, and the overall experience was “distracting”, because reality encroached upon the performance. This included references to the emotional disconnect introduced via the physical space, as “...the context isn’t as good”. To improve immersion with AR, it was recommended to use site-specific locations, as one participant noted: “You could do cooler stuff with it if you find a cool building... or location to do it in”. Furthermore, it was suggested that:

“It’s not that AR failed to provide anything, because it did what I expected it to do... I like things that will really challenge me... and that’s the type of thing that I want to get out of [VR]... I think that this technology captured that sense.” – (YH87)

Although lesser online virtual environments (e.g. Twitter, Facebook, etc.) provide a more streamlined experience, MR was still considered to be a vital, rich area for research in the arts. It was believed that advances in the next 20–30 years will be impressive and, as bandwidths increase, HMDs will become more desirable platforms for users. Although some potential resistance was predicted, research into creative practice in MR was still considered to be incredibly useful and productive within the cultural sector, as one SME noted: “I think, to a certain degree, the virtual has completely overloaded the traditional artform - drowned it out. So, the interesting work will be finding out what, within it, makes art?”. Therefore, it was also suggested that the aesthetic theories applied in art practice would require some updating regarding what is disembodied and what is embodied within a performance.

5 Conclusion

The SMEs expressed an awareness of the disruptive nature of engaging with VV content through MR technologies and likened it to “watching the train coming at you in early cinema”; the technology is still emerging and, therefore, “still comically awkward”. There still remains much work in making MR technologies more affordable and integrating the VV techniques into current film, theater, and performance practice workflows, so that practitioners are comfortable creating with it and audiences are more enthusiastic about engaging with it. Audiences currently experiencing MR struggle to suspend their disbelief; therefore, the performances and content being delivered can become overtaken by “the joy of watching the technology”. The novelty of the technology currently makes it hard to measure its effect, as it is impossible for users to separate their awareness of the technology in assessing content. Nonetheless, in the short-term, this will drive the popularity of MR as an exciting and topical medium.

Our domain experts affirmed the potential for VV techniques in MR to significantly alter existing modes of artistic practice and audience engagement. They affirmed a contextual understanding of mixing realities with emergent MR platforms (within the film, theater, and performance domains) and provided insight into the disruption that could be caused by the widespread adoption of the technologies in future modes of creative practice and audience consumption. However, they highlighted the need to establish a new storytelling grammar that, from the maker’s perspective, prioritizes specialization and mise-en-scène over the incumbent temporal paradigm, and from the audience’s perspective, that facilitates enjoyment and engagement constituted by exploration and discovery over and above listening: “storyworlding”. Therefore, further research is required to explore the idiosyncrasies of immersive technology that will dictate how this paradigm shift will affect the practice and consumption of creative cultural performances. Important factors that surfaced in discussion that will have to be carefully considered include the time, planning, and attention to detail needed to create both the IVE and the site of performance, in the case of AR. However, the specificities of the technology need to be linked to the content to avoid art being conflated with technical exhibitionism. From the perspective of keeping audiences engaged, performances will need to be reconceived on the basis of: timing and pace; existing, interactive narrative models (e.g. in gaming) that successfully employ user profiles, actions, rewards, etc.; and reconciling the shared communal experience.

There were warnings about attempting to overtake existing, effective, and time-honored modes of storytelling; MR should be explored as a stand-alone new media. Specifically relating to the cultural heritage of the playtext, there was considerable debate about the choice to reinterpret a play by Samuel Beckett for MR technologies, when the author originally and explicitly conceived it for the proscenium auditorium format. However, as argued elsewhere, any act of performance implies a translation that generates difference; the responsibility to maintain the integrity of the original (conceptually) coexists with strong incentives to innovate and push the boundaries of the work (practically) [11]. Modifying the medium of the work, while a fundamental shift, also serves to make the content accessible to contemporary audiences. This reactivation implies a survival of the literary work as a part of our intangible cultural heritage, but it does not guarantee that such survival will be mutation-free.

In this paper we focused on one strand of the SME discussions: those that related to practitioner versus audience viewing. Other topics of discussion related to cultural heritage and fidelity to the original, MR as a pedagogical tool, performance theory, and the (bio)Politics of access and data privacy. Therefore, the gathered data demands further reflection and research. Moreover, the potential of MR for developing empathy-building experiences will also require further exploration to help define the effects of this technology on perspective-taking.