research-article

Open access

Crafting Trajectories of Smart Phone Use at the Opera

Authors:

Elizabeth KellyAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 29, Issue 6

Article No.: 59, Pages 1 - 37

https://doi.org/10.1145/3531007

Published: 16 November 2022 Publication History

All formats PDF

Abstract

Losing Her Voice is a new opera which highlights the challenges of subtly interweaving digital technologies into established cultural forms. Audience members were encouraged to use their own mobile phones to interact with on-stage projections before, during and after the performance. We chart the trajectories of participation that were designed into the premiere performances of this work, and how these unfolded in practice. We identify the strategies that were effective to: encourage the adoption of the mobile app; interweave use with the other elements of the opera performance; make it consistent with the content; and ensure it complemented but was not essential to the show. We highlight the way in which three canonical trajectories must be woven together in this experience: the standard trajectory of “a night at the opera”; the dramatic arc of the specific work, and the audience member's use of their own device.

1 Introduction

“Please turn off your phone or set it to silent”: this familiar refrain greets us almost every time we visit the cinema, theatre or opera. There are exceptions, especially in more immersive forms of theatre (e.g., [31]), but in traditional proscenium theatre experiences—including most opera—the audience still expects to sit attentively, silently and usually devoid of direct engagement with technology. Sometimes tailored devices may be provided to the audience—such as seat-back displays for subtitles [39]. But the audience's own devices are relegated to a bag or pocket, lest they “ruin” the show with an incoming call or notification, or the dazzle of the backlight in the darkened auditorium.

Losing Her Voice is an opera written and composed by Elizabeth Kelly (University of Nottingham), which premiered at Nottingham's Lakeside Theatre in April 2019. In many ways the opera is quite conventional: it uses traditional “end-on” staging, is fully scored for an instrumental ensemble and singers and has a single fixed narrative and libretto. But, in addition to live music, singing and projected visuals the performance incorporates a mobile phone app, designed to be used before, during and after the performance on audience members’ own devices.

This article presents the design and evaluation of the audience app (as used in the premiere performances) and the associated audience journeys (i.e., trajectories) through the opera experience. The key research challenge that emerged from the early stages of our practice-led design process was how to subtly interleave the app with the wider show in a way that complemented it but without disruption and encouraged uptake but without demanding use. In describing how we negotiated this challenge, we build on and contribute to key HCI concepts of user journeys or trajectories, display ecologies and spectator interfaces, as well as providing rich data on potential uses of smartphones in relatively traditional theatre contexts.

We begin by charting the various ways that digital technologies can enhance or transform live performances, then focus specifically on approaches to audience interaction, and the use of mobile phones in particular in theatre settings. We also survey the trajectories framework [8] and related strong concepts [22] that inform the design of how audiences engage and experience a performance of Losing Her Voice. After introducing the opera, Losing Her Voice, we report early design explorations that highlighted the challenge of subtle interweaving. We then present the goals, design and realization of the audience app and the complete audience trajectory. Drawing on post-performance audience questionnaires, interviews and system logs of app use we analyze the audience's experiences of the app. Reflecting on these we discuss our key strategies and recommendations for enhancing audience involvement through their own devices.

2 Related Work

We begin by charting how digital technology has been used to support audiences in theatre settings, ranging from passive viewing to audience interaction, and smart phones in particular. We then review trajectories and related “strong concepts” from HCI that frame our consideration of audience experience and interaction design. As an initial framing, we observe how digital technologies scaffold the majority of our contemporary performance experiences, whether they are stadium rock concerts, West End theatre productions or virtual performances experienced online. Technologies support both the “on-stage” presentation but also increasingly mediate the audience experience. Troyer [42] explores how such audiences are “becoming communicative and participative in the performance using technologies” (ibid., page 23), referring to them as the hyper audience. Our work is interested in the application of mobile devices to engage and draw the audience into participation with the performance of a new opera, Losing her Voice.

2.1 Audience Technologies in Theatre Settings

The most widely accepted use of personal devices in live performance is to provide subtitles and perhaps other supporting material (usually visual, but in some cases music), usually synchronized with the main performance. For over 30 years, dedicated seat-back devices have been used to provide access to captions or surtitles [39] and often other programme-style content. While there is often a reluctance to rely on audience devices, site-specific productions may provide surtitles via dedicated mobile apps [11]. In Self [48], an experimental play about a woman's social media interaction, a dedicated app on audience members’ phones replicates the content appearing on her phone on-stage. In these cases, the audience does not strictly influence other elements of the show, but they are still required to continually shift their attention between the main performance and the additional content, and in some cases, also to interact with their own device (e.g., to keep it on, or to access optional content).

In live streaming there are also comparable elements. In the Salome Experience [34] a live stream of an opera is brought into a free-form conference social event, where a custom app provides summary information synchronized to the live stream. Vienna State Opera's StaatsoperLive app is a second screen app specifically for their live-streamed operas and is also available to view synchronized supporting content. This highlights the relevance of research on the use of second screens—generally personal phones and tablets—within the context of television viewing [28]. There are some significant overlaps, for example, in managing and designing for shifting attention between a primary and secondary media experience (e.g., [27]). However, the social context is very different in live theatre, and second screen functions such as remote control and time-shifting (e.g., [2]) are not applicable.

The work of MIT's Media Lab Opera of the Future group, led by Tod Machover, has explored opportunities and roles for technologies in opera settings, such as the hyper audience framework for audience participation and interaction [42] and Media Scores, a notation framework for the holistic creation and realization of mixed- and multi-media, interactive performances [41].

2.2 Support for Audience Interaction

Beyond theatre, in classroom and presentation settings, there are a small number of relatively stable interactive forms [29], either led from the front (e.g., multiple choice, question setting) or from the audience (e.g., free text backchannels). In more playful settings, a number of systems have allowed collective audience interaction, for example, using reflective paddles [12], video tracking of collective movement or mass laser-pointer use [26] or the loudness of audience cheering [3]. Immersive theatre, where the distinction between stage and audience is rejected [11], can provide distinctive opportunities for making personal devices part of the experience, as in Flatland's haptic devices in the dark [45] or Internet-of-Things style sensors that can be used to detect audience presence and motion [21].

Considering audience interaction, specifically in a theatre context, Parcival XX–XI presents an interesting case in which a small number of audience volunteers literally step forward, in order to interact [19]. At two points in the performance, three or four audience members are invited on stage to interact via Nintendo Wiimote controllers with virtual avatars projected on the stage. In this case, audience interaction is essential to the work and therefore obligatory (at least for some of the audience). This requires some specific prompting and giving of “permission” to the audience to interact, whether through aesthetics or narration (in this case by a “Wii fairy” helper role). However, these interactive elements were found to disrupt the flow of the show on several levels, one of which being that these audience interactions took place “at an inappropriate point in time during the play, thereby disturbing the rhythm of the performance” [19]. Furthermore, Friederichs–Buttner et al. [19] highlighted the need for audiences’ to familiarize themselves with the mechanics of audience interaction prior to the performance in order to be able to fully understand and engage with it when it matters.

2.3 Mobile Phones in the Performance

Considering mobile phone use, in more “permissive” performance environments such as rock concerts mobile phones may be used extensively for coordination, chat, to capture and share media, and as a sea of lights [14]. Climb!, an interactive classical piano performance, involved an interactive audience app that provided programme notes, an interactive map of the route that the performer was taking through its non-linear score as well as the ability to compare the current performance with previous ones [9]. In a quite different vein, Bootlegger systematically directed audience members to capture video footage of live music shows, dynamically directing attention to different events on stage and suggesting particular shot types with the intention to transform the audience into an improvised film crew [35].

In improvised live music the audience may be given varying degrees of control over the music to be played, for example, steering the algorithmic generation of a real-time score [1] or giving collective cues to guide the musicians’ improvisation [46]. Wu et al. [46] also tease apart some of the variable dimensions of participation, for example: audience motivation (imitative, competitive, contributing, or directing/conducting) and degree of agency (e.g., adding content or generating the entire narrative).

The mobile phone as an audio broadcast device has been explored across a number of performance settings, such as Stanford University's Mobile Phone Orchestra [30], where internal sensors on mobile devices are used as audio controllers by performers. Dialtones [25] offers an example of harnessing audiences own mobile devices into performance settings. Dialtones is a mobile phone performance work where the automated calling of audience members mobile devices is carefully coordinated to create a distributed performance. Originally composed ringtones are loaded onto audience members mobile phones pre-performance, which when performed turn the devices into a speaker array. It's worth noting in this instance that the audience takes a passive role, as content is pushed to their handsets. In Fields [36] audio elements are similarly broadcast via audience members mobile devices, but in this instance, they have direct control over its playback, for instance triggering, sequencing and some rudimentary control of synthesis. Here the designers embraced the inconsistent latency between handsets and operating systems as part of their audio design language.

We finish by looking at a number of key examples of audience interaction using phones in more scripted theatrical forms. A simulcast broadcast of the opera Death and the Powers to nine remote venues in different cities found audience members engaging with second screen content via their mobile phones in addition to collective control of the stage lighting in the venue where the performance was held. The intention was to offer remote audiences a direct connection to the live performance though additional content and a distinct experience of the work [40]. In an experimental collaboration around a sci-fi opera [47], a dedicated mobile app allowed the audience to affect aspects of the lighting and other theatrical effects, however, the director and the majority of the audience felt that it had little and unclear impact on the show. In Moori, [24] semi-improvised performances are created by the performer posing questions to the audience, who can respond via text messages (i.e., SMS) from their mobile phones, which are then displayed on stage and broadcast to the auditorium via text-to-speech software. “ADA FTW” [13] is an interactive drama, in which question are presented to the audience on a large on-stage video display. The audience can provide free text responses using SMS, twitter or a website, which also appear on the screen (after backstage moderation). 32% of audience members found the overall experience fun and exciting while 51% of audience members qualified their experience as distracting and unfocused. They also note that social norms conflicted with texting during the performance.

In Transitions [32], an interactive dance performance, an android application supports coordinated interactions across all audience devices and a number of projection screens. In one activity, each device controls a different bird in a flock on the projection screens while in another activity a single virtual dancer can be “tossed” from one handset to another. These interactions were reported to be effective in engaging the whole audience, but at other times the audience was uncertain about whether to use their device. Finally, The Smart Phone Project [31] is an interactive dance performance supported by a dedicated mobile app. The audience is told they can use their phones during the show, with each of 12 scenes using the phone in a different way. The most highly rated were: showing live text typing from an on-stage actor; voting on costume choices; music that transitions from the PA to audience phones; and showing augmented reality content linked to visual markers.

Where the goal is to use audience members own phones there is often specific support provided in the form of a dedicated Wi-Fi network and someone to provide technical assistance, e.g., [31, 32], and sometimes loan devices. There are also recurrent challenges, at least in more traditionally framed performance experiences, in getting the whole audience to engage with the technological and interactive elements.

2.4 Trajectories

We now briefly review trajectories [8] and related “strong concepts” from HCI [22] that we have applied and reflected on in this work. The trajectories framework [8] proposes a set of concepts for both the design and evaluation of complex user experiences. It is grounded in examples of performance art experiences involving digital technology. It views users’ experiences as trajectories or journeys, potentially spanning space, time, roles and interfaces. The initial trajectories framework as first described in [8] and then subsequently further developed in a book-length treatment [7] proposed that user journeys through mixed reality performances are composed of three distinct kinds of trajectory: the canonical trajectory of the scripted or intended experience; the participant trajectory as actually experienced by a particular individual; and the historical trajectory of a past experience as retold or re-presented. In this article, we are concerned primarily with the canonical and participant trajectories of audience members attending a performance of the opera Losing Her Voice. The framework calls attention to the transitions that occur in a user's experience, for example between roles, interfaces or locations, and to the significance of beginnings and endings in particular. It also highlights the social interleaving between different participants’ trajectories, especially the idea of balancing isolation as well as social contact throughout an experience. Finally, it stresses the idea that participant trajectories may diverge from canonical ones, but then may subsequently re-converge through orchestration from behind the scenes.

Later work [18] directs attention to the way that local trajectories of interaction are frequently nested, with recurrent patterns of engagement (i.e., trajectories) around specific elements within a wider global trajectory. And while the initial framework discusses convergence and divergence in participant trajectories, [17] draws attention to the ways that collocated groups (e.g., family groups) manage their own distinctive patterns of joint experience.

In its consideration of interfaces, the trajectories framework draws on and contributes to the concept of display or interface ecologies [23, 38]. The ecology metaphor calls attention to the existence of multiple displays of diverse kinds, their inter-relationships, and the ways that activity may span or move between those displays. As we discuss in more detail when we consider the design of our app, the typical theatre-going experience is replete with “displays”, although most of them are neither digital nor interactive (e.g., posters, programmes, signs, stage). Many of the audience import their personal display devices (smart phone, and so on.) into this relatively static ecology, while on-stage digital projections are now routine in many performances.

In relation to roles and role transitions, a rich thread of research has considered spectator and bystander roles. Reeves et al. [33] consider how watching someone interact with a system can have its own theatricality (for example, magical or secretive) depending on the extent to which the user's action and their results are made visible to others. Sheridan [37] draws attention to the bystander's knowledge that “something” is going on to distinguish them from the “unwitting” observer, while [6] highlights the way that this knowledge of who is and is not “involved” can be manipulated creatively. Benford et al. [5] draw this back into the framework of trajectories to highlight how spectating can be a key stage in some experiences. In terms of the theatre-goer's experiences the default expectation is that they will act solely as an observer. But as they start to use their personal devices, they first become exposed to observation in turn by other audience members, and then—at least to the extent that their interaction can affect the performance (i.e., drawing attention away from other performance elements)—they become to some extent a performer. Trajectories were used as a lens to analyze the audience experience of a mobile programme guide during the Climb! performance mentioned above, concluding that the trajectories of individual performances need to be embedded within the wider historic trajectory of multiple performances of the work in addition to audience engagement with the work out of the concert hall (e.g., through an archive of recordings) [9]. This current article directly responds to this observation by attempting to design these kinds of embedded trajectories from the outset.

More generally, the trajectories framework has been widely cited in the literature. Early on in its development, Höök and Löwgren cited trajectories as a candidate “strong concept” in their influential article on intermediate design knowledge in HCI [22]. A recent survey article presented a systematic literature review of how the framework has been employed in the wider literature [44]. Their analysis of a corpus of 60 articles that cited trajectories helped reassess the four claims made for its applicability in [8], finding that: it was frequently adopted as a “sensitizing concept” for analyses of user experiences within HCI and that it had also helped inspire “new dramaturgies” in performance studies, but there was less evidence of it serving as a “vehicle for compiling craft knowledge” or helping identify “requirements for new technologies”. A subsequent article explored the extent to which trajectories, as a theoretical HCI concept, can shape industry practice, analyzing a five year-long case study of working with a major media corporation's UX department to bring trajectories into mainstream experience design [43]. The authors conclude that it is challenging to bridge the gap between HCI theory and UX practice and called for further research to explore theory-led, practice-led and especially co-creation approaches to tackling this. The current article presents a response to this call; an attempt to put trajectories to work from the start of a design of a new interactive performance through a co-creation approach between creative practitioners and HCI researchers, as we now describe.

3 Approach

Our approach is one of “Performance-led Research in the Wild”, a research method that involves collaborating with creative practitioners to design, stage and study public performances that harness digital technologies in novel ways [4]. The approach falls under the broad umbrella of Research Through Design, being predominantly practice- rather than hypothesis-led, with generalized knowledge emerging from reflection across a portfolio of designs [20]. The creative vision, goals and motivation for embarking on a particular performance project typically originate from creative practitioners, with researchers then helping them realize their ideas by developing supporting technologies, while also reflecting on the design process and audience experience. The research takes place “in the wild” [16] in the sense that the performances are made for, and tested in, the crucible of public performance, that is in front of a live audience at an appropriate cultural venue. This requires the performance to be delivered under demanding conditions, to navigate all the challenges and constraints this entails, and also be framed as an artwork rather than an experimental research project for the audience. Particular research aims and questions typically emerge throughout the process, as was the case here with our emerging focus on the subtle interweaving of digital technologies into a traditional operatic format. In this case, the creative vision for Losing Her Voice was provided by a composer who was also an academic in our own University, but as part of the Music department (and so a Music scholar rather than an HCI one), and the theatre was a busy public theatre also located on our University campus.

Previous accounts of the approach [4] have stressed how it involves three interleaved activities: the practice; studies of the practice (both of the making and the audience experience); and theorizing new concepts that generalize findings for a wider research audience. While any individual project is usually led by the practice, with studies and theorizing following, these previous accounts have shown how these activities become interleaved over time, with emerging concepts then informing the development of further practice, as was the case here. While the creative vision of Losing Her Voice originated with the composer, the details of its design were strongly influenced by our “trajectory thinking” from the outset. In particular, previous reflections on Climb!, through the analytic lens of trajectories, shaped our thinking about how to systematically design audience trajectories in this new work.

Figure 1 shows how the development process unfolded over 21 months during which the audience journey was progressively refined and tested alongside the performance itself.

Fig. 1.

Initial research into the concept and historical context of the show by the composer was soon mirrored by parallel research into potential onstage and audience technologies, and the opportunities the technologies offered increasingly influenced early creative decisions regards the libretto, score and intended staging of the premiere performances. In parallel with writing and composing, the technical team began to develop potential tools, and these activities were brought together in two key design workshops (i.e., November 2018 and February 2019), including some preliminary audience testing of selected scenes from the show at an interactive music festival hosted by Nottingham's Royal Concert Hall (November 2018). An intense period of rehearsals with actors, musicians and stage crew then led to the two premiere public performances. The goal to have audiences interact via their mobiles was raised at very earliest stages as were some of the challenges that this would involve, especially the challenge of getting them “onboard” given the anticipated audience demographic and context. In what follows, after providing a brief description of the opera we summarize the two early workshops that shaped our technical approach, before presenting the final design of the audience journey as eventually deployed and studied.

4 Losing Her Voice

We briefly describe the opera itself, Losing Her Voice, in order to give the reader some necessary context for the subsequent discussion of the design and use of the mobile app within the total opera-going experience.

Losing Her Voice was composed and the libretto written by Elizabeth Kelly (Department of Music, University of Nottingham). The opera brings to life the story of the early 20thcentury American opera diva Geraldine Farrar, who transitioned from “stage to screen” and became a silent film star. Alongside celebrating Farrar's achievements, the opera explores ideas surrounding the cult of celebrity and the impacts of new technology, specifically those of recorded music and film at the time. The opera draws on a variety of archive material, especially contemporary newspaper articles, her fan's scrapbooks (many of which are held by the US Library of Congress) and the silent film version of Carmen (Cecil B. Demille, 1915) in which Geraldine stars.

Losing Her Voice is in two acts, with a total of 11 scenes and a running time of approximately 2 hours including the interval. The first act traces Geraldine's learning and rise to success as an opera singer, climaxing with the loss of her voice due to over-work. The second act sees her transition to silent film as an opera celebrity, traces her tumultuous relationship with co-star Lou Tellegen, and her legacy in the emerging media of sound recording and film.¹

The opera is relatively traditional in form and staging, the premiere being in a “black box” proscenium theatre with an audience capacity of approximately 140 and a stage area approximately 7 m wide and 10 m deep (see Figure 2). The set consisted primarily of three framed projection screens: one forming the back wall of the stage, and one each to left and right mid-stage. The projected visuals mixed newspaper articles, contemporary photographs, scene-setting text, a live camera feed and (in Act 2) fragments of the silent film Carmen, pre-recorded video of one of the principals and social-media style content linked to the audience app (which will be described below). There were four principal singers, ten in the chorus, and seven in the musical ensemble plus a conductor (who were also on stage, as the theatre has no orchestra pit). Surtitles, provided for accessibility, were projected on a separate small screen above the stage, controlled from a dedicated computer.

Fig. 2.

5 Design Process

The use of digital technologies in Losing Her Voice emerged through two distinct stages of the design process. First, the composer, Kelly, and the technology research team undertook an initial broad exploration of technologies that might be introduced in the piece. Second, the subsequent involvement of a professional production team narrowed these to one approach—supporting a gradually unfolding audience journey via mobile phones—that was deemed to be appropriate to the work.

5.1 Early Explorations

Kelly first encountered the early 20th Century soprano, Geraldine Farrar whilst researching information for her teaching practice at the University of Nottingham. Inspired by what she discovered, which included the observation that Farrar's career was mediated by emerging technologies of the period, Kelly decided to write an opera about the life of Geraldine Farrar and the issues that surrounded her career and very public “private life”. She began work on initial concepts and approached technologists at the Mixed Reality Lab (MRL) (School of Computer Science, University of Nottingham) in July 2017 to collaborate on technology research.

A series of open-ended design discussions took place between July 2017 and October 2018, including mock ups, and prototypes. These discussions were framed by four creative and artistic concerns that Kelly introduced at the outset. The first was to explore the implementation of novel technologies within the performance of Losing Her Voice, mirroring Farrar's engagement with the novel technologies of her day. Notwithstanding this, Kelly was also committed to creating a traditional opera, i.e., written to a standard two act structure with live singers and acoustic instrumental ensemble to be performed in a traditional theatrical venue. Any integration of novel performance technologies therefore had to enhance the audience experience without compromising the opera itself. And the work should remain an excellent piece of contemporary opera even if stripped entirely of technology. The second artistic concern was with the notions of the “screen” and “reflections”. Farrar's career was shaped by the contemporary technologies of her period, in particular silent movies, broadcast radio, vinyl recordings and newspapers, and her transitions between them including her move from opera stage to silent movie screen. These technologies were all manifestations or variants of screens or displays. For example, Farrar's singing teacher, Lilli Lehman required her to practice whilst looking at her reflection in a mirror, whereas her opera performances were framed by the theatre proscenium and her silent “movie” performances by the cinema screen; print media captured her in image and description and her voice (i.e., audio) was displayed via radio broadcasts and vinyl pressings. Third, Farrar's celebrity was to be a main feature of the narrative, as manifest by the press and her fans, who were playfully labeled “Gerry-flappers ” at the time. Forth was the idea to reanimate existing archival media to support both the storytelling and the visual aesthetic. This archive material was principally footage from her silent movies and the “Gerry-flapper” fan scrapbooks (available in the extensive US Library of Congress Geraldine Farrar collection) which highlighted the extent of her celebrity and the importance of print media in facilitating her celebrity profile.

At this early stage in the opera's development there was an intention that the work would be performed at some point in the future, but it was unclear when this might be, where it would be, and by whom. Consequently, these early design discussions were unshackled from any specific constraints or practicalities of a venue, production, and team or funding and they consequently covered significant scope in terms of potential technology interventions. Early ideas included the control of lighting, projections and DSP (digital signal processing) of the singers’ voices and instrumental ensemble triggered directly by singers’ voices or bodily gestures. Other ideas included live green screening of singers to move them between “stage” and “screen” and an interactive installation in the foyer (of a future performance venue) to capture media to then be used in the performance. The idea of focusing on audience interaction arose early in these discussions and persisted. Primarily, this was considered to be an attractive mechanism by which to position audiences as characters within a performance of the work, i.e., as members of the press and/or “Gerry-flappers”. To enable this interaction a range of approaches and technologies were considered, such as audience members moving around and engaging with distributed technologies (i.e., props with embedded sensors), or using commercial wireless “classroom” style voting systems, or even sensors embedded in audience seating. As discussions continued, high-level design goals emerged; essentially refinements of Kelly's initial concerns:

(1)

Audience interaction via an app running on their own mobile devices. We took the decision to use audiences’ own smart mobile devices, rather than providing devices, in order to support larger audiences and to make the work easier to tour.

(2)

Use of social media style commentary as a mechanism for story-telling and audience interaction. We felt that engaging audiences in social media style interactions via their own smart phones would mirror (in a contemporary form) the activities of the Gerry-flappers of earlier times.

(3)

Application of archival media projected on-stage and displayed on the audience app.

(4)

Characters (i.e., singers) and media transitioning between screens, e.g., the proscenium arch stage, projection screens and audience app.

5.2 Refining the Approach

The second stage of development was signaled by the introduction of a performance venue—who committed to staging the premiere performances of Losing Her Voice—alongside an associated team of specialist theatre and performance practitioners (i.e., Artistic Director, Musical Director, Digital Visual Designer, Lighting Designer, and Set Designers). The technologists now became part of the wider production team for the opera, eventually assuming responsibility for deploying these technologies in rehearsals and performances, while the technologies themselves become just part of the overall show rather than being the primary focus of discussions. This extended team, in collaboration with Kelly oversaw the completion of the libretto and score, digital media design, deployable performance technology and early rehearsals with the principal singers, chorus and instrumental ensemble, which led to the premiere performances of Losing Her Voice.

The introduction of a professional production team led to a significant pivot in attitude to the integration of technologies into Losing Her Voice. At this point, an unshackled and open-ended design process was no longer appropriate, as a real public performance “that worked” – both artistically and technically—needed to be delivered. Furthermore, as the theatre production team had not been involved in the early explorations, it was apparent that they were not as invested in the ideas and had to be convinced of their value to the work. The Artistic Director—with overall responsibility for the realizing the performances both artistically and practically—was understandably uncertain about the introduction of untested (in their experience) and novel technologies into what was a very traditional performance artform as the following quote captured post-premiere performances highlights:

“I think there's something around trust […] it was clear to me that everybody knew their stuff, so it wasn't a trust in that sense. But I think for a director, the “work” and the eventual production is always our key obsession. If we can't see something or hear something or whatever it might be functioning early on, you're always likely to have an artistic person just going, “Ooh, not too much of that” and their default position being to whittle it away. Which I think is where I was at because what I didn't want was for us to arrive at the tech [technical rehearsals] and have all this phone interaction for it not work, disaster, the soufflé collapses.”

Echoing this, a post-performance interview with the Digital Visual Designer (DVD)—who had considerable experience in digital design, the creation of complex projection mapping performances and working with digital technologies in both traditional and non-traditional performances settings—explained how in part they were brought on board by the venue to work with the technologists, as someone who could speak their language and assist in smoothing the way:

“Theatre does like to have very specific ways of working; people understand that and if you rock that boat, I know people do struggle quite a bit. I've known that from when I first started, especially in my line of work, video was difficult to know where that lay within the production elements and for a good number of years it was problematic, but people have got used to it now […] I sympathize for [the technologists in Losing Her Voice] being in a kind of outsider role […] I guess I saw myself as the intermediate position between, which in a way I'm kind of useful in the sense of bridging, a creative and a technical team.”

Two workshops were instrumental in further refining the approach. Preliminary audience testing of selected scenes from the show took place at an interactive music festival hosted by Nottingham's Royal Concert Hall (November 2018). A group of invited opera scholars observed stripped down performances of select “work-in-progress” scenes, accompanied with some draft projections and use of the projected mirror. The principle focus of this workshop was to discuss and critique the subject matter of the libretto. The idea of explicitly designing extended “user journeys” (i.e., trajectories) through the audience experience was formally introduced to the team at this first workshop in the form of a diagram as part of a slide deck. This was iteratively refined throughout subsequent development, ending up in the final form described in the next section.

A second workshop was attended by members of the production team and a few invited scholars. Greater emphasis was placed on testing the interactive audience app within stripped down performances of select scenes. During this workshop decisions were made, primarily by the Artistic director, to reduce the number and scope of audience app interactions, in response to concerns that too much attention was placed on interacting with the mobile device at the expense of the on-stage action, “My obsession […] was about it having a purpose within the piece and not just people going out “oh wasn't that phone stuff clever. What was the opera about?” and that one fed the other” (AD). Specifically, active audience interaction via the audience app (i.e., liking posts, taking part in polls and quizzes) was subsequently reduced to two key scenes in Act 2, specifically, Scenes 2 and 4. Furthermore, in these scenes these interactions were to take place only while the chorus was singing to maintain the audience's focus on the principal singers. On the one hand, the AD decision to reduce the scope of technological integration diminished the work's apparent “technical novelty”, but on the other hand it was a pragmatic decision in the interest of the broader piece. This was (largely) a traditional opera to be experienced in a traditional venue and thus the audience interaction needed to work with and for the performance and the audience, rather than lead it. These decisions, consequently, led to a focus on how best to design the extended “user journey” to assure that audiences could engage, and know how to engage, with the audience app and to gracefully integrate it into the experience of attending a performance of Losing Her Voice, as we now present.

6 Designing the Audience Experience

We now present the final design and realization of the audience app in particular and associated elements of the opera and audience experience trajectories (see also the accompanying video figure²). Our overall aim by this stage was to enhance audience members’ experiences and sense of involvement in the opera through their own mobile devices, without compromising the opera itself. Our basic technical strategy was to create an app for audience members to use on their own mobile device before, during and after the performance. And our overall design approach was to apply the trajectories framework to the design process. In essence, we sought to create a canonical trajectory for this particular work that would sensitively integrate the audience members’ use of their own smart phones into the more typical experience of attending an opera and Losing Her Voice in particular, as discussed below.

6.1 A Night at the Opera

We start by considering the canonical temporal and spatial trajectories of a typical night at the opera, based on personal experiences of opera and theatre. This is summarized in Table 1. While some forms (such as the “promenade performance”) deliberately structure attendance very differently, we are concerned here with a performance that used this familiar structure. Our thinking is that the specific canonical trajectory of Losing Her Voice has to be appropriately embedded within this.

Table 1.

When	Where	What
Perhaps a long time before	At home or wherever…	Discover the forthcoming performance, decide to come, order a ticket
Hours or days before	At home	Reminded of or remember the forthcoming performance, perhaps do some preparatory research
An hour or so before	At home	Dress appropriately; gather phone, wallet, keys, ticket (if already received); travel to venue, chat if travelling together
Less than an hour before	Theatre ticket office and front of house	Collect/show tickets, collect/buy programme, perhaps order drinks, chat, listen for announcements, find correct entrance to theatre
Minutes before	Theatre auditorium	Find seat, remove coat, silence or turn off phone, settle, read programme

During performance	Theatre auditorium	Sit quietly and attentively (or otherwise), attempt to read the programme in the twilight
During interval	Theatre auditorium or front of house	Remain in seat and chat, read programme; or make way to toilets and/or bar, drink, chat, listen for announcements, return to seat; perhaps temporarily turn on phone
Repeat for number of acts
End of performance	Theatre auditorium	Applaud, collect belongings, chat, turn on phone, queue to leave
Just after performance	Theatre front of house	Chat, leave, take or discard programme
Shortly after performance	On way to and at home	Travel home, talk about the performance if together; read, file and/or discard the programme; perhaps follow-up research
Later	At home	Reminisce, perhaps follow-up research, perhaps plan to attend another performance

Table 1. Spatial and Temporal Trajectories of a Night at the Opera

There is clearly a strong temporal structure to attending an opera (or any other scheduled performance), with practical and logistical considerations giving a basic shape to the pre-performance experience. Within the venue there is a tight binding of temporal and spatial aspects, and a clear flow through arrival spaces and rituals to the seated appreciation of the live performance within the theatre's auditorium. Often at least one intermission allows a temporary escape to neighboring public spaces, while departure reverses the journey of arrival. The major transitions of the generic experience follow from this temporal and spatial ordering.

The social structure of the experience is also patterned by the same temporal and spatial constraints, for example, with the previously scattered audience members forced together as they navigate the arrival and seating process. The biggest point of social variation is usually whether or not someone visits the performance together with friends or family, which extends the performance's social influence directly to the travel and home stages.

The trajectories framework [8] also highlights the way that experiences, in addition to their spatial and temporal structure, may also move from one interface to another in what can be seen as ecology of interfaces or displays [23]. To design for the complete opera experience it is helpful to interpret “interface” or “display” broadly, as summarized in Table 2. Note that, with the exception of audience members’ own devices (e.g., smart phones), all of these “displays” are under the control of those producing and hosting the opera.

Table 2.

What	Where/When	Function/Interaction
Posters, flyers, brochures	Various/sometime before the show	Announcement of performance
Personal smart phone	In a pocket or bag or in hand/all the time	Personal email, web, apps, calling, notifications, social media, personal media…
Personal email	Wherever/especially before the show	Announcement of performance, confirmation of booking, pre-performance reminder
Ticket	Before and at venue	Confirmation, access, direction
Venue staff	Theatre front of house	Welcome, direction, assistance
Signage	Theatre front of house	Direction
PA	Theatre front of house	Timed prompts, directions
PA	Theatre auditorium	Timed prompts, directions; Supporting audio for performance
Musicians and singers	On stage in auditorium	Main content of performance
Set (including projection screens)	On stage in auditorium	Scenery, visual media for performance
Surtitle screen	Above stage in auditorium	Surtitles - accessibility
Someone else's smart phone	In their hand, pocket or bag/Theatre front of house or in theatre auditorium	(variously) awareness or prompt, shared resource, distraction

Table 2. The Extended Interface Ecology of a Night at the Opera

Within this generic structure of staged performances another critical trajectory is that of the performed work itself. This typically includes basic structural divisions (2 acts and 11 scenes—6 scenes in Act 1 and 5 scenes in Act two—in the case of Losing Her Voice), but perhaps more importantly many works—including most operas—have a temporal and/or narrative structure. This gives shape to the on-stage performance and (hopefully) the audiences’ emotional and intellectual journey through the work. Within this structure particular scenes and sections typically have distinct characteristics and intents, at least in the minds of the writer/composer and director. Although this particular trajectory is usually tightly bracketed by the (literal or metaphorical) raising and lowering of the curtain, some elements of it will shape at least the marketing material and any printed programme that accompanies the performance.

6.2 Design Goals and Strategies

Within the context of a night at the opera, and Losing Her Voice in particular, our overall aim and approach led to a number of more specific design goals and strategies for the app and associated audience experience, as described below.

Goal 1: support the usefulness and adoption of the app across the entire pre-, during-, post-performance experience. To address this goal, we included distinct functionality and content within the app to address each stage of the audience trajectory. We also ensured that the app was consistently signposted across the whole experience (e.g., from marketing material, the printed programme, front of house announcements) and provided a range of other supports.

Goal 2: sensitively interweave use of the mobile phone with the other elements of the opera performance. To address this goal, we carefully located periods of particular app activity (e.g., interactive social media posts) within and around the ebb and flow of on-stage action. We designed the app so that it could be used episodically (or continuously) and made it relatively discrete during the performance itself except at certain key points.

Goal 3: make the app consistent with and complementary to the work as a whole, including its themes, setting, narrative and aesthetic. To address this goal, we identified key organizing metaphors linking the app and the rest of the performance, in particular, the fan scrapbook and (anachronistically) social media styled as silent-film inter-titles, both responding to the key themes within the work of celebrity and fandom. We also ensured that the app was visually consistent with the rest of the work. In addition, it was decided that the opera should also cater to audience members with no suitable device (or who would not use one even if devices were provided). This was reflected in a further goal:

Goal 4: make the mobile app complement but not be essential to the show (i.e., staging it or attending it). To address this goal in relation to individual audience members we chose to treat the app like a kind of second screen, with the functions provided by the app complementary but secondary to the main performance, and where possible provided alternative channels to deliver its content, e.g., a printed programme, shared devices available to contribute photos before the show. To address this goal in relation to the show as a whole we also included a range of fail-safes and contingencies, so that the opera could continue even with a major technical failure of the app, or if few or no audience members chose to use the app, or if audience members attempted to undermine the performance through the app (e.g., by moderating user contributions).

There were a number of other pragmatic constraints that also influenced the design and development of the app: compliance with data protection legislation and research ethics; robustness; scalability, at least for a full house (100+ simultaneous users); compatibility, in particular with both Android and iOS mobile phones; and low-to-moderate network consumption (in light of potential concurrent usage and for audience members relying on mobile data). To address these pragmatic goals, we created a cross-platform web app (rather than, for example, a native app) and avoided personal data wherever possible.

6.3 App Design

Having outlined the design goals and strategies for the app we now briefly describe the appearance, core functionality and use of the audience app before returning to the complete audience trajectory in the following section. The app is configured for each individual performance (which has its own specific URL), and has two distinct modes: in-show and out-of-show. We describe each in turn, followed by the “social media” functions that provide interaction.

When the app is opened, its splash screen shows a gilt embossed fan scrapbook cover (Figure 3(a)), the same as the visual asset projected on-stage at the start of Act 1, an animation of the opening of the scrap book. The user touch required to “open” this scrapbook also gives the web app the user consent that it needs to play audio subsequently. In out-of-show mode the app provides free navigation of a collection of content pages (e.g., Figure 3(b)), styled as the pages of the scrapbook. Some of these replicate elements of the printed programme (e.g., synopsis, cast list) and some provide additional content (e.g., “Who was Geraldine Farrar?”). The app synchronizes with the server back-stage that controls the on-stage visuals and sound effects, so the app is always aware of which mode it should be in and more specifically (for out-of-show mode) whether it is before the show, during the interval or after the show: some of the content pages are only available in the interval or after the show.

Fig. 3.

In contrast, the app's in-show mode (e.g., Figure 3(c)) draws inspiration from silent film surtitles for its styling, and has a dark background to minimize screen glare during the performance. The user has no control over the in-show content, which is again synchronized to the performance. In most scenes (including all of Act 1) the in-show mode displays a context cue synchronized to appear at the outset of the current scene. It was agreed in development to minimize the detail of the content presented here so that it could be taken in “at a glance”. Typically, these were context setting cues, which included an archival photograph related to the setting of the scene and a single line of text that may highlight the time, place, character or significant narrative detail (e.g., Figure 3(c)). It was also decided not to offer surtitles on the mobile device, as the projection of surtitles above the stage addressed accessibility (i.e., for those not engaging with the app) and also it was felt to allow the audience to more easily view both the surtitles and performance itself, compared to looking down at their phone.

To enhance audience members’ sense of involvement in the opera we wanted to include elements of interaction and participation, not just give them information. As noted in goal 3, with the composer we chose the metaphor of social media because of its resonance with the themes of celebrity and fandom in the opera. But to integrate this visually and aesthetically they were styled as silent film inter-titles (see Figures 4 and 5).

Fig. 4.

Fig. 5.

We integrated a number of social media elements into the app—and the performance—as described below.

In out-of-show mode (before the show or in the interval) a “Make a Profile” option allows the user to select a name for themselves. This name was chosen from 100+ period first names and last names rather than being free text to avoid the need for moderation of names (although it is possible that two people will choose the same name). This gives the user a chance to associate themselves with the app.

There is a “social media post” view that is available both in out-of-show mode (Figure 4(a)), and at certain key point during the show (Figure 4(b)). There are three types of “social media post”: simple posts, polls and quizzes. An audience member can “like” simple posts, and can vote in polls and quizzes (see Figure 5(a)) which give continuous feedback of votes and outcomes.

Also, in out-of-show mode (before the show or in the interval), a “Take a Selfie” option allows the user to take a selfie with the phone camera (or select an image from their phone) and upload it. This option is not available during the performance itself because of the potential disruption and the low lighting in the theatre. Uploaded pictures are moderated (by a human, before the performance and again during the interval) before appearing projected on stage in Act 2, Scene 4. There is also a default set of pre-collected selfies and stock images which are during this scene if there are not enough audience-provided selfies or there is a technical failure in the link to the app.

The app can also play audio in in-show mode, which is used at the very end of the show (see below).

6.4 Audience Trajectory

Having presented the design of the app we now describe how we sought to support the complete audience trajectory through the performance of Losing her Voice.

As already described, the app content was synchronized in relation to the performance, providing free access to distinct content pre-show, in the interval and post-performance, with specific content and interaction delivered in sync with the performance itself in each act. So the basic use of the app can be seen as a trajectory through five stages: pre-show (programme, pre-show quizzes and taking selfies), Act 1 (context cues), interval (extra programme content, polls and quizzes including on-stage projections and taking selfies), Act 2 (context cues, posts, polls, recorded media) and post-show (programme and post-show questionnaire).

We also designed the fine-grained use of the app to integrate gracefully with the narrative of the performance and the in-theatre experience (goal 2), as described below. Before the performance, in addition to a digital version of programme guide (e.g., background, cast, and synopsis) some quizzes and polls are distributed to the mobile app. These pre-show interactions are designed to begin the process of establishing the language of interaction with the mobile phones for the performance itself, but they also begin to position the audience members as fans of Farrar; offering detail about her life and the fandom that surrounded her.

Throughout the first act the audience is in passive engagement with Losing Her Voice. The action is played out in traditional form behind the proscenium. The visual design of the projections displays fan scrapbook montages throughout, thus the on-stage action appears to take place within the pages of “Gerry-flapper” fan scrapbooks, meaning the audience observe Act 1 through the eyes of a “Gerry-flapper”.

The interval presents audience members with another opportunity to establish their “Gerry-flapper” persona (e.g., choose a name and contribute a selfie) if they had not done so before Act 1. As was the case pre-show, interactive social media polls and quizzes pop up throughout the interval, however, now their content reflects the narrative witnessed in Act 1, prompting audience members to engage from the perspective of a fan answering questions that seek opinion, speculation and gossip about the events Farrar has experienced so far. Furthermore, during the interval these social media posts along with the aggregate results of audience member interactions with them are projected across the on-stage screens (Figure 4(a)), visible to anyone who stayed in the theatre during the interval, and intended to prompt and promote responses.

In Act 2 Scene 1, the characters of Farrar and her silent movie Carmen co-star transition from on-stage, as embodied singers, into the silent movie screen in their historical, archival footage form (i.e., scenes from the film Carmen). As Act 2 progresses the audience are encouraged to be in (increasing) active engagement with the unfolding action. The social media interactions appear for the first time in Act 2, Scene 2, then again in Scene 4 (Figure 4(b)), echoing ideas and sentiments from the libretto in a “fan” voice (e.g., “They are the ideal lovers!”). There are also two polls which invite a “fan” response to the narrative at that point (“Is it time for Gerry to leave the Metropolitan?”, “What would you like to see Gerry do next?”), although they do not change the subsequent performance.

The audience-provided selfies are incorporated into the projected visuals in a dramatic scene towards the end of Act 2 when Lou Tellegen is progressively obscured by selfies of Geraldine's “fans” (Figure 4(c)). These audience members agreed to be an active element of the performance when they contributed their selfie pre-show or in the interval, not knowing exactly how or when these would be used. During this moment the audience occupies both the space of the auditorium, as a spectator, and the stage as a performer.

The final scene of Act 2 concludes with embodied Geraldine, now an older woman, reflecting back on her career. In what is the climatic moment of the performance, the chorus takes out their mobile phones that are running the app, move off stage into the aisles of the auditorium and hold them out over the audience. As Geraldine stops, a recorded version of the “Habanera” aria from the opera Carmen (a role strongly associated with Farrar) is broadcast via the app (on both chorus and audience phones) producing a swelling of sound from many mobile phones throughout the room that grows in response as Geraldine's part fades away. The chorus’ use of phones serves to acknowledge the app explicitly within the performance, to encourage audience members to follow suit and unmute their phones, and also provides a contingency in case few or no members of the audience have the app open (and not silent) at this point. In this way, the chorus and the audience (co-opted as “Gerry-flappers”) jointly observe Farrar's transition from stage to broadcast audio.

At the end of each act the app switches back to out-of-show mode, and defaults to a particular page: “Take a Selfie” in the interval, and a page which prompts the visitor to complete an online survey about their experience after the performance. The latter is a concession to the research needs of the project rather than the opera per se.

6.4.1 Supporting the Audience Trajectory.

Beyond this we also wanted as many people as possible to use the app (goal 1) and to make it clear that the app was a first-class part of the performance. We were conscious of people's normal expectations about the non-use of phones in public performances, and also that a significant proportion of our audience was likely to be older and less comfortable with technology (given the typical audience that attends opera performances and indeed this venue).

So, considering an audience member's complete end-to-end visit trajectory (Table 1) and the range of displays involved (Table 2), for each key stage or activity we sought to identify potential support mechanisms for effective use of the app and to mitigate potential barriers which are shown in Figure 6.

Fig. 6.

These supports included the following: clear messaging in pre-show marketing material about the mobile phone element of the performance; explicit reminders in pre-show e-mails to charge and bring a smart phone if possible; comments interleaved with the synopsis in the printed programme highlighting when the app should be used; the theatre's Wi-Fi network was upgraded for the performance; a flyer handed to every audience member on arrival, and placed on tables in the foyer, explaining how to join the Wi-Fi network and access the app (with a QRCode and short URL); dedicated support staff (“Gerry-Appers”!) in the foyer before the show and in the interval to help people set up the app and submit selfies; charging stations and external batteries available in the foyer if someone's phone had limited charge; PA announcements reminding people to keep their phone to hand (but initially silent); social media messages on-stage in the interval and Act 2; and the chorus’ visible use of the phones at the end of Act 2.

Having described the design and realization of the mobile app and associated audience trajectory we now describe how we collected evaluative data from the premiere performances of Losing Her Voice.

7 Data Collection

Losing Her Voice premiered at Nottingham's Lakeside Theatre, with two performances, one on the 6th and one on the 7th April 2019 (the dress rehearsal was on 5th). The performances were advertised to the public and ticketed as part of the venue's normal programme, although tickets were free (a condition of one of the project's funders). The first performance audience number was 125 and the second was 131 (the dress rehearsal was watched by around 20 staff/students).

Data from each performance was gathered in the form of: system log files; audience questionnaires (i.e., article and online via the app); audience interviews; and interviews with the composer, artistic director and digital visual designer. These are described in more detail below. The study was approved by the School of Computer Science Research Ethics Committee.

7.1 System Log Files

The audience app was written to send interaction events (e.g., screen navigation) to the public audience server, which logged these to file together with records of all data relayed between the audience apps and the backend server. The logs contain most audience app events, but the last few event(s) when the app is closed or the device locked can be lost (this is more common with some browsers/handsets than others).

The log files from both performances (up to 7 am on the morning after the second performance) were processed using a custom script³ to generate a per-app breakdown of user actions (e.g., likes and quiz responses) and time spent on each page/view. A best effort is made based on logged events to work around missing log entries. However, the times spent on a view may be over-stated if the event marking the closing of that view is missing. Only clients active around the time of each show (+/−1 hour) were included.

The chorus’ devices were excluded from analysis (based on their distinctive pattern of activity). Because the logs are anonymous it was not possible to identify researchers’ devices—who were also stepping through app interactions throughout—(2 per show) with confidence, and these are therefore included in the presented results.

7.2 Audience Questionnaire

A post-show questionnaire was created to probe audience member's experience of the performance, including the app (some of the main questions are listed in Table 4). After the show the mobile app presented audience members with a link to the questionnaire (online). Article versions of the questionnaire were also distributed to audience members after the show and the “Gerry-Appers” encouraged audience members to complete them. In total 10 online questionnaires and 30 article questionnaires were returned (approximately 15% of the total audience). One of the questionnaires was completed after the dress rehearsal; 20 after the first public performance; and 19 after the second.

The responses were combined in a single spreadsheet. The quantitative responses were summarized using SPSS. The questionnaire open responses were combined with the interview transcripts for thematic analysis (see below).

7.3 Audience Interviews

The “Gerry-Appers” also approached audience members immediately after the show asking them to give a short interview (5–10 minutes). They were then taken through a brief consent process and the interview was recorded with an audio recorder. In total seven interviews were conducted, three with individuals and four with pairs (i.e., 11 interviewees in total). The interview was semi-structured and probed the experience of the performance as a whole and in particular whether or not the various elements of the mobile app and the projected visuals enhanced or detracted from the performance, and what if any pre- or post-visit activities people undertook.

The interviews were transcribed and anonymised. They were then combined with the questionnaire open responses and imported into NVivo. All responses were systematically reviewed and all quotes (codes) were marked up, paying particular attention to comments about use or otherwise of the app, responses to the social media elements and any pre- and post-show activities. Codes were then iteratively grouped to reflect common concerns, i.e., emerging themes. These were triangulated against (aggregate) quantitative responses and logs, and illustrative quotes identified for presentation. Where quotes are given in the findings [Qn] (e.g., “Q1”) indicates (open) questionnaire response n, and [xn] or [xn/m] (e.g., “C1”) indicates Interviewer x interview n, and if interviewing a couple, person m (i.e., 1 or 2).

We also interviewed the opera's artistic director, graphic designer, and composer after the performance (see also Section 5). However, the following analysis is based on the audience experience.

8 Findings

The majority of our findings are from the system logs, audience interviews and questionnaire responses. We begin by giving a brief characterization of the audience (from the questionnaires) and an overview of how the app was used (from the system logs). We then present the main themes emerging from the qualitative data.

8.1 The Audience

As noted, we had questionnaire responses from above 15% of the total audience (N = 40). This included 75% who had access to the app and 25% who did not. From the system logs it appears that approximately 62% of the audience is likely to have used the app, so the questionnaire data may slightly under-represent those without the app.

Respondents reflected a fair balance of gender attending the performances (63% female, 37% male). All respondents were over 18 with a good spread of ages (17.5% 18–29 years; 37.5% 30–49; 30% 50–64; and 15% 65+). There was also a broad range of self-reported knowledge of both technology (7.5% no knowledge; 32.5% a little; 32.5% a moderate amount; and 27.5% good knowledge) and of opera (12.5% no knowledge; 42.5% a little; 12.5% a moderate amount; and 32.5% good knowledge). The vast majority enjoyed the performance (2.5% a little; 12.5% a moderate amount; and 85% a lot).

8.2 App Use

From the analysis of the system logs (see Table 3) there were 160 active app clients over the two shows, i.e., approximately 62.5% of the audience (N = 256) used the app. 72 selfies were contributed, i.e., 28% of the audience or 45% of app users (see upper part of Table 3), 54 clients liked a post and 94 responded to at least one quiz or poll. 45 clients were active during the audio play-back in Act 2 Scene 5 (although we cannot tell if they were muted or not from the app/logs).

Table 3.

	Use app	>10 seconds	Set name	Selfie	Like	Vote	Sound
For all audience	62.5%	61%	37%	28%	21%	37%	18%
For app users (at any time), Of which:	100%	98%	59%	45%	34%	59%	28%
Much earlier (>1 hour)	18%	18%	7%	3%
Just before (<1 hour)	86%	79%	41%	29%		14%
Act 1	42%	38%
Interval	80%	71%	11%	13%		43%
Act2	53%	47%	1%		34%	27%	28%
Shortly after (<1 hour)	51%	23%
Much later (>1 hour)	29%	16%

Table 3. App use and Feature use, Overall and by Time, from System Logs

The lower part of Table 3 shows a breakdown of how the app was used at various stages in the complete performance trajectory. In each case it shows the percentage of clients (i.e., app users only) engaging in that activity in that time period. The app being visible for at least 10 seconds (second value) is intended to exclude very brief or accidental waking of the device. It can be seen that there is relatively little app use much earlier (more than an hour) before the show (18% of those who ever use the app). Use is greatest just before the show (79% >10 seconds), presumably on arrival at the venue, then in the interval (71%), then Act 2 (47%), then Act 1 (38%), then shortly after the show (23% less than one hour), then much later (16%). Most selfies are contributed just before the show (64%), with 29% in the interval and 7% further in advance. The polls and quizzes in the interval received the highest rates of response (43% any vote), followed by those in Act 2 (27%) and then the quizzes before the show (14%). Slightly more people (34%) liked posts than voted (27%) in Act 2.

8.3 Audience Self-reports

Having given an overview of when and how the audience app was used we now turn to the audience self-reports (questionnaire and interview responses) to gain more insight into the audience experience. We organize our findings in relation to four aspects of the app and performance: responses to the interactive elements, responses to the other media elements, reasons for non-engagement with the app, and pre-/post-show activities.

8.3.1 Interactive Elements.

To set the scene, from the questionnaire responses most people (63.7%) felt more involved compared to similar shows (5.3% slightly less involved, 21.1% neither, 24.2% slightly more involved, 39.5% much more involved; N = 38 − 2 missing responses). Table 4 summarizes responses to some of the other main questions. The app was felt to enhance the show median “a moderate amount” (Table 4(i)). The app was felt to be distracting by a minority of people, median “not at all” (Table 4(ii)).

Table 4.

Question	Not at all	A little	A moderate amount	A lot	N
(i) To what extent did the mobile app enhance the performance?	13.5%	27%	45.9%	13.5%	37
(ii) To what extent did the mobile app distract from the performance?	56.8%	35.1%	2.7%	5.4%	37
(iii) To what extent did the social media posts in the app enhance your experience of the opera?	14.3%	39.3%	14.3%	32.1%	28
(iv) To what extent did contributing a selfie enhance your experience of the opera?	27.3%	31.8%	4.5%	36.4%	28
(v) To what extent did the pre-show and interval information in the app enhance your experience of the opera?	21.4%	32.1%	32.1%	14.3%	28
(vi) To what extent did the app playing music at the end enhance your experience of the opera?	17.6%	17.6%	32.4%	32.4%	34

Table 4. Selected Questionnaire Items

The performance was enhanced by the use of social media, median “a little” (see Table 4(iii)). Some people appreciated the sense of participation that interacting with the social media elements gave them: “realizing I could participate with the social media posts was so fun and engaging” [Q13]. For some this also served as a distinctive and specifically novel element of the experience: “that I was seeing something that wasn't just another attempt to do another opera in the same form all other operas have been in” [C1]. The same person specifically enjoyed the interval quizzes as a way to reflect and test themselves: “it was interesting to see what we'd picked up from the performance and what hadn't” [C1]. But two respondents reported that the time allowed for the quizzes was not long enough.

Questionnaire responses revealed the performance was enhanced by the contribution of selfies, median “a little” (see Table 4(iv)). People who contributed selfies were generally positive about them. For example, having contributed a selfie could create anticipation: “I thought that was good fun and I looked forward to seeing it at the end” [A1/1]. People also appreciated seeing themselves and others on the screen: “that was fun to see people you know [C2]”. For some people it helped them to feel more part of the show: “it made us sort of feel part of it in some way.” [C1] And for some taking selfies provided a sociable activity before the show started: “[of taking selfies] we arrived a bit early so it was nice to kind of have something to do and then encourage all the other people around us to make sure they were doing it as well so” [C2]. But some people specifically chose not to contribute selfies because they didn't feel comfortable with being seen: “I didn't do one, I didn't want my face to be up in front of everyone” [G2/2].

The multi-display aspect of these elements—reaching all apps at the same time, and also the projected screen in the interval and Act 2—sometimes prompted use of the app: “I saw a couple of people had their phones there and I could see some things flashing and so that prompted me to turn on mine as well.” [C2].

8.3.2 Other Media Elements.

As well as the social media elements the app provided access to various show-related content. According to the questionnaires, the performance was enhanced by this information on the app median “a little” (see Table 4(v)). From the logs, we looked at the proportion of people with the app who spent at least 10 seconds with each content page visible. The most viewed “content” was the page where the quizzes appear in the interval (65%) and before the show (40%). The introduction page was also viewed by 34% shortly before the performance. A smaller proportion viewed the synopsis, cast, and about Geraldine pages. Little use was made of the help page. The limited level of use of show-related content may partly reflect that fact that all audience members were also given a copy of the printed programme, which provided an alternative route to most of this information.

At the end of Act 2 music starts to play from the app, picking up from where “Geraldine” finishes singing. As already described, the chorus also has the app open at this point in the auditorium, even if the audience members do not have their phones on. From Table 4, 28% of apps were active around this time, although some devices may still have been muted. According to questionnaires the performance was enhanced by music from the app, median “a moderate amount” (see Table 4(vi)). So this seems to have been quite effective, and one person commented specifically on this: “The music playing at the end was really effective” [Q25].

8.3.3 Non-use in the Performance.

One of the major themes to emerge from the audience interviews was around non-use of the mobile app. From the app use logs we looked specifically at the proportion of app users engaging with the social media elements (liking or voting) in the show (i.e., in Act 2), outside the show, or both—see Table 5. It is apparent that many people chose not to engage with social media on the app during the performance itself, even if they used it at other times, while others engaged with the social media only during the performance. We explore the various reasons given for this below.

Table 5.

	In and out of show	In show only	Out of show only	Total
N	42	21	35	98
Of all clients	26%	13%	22%	61%
Of clients voting or liking	43%	21%	36%	100%

Table 5. App use for Social Media, in and out of the Show

The commonest reason given for avoiding the app during the performance was the precedent and usual expectation that phones should not be used during performances, referred to at the start of this article: “because every time we go to a theatre, you know, it's there that people say to you please turn off your mobile phone and you get used to that and now i'm more comfortable with my phone turned off and put in my bag” [A1/1] Although people were made aware that there was an expectation—a license—to use their phones some still felt that it was not clear or explicit enough: “I think if there was some kind of cue that said take your phones out now or something like that, that might have felt, I would have been a bit more open to that I think” [G2/1].

With only a fraction of the audience using the app, there was also sometimes a lack of social confirmation, i.e., no clearly visible precedent: “No one around me was using the app during the performance so didn't feel able to use it myself” [Q34].

More positively several people indicated that they were too focused on the other aspects of the performance to consider using the app: “I decided not to use the app during the performance itself as I was *transfixed* on the performance, music and visuals” [Q21].

There were no specific comments about being distracted by others using the app during the performance. But at least some people did find it distracting to use themselves, and moderated their use as a result: “I used it a little bit … but I did find it a bit distracting at times” [G1/2] Similarly others anticipated that using the app would distract them from other parts of the performance and so avoided it during the performance: “It felt almost like it would have been distracting to me in terms of the performance to get it out” [C1].

Some audience members would have used the app but experienced specific technical barriers. Some of the audience did not have access to appropriate devices: “I do have a mobile phone but it's very simple and I haven't got it with me” [P2]. Others did not try to use it because they doubted their ability to do so: “Urm, I'm not very good with technology” [J2/2]. In a few cases the app failed to work on particular devices, as observed during one-to-one support during pre-show and interval. In some cases people could not get the app working on their phone in the available time: “No, I got it half loaded and then I gave up, I just went in because the bell rang” [J2/1]. Or had trouble getting back to the app during the performance, even after setting it up initially: “But I'd lost app then, I don't know where it had gone” [A1/1].

8.3.4 Pre/post Performance.

As we have already seen from Table 3, on the whole, people's interest and activity was focused on the performance itself, shortly before it and in the interval, with relatively little extension before and after. From the interviews, if people prepared for the show it was usually quite minimal, e.g., checking the address of the venue or at most “I just read the write up of it, you know the storyline, you know to know what was going on” [J2/2]. In some case this was a deliberate self-limitation: “You don't want to know too much about it before we get there do you?” [A1/2].

Similarly, there was some limited interest in following up after the performance. Some people specifically wanted to see it again: “I feel like I've missed parts, so I'd like to come back and give it another whirl” [J2/1]. And several people suggested that they might follow up by finding out a bit more about the character or the story: “I will probably go home and google them, that's what I normally do” [G1/2]. But there was little beyond that; as one respondent without the app said of the article programme: “We will read it when we get home… and then it's put away with everything else and probably won't come out again” [A1/1].

8.4 Reflections on the Audience App

We also interviewed the composer (Kelly) and artistic director's (AD) after the premiere performances of Losing Her Voice, to capture their reflections on the process and performances themselves and future work. Here we summarize their immediate reflections on the integration of the audience app into the performance of the work.

In response to the premiere performances, Kelly (composer) felt, “the phones worked really well […] It's nice to have people do it on their own phones, because they're engaging with the work on their personal device, and that's very meaningful and it's probably the same device that they use to do their own social media.” She went on to highlight that, “even the audience who weren't participating (with phones) seemed to recognize that something was actually going on, when I heard people walking out and talking that it was something they were responding to.” Kelly's fears of intrusive handset notifications within the performance were not realized, and she highlighted that the audio and text notification to silence handsets at the start of each act may have helped mitigate this. However, she also speculated that the older “opera” audience probably affected the uptake of the audience app: “Opera audiences are generally quite well behaved I think, so they are actually more likely to be afraid to make noise and therefore not use their phone—in a way they have to be encouraged”. Furthermore,

Similarly, the AD also considered the audience app to have integrated and contributed effectively—despite his initial concerns—acknowledging, “[it] would be interesting do another project. Now I've seen it, now I know it works and you trust it, the artistic minds can go, “ah”, in the same way we know we can trust lighting […] so actually, we find ourselves going, “how can we add, can we add, can we add” rather than, “can we take away and take away””. Kelly, echoed this sentiment, “I thought actually that we hit that balance pretty well […] I definitely wouldn't want to add it into every scene”. She went on to reflect that, “I'm hoping to go forward with other projects building a work around a phone, I think that would be an interesting idea”. The AD then went on to reflect on how reliant a performance of Losing Her Voice is on the audience app and associated technology. While he could envisage a stripped-down, or concert version of the work, he noted however that, “every note that Elizabeth wrote, she was imagining the visual language, be that the phone's, be that the projection. So even if you'd never seen it with all of that, I think, there's a sense that it needs all the elements to really fly in the way that it did and the way that she wants it to.”

8.5 Summary

In summary the audience was diverse in age and experience of technology and opera, but enjoyed the performance and generally felt more involved than in other past performances. More than half (62.5%) of the audience (N = 256) used the app, which was generally felt to enhance the performance, while some also found it a little distracting. Barriers to using the app include lack of a suitable device, apparent incompatibility with particular devices, and lack of knowledge, confidence and/or time to set it up and use it. Of the people who did use the app a significant minority chose not to use it during the performance itself, for example because they felt unsure about using it, i.e., observing social norms, or were focused on other aspects of the performance or were concerned that it would distract them. Contributing selfies was generally felt to enhance the performance, with 28% of the audience (45% of app users) contributing them. This was also a fun pre-show activity for some. Social media posts and quizzes are also generally felt to enhance the performance, with 38% of the audience (59% of app users) liking and/or voting at some point (in the show and/or the interval), but relatively few engaging with them pre-show. The static content in the app was also felt to enhance the performance, although to a lesser extent and by fewer audience members. The media played from phones, however, enhanced the performance more. There was generally limited interest in the app after the performance or well before it. The composer and artistic director both felt that the phones had worked well within the performance but wanted to develop this further in future works.

9 Discussion

Having presented our findings in relation to the app from the premiere performances of Losing Her Voice, we now reflect on our original design goals and strategies. We suggest that, on balance, the weight of evidence from our premiere performances is that the combination of strategies we adopted towards embedding the mobile experience within the show and wider opera-going experience led to an enjoyable experience that enhanced the sense of involvement, with all the specific elements of the app making some positive contribution to the performance as a whole. The audience's reported enjoyment—both with and without the app—and low levels of distraction also suggest that the opera was not “compromised” by the app, though there were some exceptions.

Stepping back from the specifics, the research focus that emerged from the early design process and that was carried forward into the public deployment concerned the subtle use of established digital technologies—mobile phones and on-stage projections—to augment a familiar cultural form. This necessarily means that our findings speak more to the user experience challenge of subtly interweaving familiar technologies into existing experiences than they do to innovating radically new experiences. Subtle as this interleaving may be, we contend that it is an important and difficult challenge for HCI nonetheless and offer four general insights into how it can be achieved.

9.1 Encouraging Adoption

Our first specific goal was to encourage adoption of the app by the audience. We included static programme-style content in the app to help to give it a clear function at any time, before or after the show, and some of audience found this content helpful. Relatively few people accessed the app before arriving at the venue. And after the show, in particular, most audience members seemed to feel that it was all over, and there was limited appetite to follow up after the performance. We brought some elements of interaction (selfies and quizzes) into the pre-show and interval experience, and this was quite effective for those who engaged with it.

Recommendation: Prioritize pre-show and interval as the key opportunities to get the audience “on board”. Consider specific pre-show and interval activities (e.g., in this case quizzes and selfies) to motivate device use and engagement. However, be aware that audience members may also wish to take these as opportunities for their own activities, socializing with those they came with or attending to their own communications, which may even suggest extending the time available.

We made a deliberate choice not to provide loan devices (as we have done at previous events) but to rely on audience-brought devices, in order to test the current feasibility of this approach at scale. As noted, a majority of the audience appeared to engage with app, although a significant minority did not. We promoted the app to audience members through the original marketing material, pre-show emails, the programme and a flyer given to everyone on arrival. In Dialtones [25] audience members were required to register their phone before the performance at terminals situated outside of the auditorium, which in turn issued tickets for specific seats. While Losing Her Voice did not require device registration, nonetheless, we used the pre-performance and interval period to encourage handset configuration and use (for example, setting up Wi-Fi, linking to the app, adjusting device volume, and so on). The on-site “Gerry-Appers” provided one-to-one encouragement and technical support (to 10 seconds of people) before the show and in the interval. We provided charging stations and external chargers but only two of each were used, suggesting that power was less of an issue than we feared it might be (or that people were well prepared because of the pre-show email reminders). There were a few people who made some attempt to use the app but failed, either because it was incompatible with their device (2 people) or they lacked the time or expertise to get it working (and did not get one of the support staff to help). We also experienced an annoyance on some devices that entering the short URL without “https://” at the start would only do a search, and not attempt to open the URL. Positively, we found that about 50% of devices’ built-in camera app would read the QR code.

Recommendation: achieving high levels of participation remains deeply challenging due to multiple factors, each of which might require its own solution. If very high participation is desired, then—for a general audience—loan devices should be made available. For a general audience, in-person support will enable some audience members to use their devices who would not otherwise (especially those who are less technically experienced).

As we noted from the interviews there was a significant minority of app users who chose not to use the app during the performance itself, and some suggested that more explicit instructions (e.g., PA announcements) might have persuaded them to. But others would still not have used the app, either because of a principled decision to focus on other parts of the performance, or because of a lack of confidence, for example, in being able to set their phone to silent, or an entrenched behavior that technologies should not be used within a performance. Overall, from the interviews and other informal conversations it seemed that the majority of people who did not engage with the app could not have been persuaded to do so, whatever we had done. And this may reflect the make-up of the audience for an opera or classical concert (which is the venue's more usual offering).

Recommendation: Assume that some of the audience will still not use a personal device and be careful about over pressuring them, explicitly or implicitly, into so doing as this runs the risk of spoiling their experience and excluding them.

9.2 Interweaving Across Multiple Timescales

Our second specific goal was that use of the mobile phone should be sensitively interwoven with the other elements of the opera-going performance, and Losing Her Voice in particular. As already noted, we used the insights of the trajectories design framework [8] to help us to work through this challenge. In addition, from tests during the development process we were conscious of how difficult it was to split attention between the app and any on-stage performance, which is similar to the attentional challenges experienced when using a second screen while watching TV [28].

We sought to facilitate the interweaving of app use and other performance elements at three time-scales. First, at the finest timescale, we sought to make individual in-theatre interactions self-contained and brief, so that audience member need only take their eyes off the stage for a few seconds at most. Specifically, the context cues for each scene comprised a single image and a few words of text. Each post was quite short, and liking it took a single screen touch. The two polls in Act 2 were the most demanding interactions, involving reading a short prompt then choosing one from three or four short answers, and two touches (one to make a choice and one to submit it). 10 seconds was allocated to make a response, but as noted from the interviews some people found this time too short for a considered response. 10 seconds was more than enough to read and submit a response, but this assumes audience members see and start engaging with the posts as they arrive. Likes and votes have the additional benefit that they need no moderation, and visual feedback to users typically appears in less than 100 ms when using the Wi-Fi network. We ruled out free text entry at the development stage because of the extra attentional demand it placed on the user to enter. Moderating free content would also introduce delays in displaying that content (projected on-stage), meaning delayed feedback and a potential misalignment of audience content to unfolding on-stage action, which might undermine any sense of direct involvement. For most people this timescale of our chosen modes of interaction seemed be effective, in that it was felt enhance the performance. But some still found it at least a little distracting or chose not to use the app during the performance on this basis.

Recommendation: Craft in-theatre mobile interactions to be simple, direct and responsive, so as to minimize the split of attention with events on stage.

Second, we aligned these individual interactions with parts of the performance where the split of attention was felt to be less problematic. Specifically, the composer and director requested that they be timed during the parts sung by the chorus rather than when the principals were singing, so that the audience could focus entirely on the principals. This request came about in response to our experience of testing the app during performances of Act 2 Scene 4—the social media intensive scene—at the second design workshop (see Section 5.2 and Figure 1). Similarly, the recorded “Habanera” aria broadcast from the app at the conclusion of the opera does not start until the principal stops singing. Individual interactions were also grouped together in particular sections of the performance, so the audience might keep the app to hand during those periods, but ignore it at other times. These periods of interaction are signaled and supported by the appearance of social media within the on-stage projections, and align with specific scenes and parts of scenes in which the chorus and audience (as “Gerry-flappers”) are given voice. These strategies seem to have been effective as every aspect of the app was generally found to have enhanced the experience as a whole. While some people did find the app a little distracting—mainly to their own experience of other aspects of the opera – everyone seemed to be happy to accept the app as an integral part of the experience. This is in contrast to some other examples. we In “ADA FTW” [13] the text-message based interaction lacked such clear linkage and was generally felt to be disconnected from the core performance. Similarly, a review of the hybrid opera Death and Powers [40]—which included remote audiences interacting via smartphones—noted that the smartphone interactions were, at times, “unnecessarily distracting”, or offered little to enhance the on-stage action, prompting further design iterations of the work. On the other hand, Barkhuus et al. [2] observe that a second screen can provide an alternative performance-linked focus in those moments when audience attention diverges from the stage.

Recommendation: Place in-theatre mobile interactions carefully with reference to the moment-by-moment specifics of the performance. Avoid critical moments of the performance, consider grouping them together and signal them within the performance itself.

Third, at the coarsest timescale, we brought some elements out of the on-stage performance entirely. In particular, the selfies could only be taken and contributed before the show or in the interval. As well as taking significant time and attention, taking a selfie generally requires reasonable lighting (or a flash) and so would be impossible or highly disruptive in the darkened theatre. This temporal decoupling also gave us the chance to moderate the selfies (completed by the end of the interval), and created an additional “fun” pre-show or interval activity, at least for some audience members. Similarly, having the series of quizzes and polls before the show and in the interval, while arguably not an essential part of the opera, did give the audience the chance to recognize and rehearse in-theatre interactions and also to feel more involved, and was clearly taken up by a reasonable number of people who chose not to use their phone during the performance itself. These instances of “out of performance” engagement were less time critical, thus offered opportunities for richer audience expression through the system. Troyer [42] observes that a system's ability to facilitate audience expressiveness is directly related to increased frequency of use and engagement. This these “out of performance” interactions served to engage audience members with the work while within the performance we could carefully balancing the moment-by-moment specifics of interaction.

Recommendation: Extend interaction beyond the on-stage performance (temporally and spatially), to support “onboarding” and especially save more complex or longer interactions for these moments.

9.3 Complementing the Work as a Whole

Our third goal was that the app should be consistent with and complementary to the work as a whole. As already noted, one key response to this was the identification and adoption of the fan scrapbook metaphor for the organization and visual design of the overall app and the out-of-show content, responding to the key themes of celebrity and fandom. The adoption of a social media metaphor also fitted well with a contemporary response to these themes, while the adoption of visual design elements from contemporary silent film inter-titles helped to translate the social media element into the period of the work, as well as tying visually to the use of black and white silent film material in the second act.

Recommendation: Style the app consistent with the work (and associated marketing materials).

There are many interaction mechanisms that can be used in interactive theatre and performance, including free text messaging [13], augmented reality [31] and gestural interactions [32], but this is not to say that all are relevant or helpful. In this work, the idea of social media was realized through a number of specific interactions: choosing a name, responding to posts and quizzes and contributing a selfie. The narrative's exploration of the “new” and democratic media of sound recording is reflected in the playback of media across multiple phones at the conclusion of the second act. We speculate that, in the same way that “continuity” is already a key issue in staging plays and shooting films (including ensuring that technologies that appear make sense as part of the story), so this will be true of wider audience interactions. Having immersed the audience in the world of the story it may then be problematic to ask then to use technologies that inappropriately jar with this, at least if these are no so familiar that they largely go unnoticed.

Recommendation: Select interface and interaction metaphors that are sympathetic to the work so as not to break continuity.

At a finer-grained level, we have also explained how the content of the social media elements (posts, quizzes and polls) was authored directly based on the libretto and the narrative arc of the opera. And the music played back by the app is the same as fragments of the “Habanera” aria employed at several other points in the opera (the composer employs this aria as thematic device throughout, given its strong association with Farrar's career on stage and on screen), all be it processed to sound like an old recording. At this point, the choir's visible use of the same app further underlines its integrity with the work. We also discussed whether the choir could be seen to use the app earlier in the performance, but it was decided in rehearsal that this would create too many demands on top of singing and acting. Corria et al. [15] also stress the importance of “connecting layers” within media-rich or interactive performances, where layers refer to all elements that constitute a performance including human performers, sets, projections, interactive technologies to form a coherent whole.

Recommendation: Create specific links between the on-stage work and the app experience where possible to reinforce continuity.

9.4 Non-essentiality

Our fourth specific goal was that the mobile app should complement but not be essential to the show. Behind this goal lie two underlying concerns: will the app work? And will the audience use it?

There are also two perspectives on this: whether the show as a whole will be successful only if enough of the audience use the app (and it works); and whether an individual audience member will enjoy the show only if they use the app. In terms of the show as whole, we have already identified some of the contingencies that we put in place in case the app failed or very few people used it. For example, we pre-loaded stock and crowd-sourced selfies images that would appear if selfies from the current audience were not available, and the chorus phones would play the recording at the end of the show even if no audience member did. The polls in Act 2 would clearly have been less effective if no votes had been received, but neither these nor the likes were intended to directly influence other aspects of the performance. In the event the app worked perfectly, and the level of use was good so this was something of a non-issue.

In terms of making the app non-essential to the individual audience members, we have already described how we gave the app a secondary role within the theatre itself. Similarly, the printed programme gave audience members an alternative route to most of the static content in the app. It was clear that audience members without the app still enjoyed the performance, so it that sense we were successful.

There are examples of works and performances with an app that is absolutely fundamental to the work, such as [31]. But in these cases, much higher levels of technical support are typically required, usually including the provision of pre-configured loan devices, and perhaps a less diverse audience. Similarly, Parcival XX–XI [19] depends at key points on audience members (literally) stepping forward to interact. But in this case only three or four volunteers are required, the devices (wii-motes) are provided and there is direct in-person support before and during the show (in the form of a “wii-fairy” character). In the case of Losing her Voice we lean more towards Troyer's [41] position that interaction design should not be reliant on a specific number of audience members engagement for a system to integrate and function within the performance.

There is clearly a deep tension here. Many possibilities are ruled out precisely by making the app non-essential. If the performance can succeed without the app then the app cannot make an essential contribution to the performance as a whole; its contributions must be optional, or perhaps faked if the technology fails. And the technology infrastructure for such an app is necessarily complex, novel and has clear vulnerabilities (e.g., to wireless interference or breaks in internet access) over and above the more established theatre technologies of lighting and sound.

Recommendation: Agree whether and how the app will be essential to the performance, beware of the temptation to assume that it must be so and build in fail-safes and redundancies where it is (the show must go on!).

To make a work dependent on an app demands a huge level of trust from the other members of the artistic team. And, as well as trust, it also requires sufficient knowledge and experience of how an app could be used in order to appreciate its “material qualities” [10] and to allow for them within the broader activities of composition and production. This appreciation of the qualia of a technology is something that can only develop over time, whether through direct collaboration, or through knowledge of the growing body of work that exploits apps. As the artistic director put it, “now I know it works and you trust it, the artistic minds can go, “ah”, in the same way we know we can trust lighting.” And it also requires acknowledgement and support within the mundane practicalities of theatre and opera production, e.g., budgeting, establishing the technologist's and/or interaction designer's role in the production team, time in the development and production schedules, marketing and communications.

Recommendation: Plan for an extended period of pre-production and development involving all members of the creative team and establish long-term collaborations over multiple productions and performances.

While it might be tempting to view limited uptake of a technology as a sign of failure, we propose the alternative view that an appropriately subtle approach does not demand high uptake as evidence of success. Rather, it is important to recognize various constituencies among the audience and to respect all of their interests. Even non-users of the technology should be considered as important stakeholders in the overall design and an appropriate measure of success is to deliver a good performance to all these constituencies; one that engages some with the technologies, encourages others, but does not disturb the remainder who do not wish to engage.

9.5 Trajectories Revisited

We now step back and reflect more generally on the use of the trajectories. Central to the design of the audience app and experience has been the joint consideration of three distinct canonical trajectories: the common trajectory of a night at the opera; the dramatic trajectory of this specific opera (i.e., Losing Her Voice); and the trajectory of an audience member's mobile app use within those. We consider their pair-wise interactions in turn.

To some extent the trajectory of the specific opera, Losing Her Voice, slots into particular episodes of the generic visit trajectory, i.e., “Act 1” and “Act 2”, when everyone is seated in the theatre and the house lights are down and focus is aimed towards the on-stage action. However, even in such a traditional form it also extends beyond this, to influence the marketing material and programme guide and also the audience's expectations, mood and reflections.

As we have discussed (and others have noted, e.g., [13]) within the usual theatre-going trajectory there are clear expectations about mobile phone use: that phones will be turned off (or at least silent and out of sight) when in the theatre. Indeed, some theatre goers would rather leave their phone at home than risk it “interrupting” a performance. So, bringing mobile phones into use within the theatre requires a radical change to the trajectory of phone use and audience expectations and behavior. And this in turn pushes back to some extent against the generic visit trajectory. At a minimum it requires that the directions about phones that are embedded in the “standard” visit experience are completely changed, i.e., the PA announcements, signs and notes in the programme guide which usually ask for phones to be turned off, and front of house staff must instead direct audience members to practices appropriate to the specific performance (e.g., on but initially silent). In addition, especially where a technically diverse audience is expected, elements that are alien to the standard experience—such as dedicated on-site support—may also be integrated.

In the discussion of interweaving and consistency, above, we have laid out some of the ways that the trajectory of mobile phone use has been shaped by the dramatic trajectory and detailed staging of Losing Her Voice. This ranges from high level choices of interface and interaction metaphors to the precise timing of in-theatre interaction in relation to (e.g.,) particular scenes, lines of the libretto and voices (especially chorus rather than principals). The direction of influence has been greatest from the opera to the app, at least in part because of the decision to make the app non-essential. However, over the extended design and development process of the work the app itself has also shaped elements of the opera, both as a composed work (ranging from its general conception to specific dramatic elements) and as a staged performance (at a particular venue and under a particular artistic director).

While others have pointed out the multi-scale nature of trajectories [17, 18] what we observe in this case is not only a variation of scale within a trajectory (there are multi-scale elements in each trajectory) but a collision between three distinct trajectories. Logically, we might try to view this as a single joint trajectory, but this quickly becomes impossibly complex. By preserving these distinct perspectives—of the visit, the specific opera and personal device use—the design process remains tractable. These distinct perspectives are also strongly linked to distinct stakeholders, i.e., the venue, the composer and production team, and the interaction designer, respectively. Consequently, collapsing these distinctions would at least partly conceal the collaborations that underly the total audience trajectory.

Recommendation: explicitly and separately map out three trajectories through the performance: the dramatic trajectory of the show; the wider trajectory of going to the theatre; and the intended trajectory of mobile app use. Identify barriers and opportunities to connecting the mobile app trajectory to the other two and select specific strategies (e.g., from the above recommendations) to deliver these.

Bringing together these different trajectory perspectives we can see some of the key ways in which—especially when dealing with established trajectories—each constrains and give opportunities to the others. The generic visit trajectory provides much of the gross temporal and spatial structure (e.g., the theatre, the event) and key transitions (i.e., constraints). It sets up expectations in terms of roles (“performer”, “audience”, “staff”) that may be selectively challenged, as when social media appear on-stage. It also provides an established ecosystem of “displays” and interactions (e.g., tickets, signs, screens, and so on.) that may require adaptation or be appropriated to new ends. The trajectories framework highlights the importance of transitions and especially beginnings and endings; as these different trajectories intertwine it becomes increasingly ambiguous exactly what the “beginning” and the “end” is. Nonetheless, each trajectory provides pragmatic answers (e.g., the visit, the performance) that help to anchor the total experience, while simultaneously acknowledging that these are not hard boundaries, and elements of experience can—and probably should—extend beyond them. However, while the notion of canonical trajectories and transitions are useful for describing our design, they are also insufficient. Specifically, the current framework does not explicitly provide concepts to express the potential connections and tensions between different kinds of canonical trajectories as they become interleaved as part of an overall user experience (with the partial exception of the idea of social encounters and isolations between different participants). This leads us to one final recommendation, this time for the trajectories framework.

Recommendation: the trajectories conceptual framework should be extended to include the new concepts of connections and tensions to support the interleaving of multiple canonical trajectories. Connections refer to explicitly identifying opportune points along a trajectory when it is open to connection with others, while tensions refer to moments when this would be particularly problematic. One could label each of the trajectories involved with both connections and opportunities and then explore the best strategy for joining up the connections while avoiding the tensions.

One can then consider a participant trajectory to be simultaneously anchored to each of these canonical trajectories, with a new type of transition in which the locus of the participant's experience and their focus of attention shifts from one canonical trajectory (or a subset of them) to another as their experience unfolds. A connections between canonical trajectories anticipate transitions and overlaps which enhance the overall experience, while tensions anticipate disruptions and distractions. Orchestration may then have a role, not only in bringing a participant back towards the canonical trajectory, but also in steering their focus from one to another.

9.6 Smart Phones at the Opera

For the final part of our discussion, we draw together our experiences in Losing Her Voice to reconsider the opportunities and challenges of taking smart phones to the opera and other similar conventional musical performances.

First, Losing Her Voice was successful in reaching a mainstream (rather than an “experimental interest”) audience, with more than half the audience using the app on their own device. However, although smart phones have become an established technology in many countries, their use remains profoundly uncomfortable in the context of a traditional theatre-type setting. As we know from embarrassing experience, the only way to be totally confident that a smart phone won't ring or chime is often to power it down (or leave it at home). Many (especially older) audience members also lack confidence in their ability to use or control their smart phone and may not use it frequently (assuming that they have one). Both factors can contribute to a strong resistance among some audiences to giving smart phones a place within live music and theatre. And even among those who are sympathetic to the idea, there are practical and personal concerns that discourage use. Addressing these challenges requires a deeper understanding of how to better integrate smart phones into mainstream performances.

Second, even when we can overcome this initial resistance to using a smart phone within the performance itself, the standard visual/touch interaction modality of current smart phones places unavoidable demands on audience members’ visual attention: they must look at the phone to use it! In rare examples (such as Self [48]) this shifting of attention is an intentional and essential part of the work. But in most cases attending to a phone means compromising ones’ experience of the stage. However, there are various ways in which this problem can be mitigated, for example through fast and constrained interactions with the phone, by providing feedback on-stage rather than on-device (as in the projected social media feeds in Losing Her Voice), by synchronizing these interactions with less demanding on-stage elements, and perhaps even writing in deliberate transitions from stage to phone and back. Losing her Voice has demonstrated that this can be achieved in a way that enhances the show without significant distraction, at least for a majority of the audience, but it remains a significant challenge for any new work.

Third, in contrast to visual interaction, using sound from phones within the performance—as in the final scene—naturally adds to the total performance soundscape. This is a natural consequence of the properties of sound and hearing: that sound tends to disperse widely and is perceived—at least physically—as a whole. Other projects including Dialtones [25] and The Smart Phone Project [31] have also demonstrated the potential effectiveness of placing sound elements within the audience through their own devices. Doing so brings its own technical and social challenges, including synchronizing the sounds, being able to play the intended “interruptions” but not allow other alarms or notifications, ensuring that phones and apps are active at the right time and having contingencies in place (such as the use of phones by the chorus’ in the final scene). It also implies that the audience's phones are actively considered as a part or channel during the composition and/or soundscape design process.

Fourth, moving out to the pre-show and interval spaces we find much more flexibility in terms of what might be asked of audience members’ use of smart phones. Passive information delivery via a phone app seems to have offered very little experiential benefit over a traditional printed programme, although it has some advantages in that it can be updated right up to the performance and has a smaller financial and environmental cost than a traditional programme. However more interactive elements, including the quizzes and taking selfies, were engaging and often social experiences for many audience members. Taking and submitting selfies demonstrates both the opportunities and challenges of incorporating user-generated content into a live performance. Compared to other performances where the audience contribute during the performance itself (e.g., Moori, [24] and “ADA FTW” [13], both of which use text rather than images), moving this activity into the pre-show and interval periods created time for content moderation and meant that audience members did not have to look away from the performance in order to contribute. However, it also created more distance between the contribution process and its use in the performance; while selfies remain easily recognizable, audience members might have found it harder to recognize or relate to an earlier text contribution.

Finally, we must emphasize the issues and recommendations raised in Section 9.4. While smart phones are a familiar part of everyday life, they are still far from being familiar parts of routine theatre and opera production processes and roles. As we have seen, the composer needs to develop a vision for the technology's distinctive role(s) within the work. This makes it crucial to incorporate smart phones from the beginning of the development process, both so that they are fully integrated into the work and also so that there is time for them to become a creative asset rather than an obstacle. Similarly, the artistic director needs to be fully convinced that the technology will deliver “on the night,” so that it can be permitted a prominent role in the final performance. Consequently, at least for some time to come, the effective use of smart phones in opera and similar settings is likely to require an extended pre-production and development process, and the gradual accumulation of trust and mutual understanding between all involved. In our case, the composer now feels empowered to incorporate more technology into future works. She found particular meaning in audience members engaging with the work on their own personal devices, and sees opportunities to link this to social and other media and to issues around climate concerns and social awareness. So perhaps the most effective collaborations will extend over multiple works (as in [10]).

10 Conclusions

Losing Her Voice is a contemporary opera but in a relatively traditional (linear, fully scored) form. Our aim—which we argue has been largely successfully met—was to create an app to be used with opera that would enhance audience member's experience and sense of involvement in the opera through their own mobile devices, without compromising the opera itself. This necessarily involved a design orientation that emphasized the subtle interleaving of readily available technologies into an established format, including recognizing the needs of non-adopters of the technology as valid stakeholders in our design.

We have explained how we have integrated the app with the opera (e.g., thematically, visually) and also how we have designed an audience trajectory to interweave use of the app with other aspects of the performance. This case study reflects a particular set of tradeoffs, for example that we use audience members own devices but not require everyone to have a device, and that primacy be given to the on-stage elements of the performance. This has been achieved in part through the deliberate selection of light-weight forms of interaction (e.g., likes and votes), to some extent at the cost of richer forms of interactivity or audience control over the performance.

While there are much more radically interactive and immersive applications of apps and phones in theatre and opera there are relatively few examples that push into the conflicted middle ground of supporting a relatively traditional form and production process but creating a genuine sense of interaction and involvement in the performance. We conjecture that emerging set of tools and techniques can increasingly support works that negotiate this tension, and we would like to see more new works that bring together established and new audiences with new elements of experience in otherwise recognizable forms of performance. Long-term creative partnerships, in particular, may offer a key way to develop the mutual trust and understanding needed for audience interaction to realize its creative potential.

Acknowledgments

This project was developed in collaboration with Nottingham Lakeside Arts. Thank you Velvet Spors, the “Gerry-Appers”, Martin Berry, Barret Hodgson, and the whole Losing Her Voice production team.

Data Access Statement

Data that underlies this article (anonymised questionnaire responses, audience interview transcripts and system logs) are available as DOI https://doi.org/10.17639/nott.7064.

Footnotes

A highlights reel of a performance of Losing Her Voice can be found here:

https://www.lakesidearts.org.uk/research/research-projects/losing-her-voice.html or https://youtu.be/XwgjjpSPDEY.

Also available as https://youtu.be/5tkCKo16Gyc.

https://github.com/cgreenhalgh/losing-her-voice/blob/master/logtools/src/clients.ts - see https://github.com/cgreenhalgh/losing-her-voice/blob/master/logtools/README.md for usage.

References

[1]

Kevin C. Baird. 2005. Real-time generation of music notation via audience interaction using python and GNU lilypond. In Proceedings of the 2005 Conference on New Interfaces for Musical Expression. National University of Singapore, 240–241. Retrieved January 25, 2017 from http://dl.acm.org/citation.cfm?id=1086008.

Abstract

1 Introduction

2 Related Work

2.1 Audience Technologies in Theatre Settings

2.2 Support for Audience Interaction

2.3 Mobile Phones in the Performance

2.4 Trajectories

3 Approach

4 Losing Her Voice

5 Design Process

5.1 Early Explorations

5.2 Refining the Approach

6 Designing the Audience Experience

6.1 A Night at the Opera

6.2 Design Goals and Strategies

6.3 App Design

6.4 Audience Trajectory

6.4.1 Supporting the Audience Trajectory.

7 Data Collection

7.1 System Log Files

7.2 Audience Questionnaire

7.3 Audience Interviews

8 Findings

8.1 The Audience

8.2 App Use

8.3 Audience Self-reports

8.3.1 Interactive Elements.

8.3.2 Other Media Elements.

8.3.3 Non-use in the Performance.

8.3.4 Pre/post Performance.

8.4 Reflections on the Audience App

8.5 Summary

9 Discussion

9.1 Encouraging Adoption

9.2 Interweaving Across Multiple Timescales

9.3 Complementing the Work as a Whole

9.4 Non-essentiality

9.5 Trajectories Revisited

9.6 Smart Phones at the Opera

10 Conclusions

Acknowledgments

Data Access Statement

Footnotes

References

Cited By

Index Terms

Recommendations

Understanding 'tingle' in opera performances

Engaging theatre audiences before the play: the design of playful interactive storytelling experiences

WanderOnStage: the convergence of percussion performance and media technology

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations