Becoming Q: Using Design Workshops to Explore Everyday Objects as Interaction Devices for an Augmented Reality Spy Game

Prior work investigating the use of everyday objects as interaction devices in AR can be broadly split into a three categories: studies focused on technical aspects, works that present and evaluate complete systems, and works that take a more user focused approach.

The studies that focus on technical aspects most commonly explore the use of computer vision to provide more seamless ways to track objects that might be relevant to the systems being designed. One early study, ‘Annexing Reality’, offers a solution to finding objects within a scene based on object primitives [16]. Developers using this system are able to define a virtual shape using a collection of 3D shape primitives, and the system then finds a physical object that most closely matches the defined shape. By matching and co-locating these virtual and physical objects, the virtual objects are given a tangible proxy. A later system, ‘Grip Marks’, determines the shape and size of physical objects based on the users grip on the object. The researchers chose to track the grip rather than objects to overcome the potential challenge of the object being occluded by the hand [31]. More recently Monteiro et al. [21] created an AR prototyping tool called ‘Teachable Reality’ which allows designers to author AR interactions that make use of gesture, everyday objects and the users environment. These studies give a technical foundation that other work can build upon, and contribute further to the vision of seamless interaction with objects in the users environment in AR.

Early work exploring everyday objects in AR often focused on opportunistic use of features in the user's environment as a form of tactile feedback for a tangible user interface [15]. Following this, researchers developed a system called ’iCon’, which tracked objects via an attached marker and standard computer webcam [3]. Later work removed the need for markers, tracking objects using their intrinsic shapes instead [4], however a user study evaluating this system did not indicate a clear preference for using everyday objects over other methods tested [4]. Some papers have explored using everyday objects with projected AR, allowing the objects to be augmented virtually with information about their assigned function. Although it was proposed that this might lead to more effective use of the system, this was not evaluated [12, 30]. A more recent study in this space took an approach similar to ‘Annexing Reality’ [16], by providing a system that automatically identifies real objects in this case for use as everyday object controllers [18]. In the VR space a recent paper explored the use of everyday objects and environments for playful experiences [11]. In order to ensure that systems which support the use of everyday objects as interaction devices in AR are as effective as possible, it is important to understand how potential users would like to use these systems, both in terms of what objects they would use, and how they would use these objects. Three studies in particular have contributed to answering some of these questions using an elicitation study methodology [29]. The first of these studies relevant to the use of everyday objects in VR was the work of Moran-Ledesma et al. [22], which presented study participants with a series of functions based around open world games and computer aided design (CAD). Participants were required to select from a range of objects to use as props to define an interaction for each of the referents. Another elicitation study relevant to this field tasked participants with selecting an everyday object to be used as a physical proxy for a series of three virtual objects: a sword, a shield, and a crossbow, each of which were then used to complete a short game task [14]. Analysis of the results of this study uncovered some key factors that players considered when selecting objects, such as size and shape, weight distribution, and grip feel; as well as some less common factors such as personal connection. Finally, and more recently, Stellmacher et al. [26] explored how one specific everyday object, a mobile phone, could be used as a tangible controller in VR applications. While all three of these works contribute valuable insights to the field, the aim of this research is to address some remaining gaps by incorporating more complex objects that are not powered (such as the mobile phone) and taking an even more qualitative approach specifically in the AR context. The desired outcome of this work is to gain a broader understanding of objects and interactions with these objects, rather than eliciting interactions for specific systems. The following subsection covers some of these participatory design methods, and some of their applications, in more detail.

Participatory design methods involve including stakeholders of the system that is being designed as active participants in part or all of the design process [23]. Participatory design methods vary and can be applied at many different stages along the design process for a system. The aim of including stakeholders in the design process is ensuring that the final product being designed best meets their needs, whether that is due to their contributions to the final design or simply their contributions to defining the requirements for the product [23]. As already described in Section 2.1, some participatory design work has occurred in the everyday object interaction space [14, 22, 26], but more should be done to gain a richer understanding of user's needs in this context. This section covers three aspects of participatory design that were used when conducting this work: participatory design workshops, design probes, and the use of role play in participatory design.

In this work we define a “participatory design workshop” as a meeting of people who are representative of the intended users of a system working together with the goal to design an aspect of that system. This typically involves some form of brainstorming or other ideation technique to generate requirements or solutions for the system being workshopped. This process is not without disadvantages however, for example the user group may be less familiar with design techniques and design thinking than a practiced designer, and may struggle to generate as many ideas as a practiced designer might be able to. Because of this, it is important that participatory design workshops are well facilitated and appropriate methods are used to help participants who might not be familiar with design, however it is also important to balance between facilitating and not influencing participants ideation [5].

A useful tool to help designers understand the lives of people who are involved in the participatory design process are design probes, which are instruments that allow non-designers to participate in design research [7, 25]. Design probes were inspired by cultural probes, which focus on collecting open ended information on participants to inspire design [13], with the difference being that design probes focus on fostering participation in conceptual design phases. The theory of design probes informed the design of the pre-workshop survey, described in Section 3.1, allowing data gathering while also priming the participants who went on to participate in the participatory design workshop.

A common activity to encourage idea generation during participatory design workshops is role play, which in this context generally involves participants taking on the role of a stakeholder of the system being designed, then acting out and making design decisions as if they were actually their assigned role person[27]. As an example, if the system being designed is for building a bridge, one participant may take the role of a pedestrian and might push for more walking space and sturdy barriers along the footpath, whereas another may take the role of a motorist who might push for more road space and a higher speed limit. Having participants play the role of someone other than themselves can help them to ideate and contribute to producing better solutions.

Role play has been used in the design of computer systems as early as the 1980s with the UTOPIA project [10]. As role play in design has developed it has become common to include an element of low fidelity prototyping - prototyping typically completed with pen, paper and basic craft supplies - so ideas can be quickly mocked up and evaluated [1, 19]. Some role playing has even been elevated using elements of drama, such as with “Focus Troupe" [24], and some studies even use professional actors to act out computing scenarios [17]. In this work, we have adopted a novel approach to roleplay by having study participants take on a role that is diegetic, that is to say that it exists within the concept of the game being developed. Participants in our study adopt the role of a “gadgeteer”, an individual responsible for enhancing everyday objects with special spy gadget functionality for a spy themed game. This is described in more detail in the following Methodology section.

Before completing the main section of the survey, respondents were asked some basic demographics questions regarding gender, age, frequency of using AR, as well as their experience with design. To be non-specific and inclusive, “design” was described in the survey in the following way:

“A designer can be defined as: ‘A person who plans the look or workings of something prior to it being made’. With this definition in mind, is there anything that you would consider yourself to have designed, whether professionally or as a hobby, and if so what have you designed?”

By phrasing the question in such a way we hoped to capture experience with design beyond just that in a professional setting, which we felt may be assumed without the additional context.

Four objects selected for the survey, arrows added to illustrate each affordance. Top Left - Lever as shown on a pair of tongs, Top Right - Rotation as shown on a can opener, Bottom Left - Trigger as shown on a spray bottle, Bottom Right - Button as shown on an inhaler

The main section of the survey asked respondents to identify three objects from their environment that they felt would be suitable as a prop in the context of an AR spy game, as well as to design a gadget function for each object. Some examples of these objects can be seen in Figure 1. Regarding the selection of objects the survey was worded as follows:

“Please take the time to select some objects that you find interesting or enjoy using, and have these objects handy when you complete the rest of the survey. Also think of how these objects could be given special abilities within the context of the game. Think about special functions your object could be given to assist a spy or someone completing a heist.”

Respondents were asked to upload an image of each of their objects, and respond to the following five prompts for each of the objects:

1. Please describe the object that you picked and why you selected it.
   
2. How do the moving parts on the object support its regular function?
   
3. Please describe the in-game function you have thought of for the object you have chosen.
   
4. How do the moving parts on your object support the in-game function you have created?
   
5. What limitations or challenges do you see with using the object for the in-game function?
   

These prompts were selected to gather information about objects that participants would identify, how they conceptualise the objects’ design and use, as well as getting them to generate some initial ideas for what the objects could do in the context of the game. The questions were discussed amongst the authors to ensure that they would meet these aims and be clear to survey respondents.

The survey served a dual purpose, to both gather information and as a means to stimulate ideation ahead of and in the workshop. The information gathering component was helpful for two reasons: Firstly, it allowed the collection of demographic information ahead of the workshop to help maximise time in the workshop for generating ideas, and Secondly the images of everyday objects captured by the participants, along with some initial ideas, were an important input to the thematic analysis in relation to question R₁. Using a survey also allowed for a wider sampling of participants (although in practice not as widely as intended), compared to the two hour workshop which was harder to recruit for due to the additional time commitment required from participants. Finally, the survey also served as an icebreaker towards the beginning of the workshop as described in more detail in the following Subsection 3.2.2.

Two separate workshops were completed, with 5 different participants in each, for a total of 10 workshop contributors. It was decided to run separate workshops to make it easier to accommodate participants, and to ensure that there was a manageable number of participants in each for the facilitator managing the workshop. Each workshop was run in person over 2 hours, and was split into five main parts: Introductions, Discussing Objects, Brainstorming Gadgets, Applying Ideas, and Design Review. At a high level this structure was designed to emulate the “diamond approach” to design; to have participants expand their thinking with idea generation in the first half, and then focus their thinking during the second half as they came up with ways to apply their ideas to in-game scenarios.

Before beginning the main sections of the workshops, there was a short overview period where participants were asked to review an information sheet, were able to ask questions, signed their consent forms, and received an overview of each of the five main parts to the workshop. There was also a ten minute break between the Brainstorming Gadgets and Applying Ideas sections. To provide a range of creative outlets, pens and pencils, coloured markers, A4 and A3 notepads, play dough, and popsicle sticks were provided for participants to use to explore and express their ideas. The remainder of this subsection will cover the five main parts of the workshop, beginning with Introductions.

3.2.3 Applying Design Ideas. Once participants returned from the break, they were asked to apply their ideas to four in-game scenarios: ‘Find a hidden object’, ‘Access an object behind a barrier’, ‘Disable a security system’, ‘Repair something’.

These scenarios were selected for being archetypal to the spy genre as depicted in movies or television, but were left intentionally vague so that participants could apply creativity in how they defined and then solved the scenario. This part was also mainly participant driven and open, however one participant was required to record the ideas that were generated to solve each scenario. Recording these ideas was done so that all participants could vote on this final set of ideas as described in the following paragraph, an example of this recording and voting can be seen in Figure 2. To ensure that all participants had equal opportunity to contribute during this step, a different participant was responsible for recording ideas for each scenario. Participants were given roughly ten minutes to discuss and note down ideas for each scenario, with flexibility in the time depending if conversation had stopped early or was running long. When recording ideas, participants were asked to record their more specific definition of each scenario, for example ‘Access an object behind a barrier’ could have been defined as ‘Retrieving an item from a locked safe’.

A page from Workshop one showing ideas recorded during ‘Applying Design Ideas’, the coloured labels indicate votes: green - love to play, blue - like to play and pink - wouldn't want to play.

The analysis for the results of the survey and workshops used reflexive thematic analysis [2] to define a set of themes that give rich qualitative insight. The first stage of the analysis was to process the data from the survey and workshops. For the survey this consisted of downloading and reformatting the responses so that each response contained a brief summary of the participant's demographic information and each image they had uploaded, followed by their answers for each question in a single document. Excel was used to process the quantitative demographic information, reported below in Subsection 4.1. For the workshops this data processing involved scanning all of the written material to files, and processing the video recordings to export an audio recording of the workshop. Once the audio files were exported, Microsoft Word was used for an initial transcription. In order to ensure accuracy of the transcription and to maximise immersion in the data, the researcher reviewed these transcripts by hand and edited them whenever the automatic transcription was not accurate.

The next stage was for the researcher to further immerse themselves in the data generated by the study participants. This involved reading over all of the survey responses, written workshop materials and workshop transcripts, in addition to editing the transcripts as mentioned previously. The videos for the workshops were also reviewed during this process. An example of some written workshop material can be found in Figure 3. Note that the survey and workshop data was analysed together rather than separately to input a deeper pool of information into the thematic analysis.

Once immersion was complete, the next stage was to code the data. This stage involved reviewing all of the study materials once again, this time noting down any significant ideas and assigning them a code. These codes were defined with several key ideas in mind: establishing the base elements of objects, how objects are used for interaction, and the game ideas proposed for these objects.

From these codes, a set of themes were generated by grouping codes that expressed similar ideas or naturally appeared to fit into particular categories based on the researcher's judgement. These themes were expanded and developed so that they covered as many of the defined codes as possible. Some of the theme definitions changed over time as the researcher reflected on each individual theme and the relationships between the themes. As part of this development and review process, the themes and codes were shared with co-researchers. Most codes and themes remained largely the same through this step, but some of the names and definitions were clarified so that they made sense beyond the interpretation of the principal researcher. By the end of this process four themes were largely defined, these are described in Section 4, however some minor changes occurred as the researcher reflected on these themes during the writing of this work, primarily to clarify the wording of some codes and definitions.

An excerpt taken from the notes of one participant. Noting each of the Affordances defined under the theme ‘how players want to use objects’, some ideas pertaining to Augmenting the Player's Perception (telescope and X-ray vision), showing some examples of Projecting an Effect (squirting objects or self), as well as Combat (kill enemies)

The ten workshop participants were primarily self-selected from completing the survey. Those participants that did not self-select were contacted directly via email, to ensure enough numbers to run each workshop. These additional participants also completed the survey before attending the workshop, making the ten workshop participants a subset of the sixteen survey participants. Out of respect for their time we did not require them to answer the demographic survey again, and as such they could not be linked to the survey, as that was kept anonymous for ethical reasons.

Of the n = 16 survey participants 7 identified as male, 6 as female, and 3 and non-binary. 5 were aged 18-24, 4 were aged 25-29, 5 were aged 30-34, and 2 participants were aged 40 or over. Most participants (n=10) had used AR on a mobile phone but not regularly, 1 indicated they used it 2-3 times a week, 1 once a month, and 4 indicated they had never used AR on a mobile phone. 8 participants indicated having used AR on a head mounted display but not regularly, and the remaining 8 indicated having never used it.

All participants indicated having some design experience, most (n=8) with 4 or more years, 3 indicating 1-3 years, and 5 indicating less than 6 months of design experience. Each participant had a different answer for what type of design they had done, but of note and relevance to the topic area was: “commercial AR experiences”, “product designer for 10 years”, “design of a number of intranet projects and operational workflows”, “user interfaces, video games, and computer interface hardware”, “Games, product prototypes”, “A serious game, lots of play-based activities for preschoolers” and “Haptic devices, robots, miscellaneous creations”.

This theme was derived from codes that related to the participant's rationale for having selected certain objects during the survey, or objects they selected to ideate upon in the workshop. Four codes under this theme pertain to simple aspects of the objects: Weight, Size, Convenience, and how Ergonomic the object was. An additional code, Discretion, was also identified, in this context referring to how discreet the object would be in the game context, for example would the object be small enough to hide on or be carried on one's person without being noticeable. Although Discretion is a simple aspect of the object, it is fairly specific to the game context and as such has been listed separately.

Another code contributing to this theme was Personal Connection, where participants identified a particular personal association for an object, for example their “kids used to play with” the object. Finally in relation to this theme, there were several occasions were participants noted that the more ordinary, or the less technological, an object was the more interested they were in seeing it play a role in the game. In one such example, in reference to a digital guitar tuner, one participant stated that “funnily, that is the least appealing object”, while another participant noted that the tuner “does magic already” and “it's more fun to think of loopy attributes of a toilet roll” (the toilet roll being an object that was brought along to the workshop).

The second theme describes how participants identified wanting to use the objects, specifically what features of the objects they identified as important, and how they described wanting to use the objects. This theme is split into three subthemes: Affordances, Interaction Type and Mixed Object Use. Affordances describe the features of the objects that participants identified as important for performing interactions with, for example using the Trigger on a spray bottle to activate an effect in game. Interaction Type is related to how the participants described using the objects, and how that might be interpreted by the game. Finally, Mixed Object Use covers how participants described how they might combine multiple objects to achieve a goal, or incorporate their own gestures alongside objects.

This theme is concerned with the in game functions participants designed for the objects in the workshop. This theme is divided into four subthemes: Projecting an Effect, Type of Manipulation, Virtual Agents, and Altering Perceptions. Projecting an Effect covers functions that involve the gadget projecting something into the game that alter something within the game context, for example firing a Projectile that would knock out a security guard. Type of Manipulation includes the two main categories of how participant's gadgets would manipulate elements in the game, specifically Remotely or Directly. Virtual Agents is derived from ideas that involved non-playable characters, for example by describing Combat with a guard. Finally, Altering perceptions covers functions that involved altering how the player is perceived by the game, or how the player's perception of the world could be altered. Figure 4 shows some sketches drawn by participants that illustrate a wide range of these ideas. Some illustrated examples of four of these ideas can be found in Figure 5.

Sketches from one participant the contain a range of ideas for gadget functions of several hair and make up objects.

Some participants expressed concerns about using objects in the context of the game for a few different reasons. The first of these was the Friction between real and virtual function, which relates real world functions occurring when objects are used for virtual functions and vice versa. Examples of this include spray bottles and pens, where participants expressed that there is tension between the regular use of those objects if used in game - participants did not want to be spraying fluid or drawing over surfaces in the play space when using a spray bottle or pen respectively.

The remaining concerns were centered around Feasibility of Implementation, specifically concerns over Tracking Occlusion. A few participants expressed their concerns around small objects making tracking difficult, for example stating that a hair clip “may be difficult to use in AR, as the clip is small and our hand will cover half of the clip when we use it”.

Illustrated examples of the most popular ideas for each scenario during the final workshop segment, real objects depicted in black and virtual objects in blue. Top left - A pair of glasses with enhanced X-ray vision, Top right - A phone charger being used to grapple virtual objects, Bottom left - A spray bottle revealing the code on a keypad, Bottom right - A cardboard tube revealing diagnostics to fix a car engine.

The first potentially significant limitation of this study is the choice of a spy theme as the context for the AR game. As discussed earlier, this was done for two reasons: (1) to ease participants into the workshop by presenting a context where having an augmented everyday object made immediate sense and (2) in the hopes that a modern to slightly futuristic context would generate ideas that might generalise to other AR contexts. However, this does mean that the ideas generated could be quite specific to the game theme used. Despite this, we only identified two codes, How the Player is Perceived and Discretion, was specific to the spy genre. Regarding the code How the Player is Perceived we acknowledge that deception and stealth are present in games that might not be considered within the spy genre, however we argue they are similar enough that this code may not be generalisable to other genres.

Beyond generating overly specific ideas, another concern with the game theme is that it could lead ideation in a direction that misses key ideas that other themes might have stimulated. It is difficult to estimate what the impact of this is on the results of this study, but could be avenue for future work. By conducting repeat workshops focusing on other game genres, a better understanding could be gained regarding the themes. This could provide a deeper understanding of the existing themes, and might generate additional themes or sub-themes.

There is also some limitation with the quantitative measure given in relation to the thematic analysis. This is primarily due to the group setting, the measure can not reflect participants agreeing with each other, and participants may choose not to express an idea they've heard before or they might latch onto an idea as a group thus increasing the codes count. The measure also does not effectively represent certain codes that are implied by other codes, bi-manual for example implies uni-manual, making it an important inclusion despite only having a single explicit reference in the workshop. Despite these limitations the count does give an impression of how representative each code is, which might be useful to designers when evaluating which elements to consider in their work.

Finally, it is important to consider how the demographics of the group might influence the results. The group participating in this work had an above average familiarity with design in the context of tech and games, including AR, which likely biased their responses compared to a truly average population. It is likely that certain codes identified by the researcher would have been less prevalent in a more representative sample, in particular concerns around feasibility of implementation. It seems less likely that codes such as Tracking Occlusion would have been brought up by participants that were not already familiar with how AR functions. These technical concerns are something that designers of these systems would need to overcome to make functional systems anyway, and as such we believe the most important part of the theme ‘Concerns Participants Have about Objects Use’ is the Friction between real and virtual function.

Friction between real and virtual function is not unique to this work, and other researchers have noted this conflict [3, 12]. This is important to consider when designing systems that use everyday objects as interaction devices, because it can impose serious limitations on what sorts of objects can be used, and how they should be used in these contexts. The objects most commonly identified by participants that exhibit this friction contained some sort of fluid, for example spray bottles and pens, and these objects run the risk of emitting that fluid in the process of their virtual use. While there might be some experiences where this could be an advantage, adding another sensation to the experience for example, in most cases this would be undesirable. One possible solution available on some spray bottles would be to turn the nozzle to a closed position as, in cases where that is possible, this would free up the use of the trigger affordance. However, this poses another usage limitation as it stops the use of the rotating nozzle, for example to Configure the objects virtual function. Another solution would be to only use these types of objects once they are empty, although this also is not ideal as the object loses its real functionality so one advantage of this interaction modality is lost in doing so. For these reasons, it should be a fundamental consideration when designing games that aim to incorporate everyday objects that there are not many clear universal solutions to this friction.

This friction between real and virtual function might advantage certain system implementations over others, systems that prefer a fixed assignment between object and function could benefit, where as those that aim to provide choice by the user are more likely to run into friction. In cases where the assignment is fixed objects could be selected that have no, or lower amounts of, object friction. Typically those systems will either have a specific object per function or will aim to identify a best fit object in the scene for each object, sometimes allowing some user selection. If the object as a whole is not particularly important, the system could look to individual elements rather than the whole object making it easier to find a suitable best fit. These best fit systems could also consider certain elements around how much friction is present between the real and virtual use, this might prove somewhat complex but is another possible solution to the object friction problem.

One of the codes contributing to What Players Look for in Objects that runs counter to this argument for automated assignment of object and function is Personal Connection. While this code does not provide anything specific for designers to include, it does highlight that some amount of user choice might be important. To support Personal Connection in a completely automated system would be extremely difficult, so perhaps as a compromise is for systems which highlight candidate objects for a final user selection, such as the system by [18].

Something we discovered when analysing the results of this study is that an important tool that could assist in designing interaction systems that utilise objects in the environment would be a taxonomy of objects and their uses as AR interaction devices. Each of the themes that we have defined here could serve to inform the design of such a taxonomy, and this is being considered as a direction of future work. Building this taxonomy may require conducting further workshops in other genres, as discussed in Subsection 5.1. Each of the four themes play an important role in informing the design of this taxonomy: ‘What players look for in objects’ is important as a baseline for which objects in an environment should even be considered as interaction devices and which objects should be excluded from the taxonomy; ‘How players want to use objects’ will play a key role in how these objects are classified, particularly in terms of what Affordances are important, and what Interaction Types each affordance might be best suited to support; Mixed Object Use can help to inform what should be considered in terms of the relationships between objects; and ‘What players want their objects to be capable of in game’ is important when defining how this taxonomy might be put to use, for example as a basis to define an object selection algorithm to find the best fitting object for a particular game interaction. An object's friction between its real and virtual function is a fundamental consideration for these systems, and so finding a way to describe that on an object by object basis could greatly improve the taxonomy. Beyond just considering a taxonomy for objects as AR interaction devices, understanding how people think about objects, interactions, and game functions is important to consider when designing systems using everyday objects as an interaction modality.