When XR and AI Meet - A Scoping Review on Extended Reality and Artificial Intelligence

Existing reviews on XR and AI typically focus on a particular aspect about XR and AI research, but do not cover their intersection comprehensively. Lampropoulos et al. [186], for example, reviewed applications of deep learning, semantic web, and knowledge graphs to improve augmented reality (AR). They identified object detection, image processing, and computer vision as three areas where deep learning can enhance user experience and services in AR. Throughout the paper, AI is expressed as a technology to enhance the detection of input like gestures or speech in AR. However, the paper is not specific on which techniques should be used for these purposes. Furthermore, there are many reviews of IVAs [262, 263]. Norouzi et al. [262] presented a systematic review on embodied agents in AR head-mounted displays (HMDs). They identified the application of embodied agents in assistive and collaborative roles as one of the emerging trends. One of the major challenges in this area is to enhance agents’ understanding of their physical environment. Another two emerging trends are the use of agents as companions (e.g., as therapy partners) and the modeling of agent personality and empathy. Other articles focused on IVAs in a certain domain, for example, for education and training [300], professional skills training [42], or healthcare [225]. Some reviews specifically address empathy [270] or the nonverbal behavior [30] of agents. We also found reviews that synthesize literature about applying both XR and AI for a specific use case. A frequent example of this category are works from the medical domain, such as clinical simulation for nursing pain education [119], using ML to assess surgical expertise in a VR simulation [383], personalizing doctor-patient surgical risk communication [14], or about the application of AI and AR/VR in medical education [99]. These papers cover individual topics that report on insights about applying AI and XR (mainly VR or AR) to a particular external use case, like in the medical domain. However, the state of the art of XR and AI research is not addressed by these papers. We not only differ from these reviews in methodology (i.e., using a scoping review instead of a systematic review), but also in our aim of giving an overview of the state of the art.

We found three papers that describe the broad range of research at the intersection of XR and AI. Luck and Aylett [224] coin the term intelligent virtual environments in their 2000 article about “applying artificial intelligence to VR”. A key concept that they use to discuss work on intelligent virtual environments is the concept of autonomy. Being very much an outlook into future systems, their work provides an interesting preamble to our work. Ribeiro de Oliveira et al. [299] reviewed papers with a focus on how VR and AI are applied to specific problems in “industry, commerce, services, logistics, processes, or systems”. This complements our research, which focuses on basic research at the intersection of the two fields without addressing how both technologies are applied to such an external problem. As a result, the authors point out that AI methods mostly contribute through high precision and high efficiency to VR problems (e.g., in surgery). The main drawbacks of applying AI for VR problems is a lack of training data and high computational costs. The most recent article in this area by Reiners et al. [296] is about the combination of XR and AI research. The main applications of XR and AI, as revealed in their review, are training (i.e., medical and military), gaming, robots and autonomous cars, and advanced visualization. These existing reviews focus around fields that XR and AI are applied to. In general, a lot of research is going on in the medical domain and on training. The papers point out that computational costs and limited training data are two major issues that limit the current methods. The work by Luck and Aylett [224] is more conceptual, identifying autonomy as an important axis on which to describe intelligent virtual instances. In contrast to these works, our review is focusing on the state of basic research at the intersection of XR and AI. More precisely, we are not interested in the application domains XR and AI are applied to, but how they are used with respect to fundamental research questions, for example, about interaction techniques in XR or user characteristics.

In the following, we characterize the understanding of XR and AI used in this work. XR is typically referred to as an umbrella term for VR, AR, and mixed reality (MR) [37, 112, 245, 293]. VR refers to overlaying the real world with virtual content by completely blocking real-world content. In contrast, AR refers to virtual objects being superimposed on an existing, three-dimensional real-world environment [23] using projection, optical, or video see-through devices. While researchers generally agree on these notions of VR and AR, the case is more complicated for MR [333]. The reality-virtuality continuum by Milgram and Kishino [246] typically serves as the basis for discussions around the term MR, but it has been criticised to not fully cover modern, more advanced technologies [333]. XR covers all of these notions (VR, AR, MR), and since we aim to cover the breadth of XR research, we include all of the above terms in our definition of XR.

In the case of AI, giving a definition is more challenging. Numerous articles aim to address the problem of defining AI [92]. As highlighted by Wang [375], early definitions of the term “indicate the same scope of intelligence as we see in human action” [261], or more abstractly note that “intelligence usually means the ability to solve hard problems” [249]. However, to date, “there is no widely accepted definition of AI” [375]. The ACM Computing Classification System³ lists AI, as well as ML, as computing methodologies, while in other cases ML is often considered a sub-category of AI. With our work, we do not aim to add another definition of AI to this collection. Our definition of AI is reflected in the set of keywords chosen for the search. To do that, we follow an inductive data-driven approach. We include articles that communicate on a high level that they used AI or specify a concrete method of AI. Since, to our knowledge, currently no clear list of such methods exists, we adopted an iterative approach to obtain AI terms.

This review is guided by the following research questions (RQs):

We adopted a two-phase screening process: In the first phase, we screened the papers based on title, abstract, and author keywords and in the second one based on the full text. Figure 1 shows the PRISMA diagram for scoping reviews (PRISMA-ScR) [356]. It details the complete process from initiating the search to identifying the papers included in the analysis. For both screening phases we first conducted a calibration phase on each 10% of the records in which all coders (authors A-E) screened the same set of papers, followed by single extraction for the remaining papers.

Exclusion and inclusion criteria. Based on our research questions, we defined the following exclusion (EC) and inclusion (IC) criteria. We derived them in an iterative process: Authors A and F first generated an initial set of criteria, which was refined with all authors after the screening process. Both screening phases used the same criteria, except E9, which we added after the first screening phase and thus it only applied in the second one.

Before the first round of screening, we implemented a script to identify obvious exclusion cases. First, the script identified whether there was no XR or AI term in title, abstract, or author keywords. Such cases should not have been found by the search engines in the first place, but we still had some cases in our list. Second, it identified survey papers (by looking for the words “survey” and “review” in the respective fields). It also highlighted cases where “learning” referred to an educational context and cases where “AR” referred to a false positive case, such as “LDAR”. Lastly, it identified duplicates by comparing the DOI of the papers and highlighted whether a paper was not published in the main proceedings (by highlighting the words “extended abstract”, “short paper”, “poster”, “adjunct”, “companion proceedings”). Author A reviewed and excluded these cases when needed.

First round of screening. In the initial screening, 2619 unique records were screened based on title, abstract, and author keywords by authors A-E. First, the five authors screened the same subset of 10% of the papers (258) separately. In a meeting after this calibration phase, the authors discussed discrepancies and refined the definitions of the exclusion and inclusion criteria. Before the discussion, there was an agreement of 56.6%, where all authors coded the respective record with the same decision. For another 23.6% of the records, all but one coders agreed (i.e., four coders agreed on the same decision) and the majority vote was taken. For the remaining 19.8% the authors disagreed. These papers were discussed in a meeting and and discrepancies were resolved. After the calibration, the remaining papers (2361) were distributed equally between authors A-E, resulting in a pool of 688 papers to be included for the second round of screening.

Second round of screening. The full text screening was conducted together with the data extraction phase. The authors first screened the full text for eligibility with the same exclusion and inclusion criteria as in the first round. Only one criterion was added (E9), which refers to the paper not presenting sufficient details on the methods, implementation, or results part to apply the codes. If the paper was included, data extraction was performed.

A total of 311 papers were included for the data extraction phase. We charted data for four main categories: “research objective and contribution”, “user-based evaluation”, “XR-related codes”, and “AI-related codes”. The data items are presented in Table 2. The complete code book (i.e., the codes with descriptions) is presented in the Appendix in Table 13 and Table 14. Authors A-E coded the papers. The code book was developed in an iterative process that combined an inductive and deductive strategy. We first started with a set of codes identified by author A. We then defined a random sub set of 10% of the papers (=69 papers), which we used for calibration to evaluate the suitability of the codes. During this phase we had three meetings, in which we discussed the codes’ suitability. Author A discussed the codes with author F in three separate meetings. After the calibration phase, we had a final set of 26 codes. We performed single extraction for the remaining papers. The remaining 619 papers were distributed among coders A-E (A:150, B:150, C:149, D:150, E:20).

Critical appraisal. Scoping reviews typically include all available evidence regardless of methodological quality [17]. We followed this approach. However, to receive a manageable set of papers, we decided to include the full paper proceedings and journal publications only. Besides these filters we did not exclude papers based on their methodological quality.

Limitations and potential bias. We acknowledge that our keywords selection process might have been influenced by the papers in the specific proceedings that we chose as a basis for the selection process. We chose this process to reduce subjective bias as much as possible. A second point that made the selection of keywords difficult is that both concepts (XR and AI) lack a clear definition. This is especially the case for AI. Being an ill-defined concept, it is difficult to find a comprehensive list of keywords and existing lists are likely biased towards the authors understanding of AI. Therefore, generating a list based on our outlined iterative process is the most bias-free solution that we found feasible. In conclusion, although we might have missed some keywords, we are confident that we found the majority of relevant papers. We aimed to cover the breadth of research at the intersection of XR and AI. Therefore, we defined broad keyword sets. We intentionally included terms that might only loosely be connected to XR and AI (e.g., “virtual space” in addition to “virtual reality” or “model” and “intelligent”). This approach left us with a high number of false positives. Yet, we still deemed it necessary to go for this breadth-first approach to really cover the full scope of XR and AI research. Nevertheless, we acknowledge that this approach might not be in agreement with other researchers’ definitions of XR and AI, who might have selected a more focused set of keywords. We adopted a data-driven approach to define the terms XR and AI. We did this to cover a broad spectrum of research. However, we acknowledge that there are other understandings of XR and AI, which might have led to a slightly different set. To reduce errors in the screening and coding phases, we conducted calibration phases on each 10% of the records for the screening and the data extraction phase with all the coders. Furthermore, we had extensive discussion sessions to resolve conflicts and adapt the exclusion/inclusion criteria and code descriptions (two one-hour sessions for initial screening, three one-hour sessions for full-text screening and data extraction).

Our analysis combines categorization, quantization, and narrative synthesis. First, we categorized the papers into topics based on C1. Then, we collected the research question(s) (C2) and contribution statements (C3, C4, C5) of each paper and summarized the main topic of the paper in one sentence. These sentences were grouped into topics using an approach inspired by affinity diagramming [222]. While author A performed this process, all authors discussed the topics in three meetings. The result of this process is a typology about the state of the art of XR and AI research, which is presented in the next section. Furthermore, we summarized the quantitative codes (C10, C11, C12, C17, C21, C23, and C24) and, based on the summary of all codes, we created a narrative synthesis, which is presented in the following.

The final corpus consists of 311 unique papers. The papers were published in 50 publication venues, with the most frequent ones being VRST (57), AIVR (42), TVCG (35), ISMAR (28), and CHI (22). Table 15 in the Appendix shows the full list of the number of papers per publication venue. The number of published papers per year is increasing, with a slight drop in 2020 (see Figure 2 A). Notably, the number of publications on XR and AI has more than quadrupled from 2017 to 2021.

We present the state-of-the-art XR and AI research as a typology. To create the topics, we grouped the papers into clusters based on the extracted research questions and contribution statements (C2 and C3) as described in section 3.6. In the following, we present the topics for the first three research directions; the fourth is not at the core interest of this review.

We found 15 problem areas that are addressed by the papers in our corpus (see Table 4). The list of challenges is based on the articles on challenges in AR and virtual environments by Billinghurst [36], Kim et al. [173], and Slater [329]. Most of the papers address a challenge about perception and neuroscience (21.9%). The main interest in this area is about understanding how users perceive realistic worlds and about how interacting with these worlds affects users, for example, in their feeling of presence [230], emotions [127], or visual attention [77]. The second area is interacting with IVAs (19.8%). Research on these challenges is mostly empirical (16.5%). The actual behavior of agents is rarely implemented based on a model, but mostly scripted. The third challenge is the presentation of virtual content. Here, many ML models are applied and evaluated with perceptual user studies (19.8%). These papers are about optimizing image quality or tracking of the real world and representing it in XR. This is similar for the problem of tracking technologies (17.3%). We found only one topic about health in XR, in particular simulator sickness (10.5%). The focus of these papers is on building ML models, followed by empirical research, but not all the ML models are evaluated empirically. The next two problems are creating high fidelity human characters (10.1%), which is mostly addressed by a combination of an ML model and an empirical evaluation. The same holds for interaction techniques (9.7%). Surprisingly, there were only 21 (8.9%) papers addressing social and ethical issues. Two thirds of them investigated an issue about interacting with VAs and one third about creating worlds. The vast majority of these papers contains a form of empirical evaluation, but, there is not much technical work in this area. Lastly, there is little work on more “traditional” computer graphics and computer vision topics like building devices (4.2%), rendering (3.8%), or display technology (1.7%). Lastly, as also demonstrated by the typology, there is not much research about using XR to address an AI problem (3%).

Contribution Types. When looking at the methods used to address problems (see Table 5), we see a trend of building ML models (57.8%) and evaluating them empirically (60.3%). This is present in many of the problems, as discussed in the previous paragraph. In general, there is very little theoretical and methodological work. Interestingly, there are some dataset contributions, in particular, in the problem areas of tracking technologies and health-related impacts. We collected all the datasets presented by the papers and provide a list of them in the Appendix in Table 8.

In the following, we summarize what type of algorithm techniques and classes are used in the reviewed papers. Furthermore, we present a list of commonly used tools, datasets, and networks. We also discuss the data types and summarize data about the users that is used to train and evaluate the ML models.

We found five topic clusters on XR and AI research. Most of the reviewed papers address a topic related to using AI to create XR worlds (89), using AI to understand users (61), and using AI to support interaction (48). Papers on these three topics typically address classification (72) or regression (58) problems and often present an ML model (134) together with an empirical (117) contribution. The fourth topic cluster is about interacting with VAs (25). Papers addressing this typically present empirical research (23), investigating user perception of interacting with agents, such as emotions or trust, but rarely present an implementation of agents. Lastly, there is very little work on using XR for AI (7). These seven papers present either a technique to enhance understandability by visualizing AI models in VR (5) or address the problem of limited training data in XR (2).

We analyzed the distribution of research directions and topics separately for HCI papers and compared them to the complete paper corpus. The distribution of research directions for HCI papers is almost the same as for the corpus in general. This might suggest that the topics at the intersection of XR and AI addressed by HCI research reflect the general distribution of topics as well. This is, however, not the case, as we discuss in the following. Almost half of the HCI papers (48%) are in the category of Using AI to Support Interaction. While this might not come as a surprise, given that this topic is the one that is arguably most relevant to HCI, it is still interesting to note. Thus, the primary interest of HCI research at the intersection of XR and AI is using AI for the improvement of interaction techniques in XR. With 39% the second largest group of HCI papers is about Creating XR Environments. We conclude that HCI researchers’ second most important interest is on investigating how AI methods can be used to enhance and ease content presentation in XR, mostly focusing on user body representations (avatars) and agents. Interestingly, our results show that only a few HCI papers use AI for what we labeled as Understanding Users (6%). This reveals a lack, where the core HCI venues (like CHI) could take inspiration from other venues (in particular XR venues), where AI methods are already applied to understand user characteristics and other properties of users. Lastly, we found only one HCI paper for the topic of Interaction with Intelligent VAs. This is surprising, since the core interest of HCI is about how users interact with computer systems and with more and more intelligent systems entering our lives, we argue that the research on interaction with virtual agents can be very beneficial and helpful to understand how users perceive and interact with cognitively enhanced computer systems, like agents. In general, we note that research at the intersection of XR and AI is highly relevant for HCI, since three of the five topics in out typology are about core HCI problems (understanding users, supporting interaction, and interaction with IVAs). Speaking from an HCI lens, we understand XR research as inherently connected to HCI, given that XR devices will likely become next-generation personal computing devices that we will interact with on a regular basis. Therefore, we are convinced that it is important for the field of HCI to understand how novel sub-areas (in this case the intersection of XR and AI) can influence and shape the field of HCI in general.

Based on our results, we formulate 13 research opportunities and recommend promising research directions. We first summarize five opportunities based on the analysis of the topics and conclude with eight general opportunities.

Scopus.

Web of Science.