Immersive analytics: An overview

2.1 Immersion

A term that covers central aspects of the underlying concepts of IA is immersion. As it is the case with many other terms that are used to describe related concepts, immersion is overloaded and has diverging and sometimes fuzzy definitions in the literature. It has been discussed in a variety of contexts, e. g. for learning, gaming, story-telling, and data analysis, and has been identified with or delineated from a variety of other terms such as engagement or flow [60].

Generally speaking, the concept of immersion in IA is supposed to cover characteristics of an interaction of a human with an environment. This environment can comprise a single medium, or a combination of media, and can be computer-mediated, as in the example of a VR environment. The interaction can be as reduced as pure consumption, e. g. simply reading a text in a book or on a screen, or as complex as a multi-modal interaction with physical navigation, gesture, and speech recognition. Immersion is assumed to support the engagement of the analyst and thus subsequently to support the analysis process.

A main distinction that is found in the literature is the separation of the capabilities of the environment for multi-modal / visual representation and interaction, and the impact on as well as the (re)action of the human. The former are rather static, technological aspects, but might also comprise the specific design of a representation as well as its dynamics, e. g. in adaptive visualisations, which adjust to the user’s requirements, abilities, and actions, whereas the latter comprises elements of the psychological state, engagement, emotion, and perception. Consequently, one attempt to capture these aspects is the separation into ‘psychological’ and ‘technological’ immersion [27]. However, there are also at least questions if further or more detailed aspects, such as long-term acquisition of skills (also including time outside of an IE), or more general learning, should be explicitly taken into account (see [65]).

The investigation of immersion has been conducted for many years already in the area of gaming research [10], [14], [41], and more recently also for visual data stories, see Isenberg et al. [37]. Agrawal et al. [2] give a literature review on the term immersion in the context of immersive audiovisual experiences. For data analysis, the term became popular in the context of IA [49]. Donalek et al. [25] discuss immersive and collaborative data visualisations in VR, and note that immersion provides benefits over traditional visualisation (perception, understanding, relationship retention), but don’t explicitly define the term and seem to identify it with 3D VR visualisation. While aspects of technological immersion can be a precursor or requirement for psychological immersion, there are further non-technical aspects such as a narrative or a specific visualisation metaphor that might strongly influence psychological immersion.

Slater and Wilbur [72] distinguish between immersion and presence, where the former describes characteristics of the employed technology, such as field of view, resolution, and degree of freedom, and the latter is the sense of ‘being there’, a state of consciousness that is the ultimate goal of virtual reality. Presence is supposed to increase the engagement, and to turn the experience of the user from the consumption of computer-generated imagery to the impression of an experienced reality, e. g. places visited or real-world objects explored. Note that according to the authors in this definition immersion requires a self-representation in the VE, and immersion is increased when physical reality is shut out, which is only partially the case across the reality-virtuality continuum. Thus, immersion here is restricted to the characteristics of the system, and sensori-motor contingencies that it supports, a definition which according to Thomas et al. [78] is widely used in the VR/AR communities. Given the goal of IA, to improve data analysis by employing immersive technologies, these characterisations seem to be more demanding than required, as the lack of the impression of being somewhere else might not comprise the analysis quality. More recently, presence as the sense of ‘being there’ has also been termed place illusion [71] to avoid confusion about the term’s intended meaning.

Brown and Cairns [10], [13] investigate immersion in the context of games, and propose a definition that includes three levels, engagement, engrossment, and total immersion. Jennett et al. [41] state that a clear understanding of the concept in the context of gaming is still missing, explore quantification measures and present corresponding experiments. They conclude that immersion in their definition can be measured subjectively and objectively, and that it can be associated also with negative experience, such as anxiety. A few years later, Calleja states that there is rather more than less confusion about the term [14], and proposes to distinguish absorption-sense (sense of involvement and engagement, psychological immersion) and transportation-sense (feeling of being transported in a different world) of immersion. In particular for gaming, immersion is often described to include the player’s response to the system and the corresponding psychological state.

Further terms that are often used in the discussion of immersion are spatial immersion, where the user’s first-person 3D view of the surrounding world is supported by characteristics such as large field of view, movement tracking, haptic feedback, and embodied interaction. While often associated with VRE, spatial immersion can also be induced by spatial AR, where projections on the real-world environment are employed [77].

2.2 Multi-modal representation and interaction

A major assumption in immersive analytics is that natural and rich interaction supports immersion by removing the barrier between the analyst and the data [12]. Thus, natural user interfaces that use physical metaphors and everyday operations and movements are investigated in the context of data analysis and corresponding tasks. Using S3D already extends the possibilities on how to perceive and interact with data representations. The integration of multiple senses beyond pure vision has the potential to create more natural interfaces that further improve the analyst’s experience and analysis capabilities. McCormack et al. give an overview and a design framework on multisensory immersive analytics [53], and Martin et al. present a survey on multimodality in VR [50].

2.2.1 Visualisation

Data visualisation is the most established way for data representation both in classical and in immersive environments, using our main, high-bandwidth, sense for perceiving our environment. A major differentiation between visualisation in classical and in many immersive environments is the use of S3D. For certain tasks and data [47], S3D can provide major advances compared to standard 2D data visualisation, as the third dimension can be used as an additional channel for further data dimensions, provides multiple viewing perspectives, and the space in the third dimension can be used for both arrangement of objects and their encoding, for example eliminating the overlap that often occurs in 2D arrangements. However, this additional freedom comes with further potential issues [54], such as dependence on a viewing perspective, where the selection of a favourable one can incur further effort, the occlusion from stacking of objects along the depth direction, depth perception and the corresponding requirement to incorporate proper depth cues for orientation. Individual disabilities such as the lack of 3D perception in a part of the population are further issues that need to be taken into consideration.

Figure 1

Visualisations and physicalisation of ser/thr protein phosphatase 1 (PP1). From left to right: 3D holographic visualisation of PP1 (PDB ID: 1FJM [7], [32], [33]) on a looking glass device, data physicalisation with two 3D printed surface models in different sizes and a standard stereoscopic visualisation on a 2D screen with the molecular visualisation software ChimeraX [31], [64].

Figure 2

VR exploration of a cell with information regarding protein localisation and metabolism using a combined VR visualisation environment based on CAVE2 and zSpace (taken from [73]).

2.2.2 Haptics and data physicalisation

The main communication channel of classical visualisations is, unsurprisingly, the visual channel. By venturing into immersive spaces, however, multi-modal and multi-sensory experiences of data might be delivered. When immersing into data, it is specifically interesting to explore the idea of haptically “touching data”. One way of providing a haptic data experience of data is through data physicalisation. Here, physical objects, e. g. from 3D printing, are used to represent data. For example, Figure 1 shows a 3D-printed surface model of a protein. Touching a physical data object might reveal additional information such as texture, stiffness, temperature, and weight [40] that goes beyond traditional visualisations. In addition, the objects might be tracked to act as a interaction token, e. g. to support basic manipulation operations such as rotation. The challenge of physical objects as visualisations lies mainly in their inflexibility to change and the cost of creating them.

Figure 3

Spatially-resolved transcriptomics data of the heart in an immersive analytics tool using the zSpace (taken from [8]).

Figure 4

PropellerHand is an ungrounded hand-mounted force-feedback device based on two rotatable propellers; it can be used to simulate touching visualisations such as immersive line charts (taken from [1]).

One way to overcome these issues is to leverage haptic feedback devices that allow to “touch” purely virtual objects. There is a plethora of haptic feedback devices that could be used to physically experience objects in Virtual Reality [68]. Defining an immersive visualisation as such a virtual object also would allow to touch it with such devices. Figure 4 provides an example of an hand-mounted, propeller-based feedback device that was used to touch and haptically interact with abstract data visualisations. Usually, the haptic experience of such devices is inferior to touching real physical objects and necessitate to instrument the user to different degrees. Still, they allow to deal with dynamically changing data which is inherent to almost all interactive visualisations. Furthermore, they offer interesting opportunities for visually disabled people to interact with data (little research is available on that though).

2.3 Collaboration

Support for collaboration beyond the capabilities of standard desktop environments was stated as an explicit goal of IA research [9]. By breaking the barriers of the confined space in desktop environments, IE provide ample opportunities for the support of collaborative data analysis [17]. This includes both collocated as well as remote collaboration and their combination. Large display walls and environments such as the CAVE provide the physical space for collocated collaboration, whereas in particular virtual environments allow to scale remote collaboration easily [67]. Important factors for the design of corresponding IEs are proper representations of collaborators and their actions, support for communication and interaction between participants, as well as support for provenance, e. g. to hand over the current state of analysis in asynchronous settings. Important related questions are what the required characteristics of collaborator representation are, how communication is supported, how IEs and potentially different devices [30] can be used to create both shared and private interactive views on the data, and how analysis methods are integrated in a consistent way.

2.4 Situated analytics

Much of the existing research in IA focuses on leveraging VR displays for the purpose of data visualisation and analytics. The interesting characteristics of VR for such scenarios include stereoscopy for spatial tasks on 3D data, but also an almost unlimited “space to think” [4]. While some of the existing IA approaches have also been used in AR, so far they have been rarely truly situated or embedded into the real world context. Doing exactly this is the goal of situated analytics [28].

Situated analytics focuses on how analytical work can be embedded into the user’s spatial environment. To do this, a computational understanding of the real world is needed, which then can be leveraged by AR to display virtual data representations along with real physical referents. The interesting question of course is how exactly the relation between real physical objects and virtual data visualisations is implemented. Thomas et al. [78], for instance, see in situated analytics those visualisations that are perceived in close spatial and temporal proximity to its referent. Data from a smart meter might for instance be shown right next to it, like a virtually attached display [28]. Willet et al. [83] follow a similar idea of situatedness, but go one step further and call visualisations that are directly overlaid onto referents “embedded visualisations”.^[1] CAD model data might for instance be shown directly in place to simulate different setups of production chains [36]. Or you show pollution data right in the streets where they were measured [81]. An accurate registration of the virtual data with the physical environment is needed for such solutions.

Situated analytics displays bear a huge potential as they merge the real and the virtual in a seamless way. In comparison to VR-approaches, they thus do not necessitate the user to get detached from the real world and instead bring data visualisation exactly into their current spatial and social environment. With highly sophisticated AR displays (which do not exist yet) one can even imagine that such an approach might eventually replace traditional screen-based displays. For the time being, however, the main benefit lies in tasks that intrinsically need to combine physical and virtual components.

Characterising the full potentials of situated analytics is still an open research question. One of the challenges is that developing truly situated/embedded visualisations is non-trivial. While much progress has been made on the technical sophistication of AR hardware (e. g., Microsoft’s HoloLens 2), developing situated analytics tools still requires to combine different concepts from computer vision, augmented reality, internet of things, and visualisation. Dedicated toolkits that support this development process have only very recently started to become available [28]. Even with good toolkits, however, many design challenges remain. These primarily stem from the fact that, for situated analytics, visualisation designs need to be directly incorporated into real world circumstances. Classical visualisations on screen, and even more so immersive visualisations in VR, are viewed in a fairly controlled environment. Situated analytics solutions, in contrast, might be strongly impacted by environmental factors, such as environmental visual clutter, changing light conditions, or simply users moving around with their AR displays. All of these factors pose a plethora of open research questions for visualisation, and call for novel robust and adaptable visualisation designs.

3.1 Concepts for and evaluation of immersion

As we have seen in the previous discussion, immersion is a term that has multiple descriptions and approaches for measuring, including quantitative and qualitative measures. Characterising and quantifying immersion in the context of IA is thus a major topic and challenge in current research. In particular the support for different levels of immersion induced by the affordances and the design of the IE is a major open question. Further, the immersion along complex workflows in application areas, and the impact of changes of environments and corresponding changes of the immersion and focus on the analysis is investigated. Cummings and Bailenson [19] performed a meta-analysis on 83 studies to investigate if greater levels in the immersive qualities of an environment elicit higher levels of presence (as defined by Slater and Wilbur). They conclude that technological immersion has a medium-sized effect on presence. Miller and Bugnariu conclude from their meta-study that the impact of technological immersion might be strongly depending on user characteristics and tasks [58].

3.2 Understanding and evaluating perception for data analysis in IE

Perception plays a major role in data visualisation, providing the fundamental mechanisms underlying any visual data analysis and communication. A large body of research has been concerned with perception on 2D displays, also for data analysis. Immersive environments often provide S3D visualisation, and through 3 or 6 degrees of freedom also different perspectives and frames of reference for the data representation. Large display walls, and CAVE or VR environments also often support larger display space and field of regard than classical desktop environments. Occlusion, overload, depth distortion, and distance can however make perception, visual search, and the mapping of spatial organisation to a mental map more difficult. Thus, the effect and limits of IE for perception in analysis tasks need to be investigated and characterised, and corresponding guidelines created regarding effective encoding and layout of data in IE. There is significantly less research so far in this direction compared to the research on classical 2D displays. An initial step in that direction is the investigation of graphical perception for IA [82], which compares three display modalities for scatterplot visualisation in a limited setting of 100−200 data points, including VR and AR HMDs.

3.3 Understanding and supporting the mental map

The problem of supporting the creation and maintenance of a mental map is concerned with the question of the internal representation of knowledge on a certain data set, its structure and content, and the connection to the external representation, i. e. the computer-mediated content. The visualisation, but also the interaction with and the navigation in the data and representation spaces, need to be designed in a way that facilitates this maintenance as well as the user’s orientation and cognition using this map. While for standard 2D environments the investigation of the mental map with respect to data analysis has been conducted already for many years [5], [59], the transfer to IE has been performed only recently [45]. In particular the effects of S3D, spatial immersion, and multi-modal representation and interaction are of interest. The exploitation of the spatial memory through spatial organisation and physical navigation [76], as well as the effects of large field of view and field of regard and of direct interaction [12] are supposed to improve the mental map, which however needs further evidence from user studies.

3.4 Scalability

Scalability in IA has several aspects, including the limits of human perception and cognition and the technological restrictions such as rendering performance. While more and more powerful hardware supports to render large numbers of objects in virtual and augmented environments, the requirements for low-latency high-framerate rendering of content are still demanding as even a small deviation can have detrimental effects on the user experience [11] and subsequently on the analysis. A major challenge in IEs that goes beyond the considerations for traditional visual analytics [66] is to exploit the specific IE affordances to create scalable representations and navigation schemes. While the large design space offers opportunities for scalability that go beyond traditional environments, the design has to be carefully considered to not put too much strain on the user and to balance the tradeoff between the potentially influencing factors for efficiency, effort, and orientation such as the number of navigation operations, representation levels, and the amount of physical navigation [6]. An important research direction for scalability is the investigation of the suitability of abstraction and aggregation approaches and potential transfer of established approaches into IE.

3.5 Interaction and navigation

Immersive environments provide a rich set of options for interaction with data representations, and navigation in the data and its representations. Gesture and speech recognition, eye tracking, physical navigation, physicalisation, mid-air interaction, and hand-held controllers can be used to support corresponding operations. These can provide direct interaction, for example with 3D representations of data points, or target indirect interaction, for example through 2D menus blended into a S3D environment. The designs have to be carefully chosen to ensure that the requirements are met, e. g. regarding required precision, and that no heavy burden is put on the user, e. g. avoiding effects such as the ‘gorilla arm’ or ‘midas touch/gaze’ [38]. Wang et al. [79] investigated the impact of matching the input and output device dimensionality. They found no evidence to suggest that the parity of dimensionality between input and output devices plays an important role on performance, but a tradeoff between accuracy and speed when using either 2D or 3D devices.

3.6 Transitional interfaces

Given the availability of a broad range of devices and technologies used in data analysis for representation and interaction, as well as the complexity of analysis workflows which might include multiple stages and a collaboration within or between groups, switching between different devices and environments might be necessary or advantageous for analysis. The research direction of transitional interfaces [35] looks into the requirements and the possible designs for such situations, with the goal that users can seamlessly switch between contexts, which are constituted by interaction spaces such as VR and AR, and potentially also by different data scales and representations. One vision for transitional interfaces is that systems from different locations along the RV-continuum can be integrated as coherent and consistent interfaces in a shared physical space [42]. That way, transitions between traditional 2D, VR, and AR would allow to choose the best available support of a task at hand, and also collaborations across degrees of virtuality could be facilitated [69].

4.1 Analysis of data in the context of the environment

4.1.1 Animal behaviour research

Figure 5

TeamWise animal movement analysis tool showing a visualisation of a bird returning from winter migration to Lake Constance (taken from Klein et al. [44] under creative commons license [18]).

Figure 6

Storks soaring, shown in VR with satellite imagery of the environment.

Animal behaviour research investigates the drivers and mechanisms of animal behaviour and interaction. Lightweight sensor tags allow to collect large data sets of movement and further parameters such as acceleration, and physiological or environmental conditions. The resulting trajectories and further derived data structures can be analysed to detect behavioural patterns and events and to categorise periods into classes of behaviour such as foraging or fighting [24]. Thus, a combination of spatial and temporal data is relevant for analysis. While this analysis can make use of automated analysis methods and statistics, the results as well as the original movement data often need to be investigated in the context of the geographic environment it occurred in. Both the integration of potential environmental drivers for an animal’s decision as well as the visual interpretation by an analyst are thus components that are required in the analysis workflow. Immersive environments are well suited to support the resulting combination of 3D spatial information, such as terrain, satellite imagery, trajectories and abstract data representations such as interaction networks or charts, see Figures 5 and 6. The display space can be used to show further environmental context that an animal might use as orientation, alternative perspectives to better analyse the animal’s perception of the environment or other animals, or for additional linked data views. An important associated question is which or how much information can be embedded in the 3D visualisation, and where instead linked views and 2D charts provide better overview and interpretation possibilities. The evaluation of the interpretation of the displayed information in a 3D visualisation is thus an important part of immersive analytics research.

4.2 Analysis of data with spatial referents

4.2.3 Brain data analysis

Brain data analysis can involve several aspects of importance for the analysis of brain-related questions, such as brain anatomy, structure, and function and temporal activity. The data can be acquired from sources such as electroencephalography (EEG) and functional magnetic resonance imagery (fMRI) measurements for activity, and diffusion tensor imaging (DTI) for structure, and MRI, computed tomography (CT) or positron emission tomography (PET) for the anatomy. General goals are a better understanding of the inner workings of the brain and its architecture, to find activity patterns, and to compare cohorts or individuals regarding structure or function. Typical applications are to detect, classify, monitor, predict diseases, such as Dementia or Alzheimer’s disease, as well as the impact of physical damage, to identify inter-species differences and similarities, and potential evolutionary developments, and to correlate behaviour and brain activity for a deeper understanding of the underlying mechanisms.

For most of these application tasks, both the understanding of the anatomy and structure as well as the topology of functional relations are relevant. Due to this combination of spatial information, such as the brain’s anatomy and structure, and abstract data, such as activity correlations, IE that use S3D representations of the brain as well as charts and abstract representations, e. g. node-link diagrams, have the potential to provide more direct and intuitive analysis interfaces. See Figure 7 for an example of such a combination. Immersive environments might not only help in the analysis of data [62], [39], but already in the preprocessing steps, e. g. the modelling of 3D structures from imaging slices [84].

Figure 7

Brain data analytics – a brain model that allows to distinguish anatomical regions is combined with brain activity data. The activity data acquisition provides time series that can be depicted in a classical chart, but also turned into a network by correlation analysis, which allows to map activity correlations between regions into the 3D model. Interaction allows to rotate the model and to select regions for in-depth analysis of their activity and correlations. Taken from [39], © 2019 IEEE.

The design of proper environments for use in practice poses however several open challenges, including proper metaphors for representation, intuitive interaction and linking, and the seamless integration of analysis methods [39]. In addition, the difficulties stemming from the dynamic nature of the data, the challenges stemming from the application and parameterisation of sophisticated processing steps along the analysis pipeline, and our still restricted understanding of the brain’s inner workings require not only interactive visualisations, and different views and perspectives, but also human-centered computational analytics to allow assessment of intermediate results, e. g. from image processing and parcellation, and subsequent adjustments to method settings.

Besides the dynamics, also the uncertainty that is built up along the processing pipeline [23] is a major challenge. Consideration of uncertainty in analysis methods as well as the integration of corresponding information for visualisation are still a major challenge in current research [80].

4.3 Abstract data

While mapping of spatial data, in particular three-dimensional data, into S3D environments looks like a natural choice, the value of S3D visualisations, in particular for abstract data, needs further motivation and justification [46]. Met with reservations from the data visualisation community for some time, visualisation of abstract data in IE still offers several potential advantages, and in particular in combination with spatial data these advantages can be exploited. Recent initial research shows the potential of S3D visualisations for exploration and analysis of abstract data such as high-dimensional data [34], [57], e. g. when dimensionality reduction is used, or networks [45], [17], [48].

Due to their prevalence in a variety of application areas, where they are used to model and represent data for both communication of knowledge and analysis purposes, networks have attracted high attention in recent research on IA. As they have no intrinsic spatial structure or direction, there is a lot of freedom in the design of visual representations, including encoding, layout, and metaphor. Most representations employ the node-link or matrix metaphor, of which the former lends itself more naturally to a 3D representation, see Figure 8. The layout is usually given as node positions and routing of the edges as curves or segments, and established layout methods such as the force-directed and stress minimisation methods can be easily extended to work on three dimensions. While layout methods can more easily optimise certain goals in 3D, such as distance representation due to the additional dimension, that however does not automatically lead to improved readability, e. g. due to perspective-dependent occlusion and distortion. Navigation, in particular in large-scale networks, poses a further challenge that is not yet well investigated, requiring efforts to better understand the mechanisms behind mental map creation, orientation in networks and S3D network representations, including the evaluation of traversal methods [75], [26].

Figure 8

Investigation of mental map preserving node-link network visualisation where a user interacts with an immersive 3D network visualisation in VR. Taken from [45], © 2020 IEEE.