Imprint: The content of this blog entry is based on VR experiments in the context of projects carried out in the scope of the authors masters degree studies at FHNW University of Applied Sciences and Arts Northwestern Switzerland under the supervision of Prof. Dr. Doris Agotai.
What is data exploration and what is the state of the art for exploration and visualization of dataspaces in Virtual Reality in general, and for data related to cultural heritage archives in particular? This article aims to define the terminology, show the benefits of VR for this use and present selected examples of related work in this field.
Terminology and definitions
Data exploration is defined by Idreos et al. as “efficiently extracting knowledge from data even if we do not know exactly what we are looking for”. Some key facets to achieve this objective are advanced data visualization and alternative exploration interfaces that help users navigate the underlying data space. [1]
A related term from the field of statistics is exploratory data analysis, which is an approach for data analysis that often employs visual methods with the objective of maximizing insight into a data set, uncovering underlying structures, extracting important variables, detecting outliers and anomalies, testing underlying assumptions, developing parsimonious models, and determining optimal factor settings. Seminal work on this topic was done by Tukey. [2]
Another term that is closely linked to the above is visual analytics, an approach that integrates visual and automatic data analysis methods. Thomas and Cook define visual analytics in their foundational work on this topic rather broadly as “the science of analytical reasoning facilitated by interactive visual interfaces”. [3] Kiem et al. give the following, more specific definition: “Visual analytics combines automated analysis techniques with interactive visualisations for an effective understanding, reasoning and decision making on the basis of very large and complex datasets.” [4] The goal of visual analytics is the creation of tools and techniques to enable people to (1) synthesize information and derive insight from massive, dynamic, ambiguous, and often conflicting data, (2) detect the expected and discover the unexpected, (3) provide timely, defensible, and understandable assessments and (4) communicate assessment effectively for action. [5] Visual data exploration is considered a subfield of visual analytics.
Figure 1: Visual data exploration a subfield of visual analytics. Figure based on the visual analytics process model by Kiem et al. [6]
As seen above, data visualization plays an important role in the data exploration task. Elena Zuidlova-Seinstra et al. write in their State of the Art Survey on Trends in Interactive Visualization that “the purpose of interactive visualization is to develop new methods to increase a person's abilities to explore and understand the data, so that an increased awareness of meaning in the data is possible. Interactive visualization techniques not only provide users with the possibility of viewing data but also permit them to use interaction capabilities to interrogate and navigate through datasets and communicate these insights with others.” [7] This affirms our hypothesis that interaction with the data set in combination with contextualized data visualisation will provide a better understanding of the data at hand und lead to the discovery of insights.
Benefits of VR for data exploration
Regarding the use of Virtual Reality technology in the field of data visualisation and exploration, a number of researcher have shown benefits in the use of VR for helping people better understand their data from a perceptual point of view. They notably have demonstrated that, with the use of appropriate depth cues, 3D perception can improve the intelligibility of the data and allow to disambiguate complex abstract representations. [8]
As a matter of fact, virtual environments such as the CAVE have since their inception been used for immersive data visualization and analytics and the rise of inexpensive stereoscopic head mounted displays increased the use of VR for this purpose. [9] Joseph J. LaViola Jr. et al. state that “there is a mounting evidence that immersive Virtual Reality’s ability to let users be “inside” their data or model and interact directly with the data through body-centric interaction (e.g. by moving their heads, bodies and hands) speeds up the processes of perception and interpretation”. The authors explain that such direct manipulation interactions offered by immersive Virtual Reality environments can be more fluid and more efficient in comparison to non-immersive data displays such as the standard desktop environment with interaction via keyboard and mouse-driven GUI, and thus reduce the cognitive load of the user. [10] Ongoing research in the field of immersive analytics is investigating how new interaction and display technologies can be used to support analytical reasoning and decision making. A particular focus of Chandler et al. is on bringing attention to higher-level usability and design issues in creating effective user interfaces for data analytics in immersive environments. [11]
Brief digression on automatic exploration
Modern data systems are generally designed for data retrieval rather than exploration. This can restrict knowledge discovery to expert users like data scientists. However curiosity and playfulness is what enables exploration, learning, creativity and eventually making sense of ever growing amounts of data. Exploration and curiosity are widely studied notions in artificial intelligence and machine learning within computer science, as well as in other fields such as neuroscience and behavioral psychology. Wasay et al propose a paradigm shift in data exploration with their concept of Queriosity, a system that autonomously proposes items of interest to the user. [12] We expect that novel approaches such as spatial user interfaces and interactive 3D data visualisation will likewise foster curiosity to never stop exploring and in return reward the user with insights and understanding.
Related Work
Research revealed that many museums and collections have an online presence offering search and filtering features to explore their inventory, interactive web-based maps are likewise widespread applications and VR technology has been employed to present cultural heritage objects in exhibitions, yet the use of VR for data exploration is still largely uncharted in this domain. Hereafter we present some selected notable examples we discovered in our investigation.
Presentation of Cultural Heritage Data by the means of Interactive and Immersive Systems
mARChive is a stereoscopic data browser that contains 100’000 objects of the collection of Museum Victoria in Melbourne. This navigable interactive 360-degree data landscape developed by Sarah Kenderdine and Jeffrey Shaw offers users an intuitive platform to engage with the objects found at the museum. The applications allows to select items from 18 topic fields and subsequently displays an image and meta data for selected data point. [13]
Figures 2-5: Museum visitor interacting with mARChive [13]
Open Heritage by Google Arts & Culture in collaboration with CyArk allows to explore iconic cultural heritage locations in 3D. Starting from a map offering an overview of heritage preservation around the world, the user is then launched into interactive scenes. In the presentation of the Bagan temple of Myanmar for in instance then another local map is offered as red thread for the user’s journey through this interactive narrated sequence. The focus of the Open Heritage project is on digital digital preservation of cultural heritage sites for future generations in light of destruction by natural disasters or human conflicts. [14]
Figures 6-7: Screenshots of Open Heritage's presentation of the Bagan temple of Myanmar where a local map is offered as red thread for the user’s journey through this interactive narrated sequence [14]
In the domain of historical VR, Lithodomos VR offers to experience and explore ancient worlds by the means of Virtual Reality for tourism, entertainment and education with high commitment to archaeological accuracy. [15] And the Singapore-based start-up Hiverlab is working to preserve heritage sites in a virtual world where users can get a close look at these famed structures and learn about the histories through interactive links within their VR headsets. Cultural heritage will eventually decline, therefore digitalisation is a way of conservation. The ambition of Ender Jiang, founder of Hiverlab, is to amass enough VR content to create kind of an archive or digital library of an expansive history of mankind.
The Venice Time Machine is an international scientific programme launched by the EPFL and the University Ca’Foscari of Venice. It aims at building a multidimensional model of Venice and its evolution covering a period of more than 1000 years. Kilometers of archives will be digitized, transcribed and indexed setting the base of the largest database ever created on Venetian documents. The information extracted from these sources will be organized in a semantic graph of linked data and unfolded in space and time in an historical geographical information system, both a “Facebook” and a “Google map” of the past. [17]
Figure 8: the Venice Time Machine project [18]
Exploration of Spatial Data by the means of Interactive Maps
In a special issue of the International Journal of Geographical Information Science devoted to visualization for exploration of spatial data the authors highlight the research by MacEachren et al. that proposes an approach for constructing knowledge from large spatiotemporal data sets by combining methods and techniques from knowledge discovery in databases and geographical exploratory visualization. [19] [20]
Besides that some fundamental research in the field of interactive maps for visual data exploration was accomplished by Andrienko and Andrienko. Their software system Descartes is designed to support visual exploration of spatially referenced data. Descartes offers two integrated services: automated presentation of data on maps, and facilities to interactively manipulate these maps. Because exploratory data analysis requires highly interactive, dynamic data displays, Descartes strives to develop various interactive techniques for map manipulation that could enhance the expressiveness of maps and thus promote data exploration. [21]
An other early contribution to research in interactive visualization of geo data is offered by GeoSpace. This interactive visualization system allows information seekers to explore complex information spaces. By putting strong emphasis on visual clarity, GeoSpace allows users to rapidly identify information in a dense display and it can guide a users' attention in a fluid manner while preserving overall context. [22]
Figure 9 & 10: Examples of 2D and 3D data visualization in GeoSpace [22]
In the field of cultural heritage, the UNESCO World Heritage Centre offers an interactive, yet not immersive map that gives an overview of cultural heritage sites around the world and emphasizes in particular on world heritage sites that are in danger.
More recently, Google Earth VR helps the world to see the world by bringing the whole wide world to almost photorealistic Virtual Reality. Users can walk around, fly over an area or even soar into space. Moreover the app allows to browse on a miniature globe and teleport yourself to any location. [24] [25]
VR Tools for Data Visualisation and Exploration
SphereViz is a 3D interface for the visual exploration of multi-dimensional image data sets. It allows users to freely walk through the data space and interact with the data objects through natural actions like grabbing or moving. By applying additional advanced interaction capabilities like interacting with the dataset’s dimensions directly within the VR environment it allows to visually search an archive and enables the discovery of relations between parameters and grouping of images with similar properties. The authors conclude that VR increases the legibility of large data sets and provides more intuitive techniques for browsing through data. [26]
Figure 11: Photo archive in SphereViz: items are positioned in the sphere according to their colour components. [26]
Substantial fundamental work is presented with iViz, a tool for scientific data visualization in VR, that allows to experiment with different approaches for multidimensional data representation. The user can easily select and shuffle which data parameters are mapped to which graphical variable (xyz axis, position, color, shape size, transparency, texture, etc.) in order to determine the optimal mapping choice for a given scientific application like clustering of different object classes or search for outliers. iViz has proven that immersion provides benefits beyond traditional visualization and leads to a demonstrably better perception of datascapes, more intuitive data understanding and a better retention of perceived relationships in data. [27]
Virtualitics is a data analytics platform in a collaborative virtual environment targeting business customers. It is developed by Prof. George Djorgovski and based on his previous research at the California Institute for Technology – where he worked inter alia on the iViz project mentioned above – and NASA. The tool offers better understanding of data through machine learning, immersive visualisation and collaborative analysis. In one of their showcases they demonstrate the exploration of data using innovative interactive 2D maps and 3D globes. [28]
Figure 12: Data exploration showcase by Virtualitics using interactive 3D globes. [28]
Open Data Exploration in Virtual Reality (ODxVR) is a research project by Linnaeus University that proposes novel interface design approaches that enable interactive visualization of Open Data within immersive VR environments. A particular focus is on natural interaction for information exploration. [29] [30]
Figure 13: The ODxVR project is concerned with the visualization and interaction with Open Data in immersive Virtual Reality environments. [30]
A recent landmark for data visualisation in VR is the Avon Longitudinal Study of Parents and Children (ALSPAC). University of Bristol’s ALSPAC Children of the 90s research on health and wellbeing holds an enormous amount of data, which scientists all over the world use to ask and answer important public-health questions. As part of the Big Data VR Challenge set up by the Wellcome Trust and Epic Games in 2015, LumaPie successfully created a fully functional, scalable VR visualization environment built up from the study’s data itself. This immersive multiplayer VR space allows researchers to intuitively interact with and manipulate the data doing remote collaborative analysis. The system employs custom designed visualisation methods that tap into the unique human ability to quickly recognise patterns in colour, size, movement and 3D spatial position. Researchers have the freedom to explore the data directly using 3D hand-tracking technology to point, click, slide and drag the data all around them. [31] [32]
Figures 14 & 15: interactive immersive data visualization environment for the ALSPAC data. [33]
conclusion
While the examples presented above show many interesting facets about the presentation of cultural heritage data by the means of interactive, immersive systems and serve as inspiration for data visualization in general and the exploration of spatial data by the means of interactive map in particular, the aspect of dealing with incompleteness and imprecision is hardly not addressed. A particular focus of this project is in the application of learnings from spatial data visualization and exploration techniques to cultural heritage data collections and their special demands regarding visualization fidelity.
[1] Idreos, Stratos, Papaemmanouil, Olga and Chaudhuri, Surajit (2015): “Overview of Data Exploration Techniques”. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD '15). ACM, New York, NY, USA
[2] Tukey, John W. (1977): “Exploratory Data Analysis”. Pearson.
[3] Thomas, J.J., Cook, K.A. (2005): “Illuminating the Path. The Research and Development Agenda for Visual Analytics”. IEEE Computer Society Press, Los Alamitos.
[4] Daniel Keim, Jörn Kohlhammer, Geoffrey Ellis and Florian Mansmann (2010): “Mastering the Information Age: Solving Problems with Visual Analytics”. Eurographics Association.
[5] Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, et al. (2008): “Visual Analytics: Definition, Process and Challenges”. In Andreas Kerren, John T. Stasko, Jean Daniel Fekete, Chris North. Information Visualization - Human-Centered Issues and Perspectives, Springer, pp.154-175, LNCS.
[6] Keim, D.A., Bak, P., Bertini, E., Oelke, D., Spretke, D., Ziegler, H. (2010): "Advancedvisual analytics interfaces". In: Proceedings of the International Conference on Ad-vanced Visual Interfaces. pp. 3–10.
[7] Zudilova-Seinstra, Elena, Adriaansen, Tony and van Liere, Robert (2009): Chapter “Overview of Interactive Visualization” in “Trends in Interactive Visualization, State of the Art Survey”, Springer.
[8] Grégoire Cliquet, Matthieu Perreira, Fabien Picarougne, Yannick Prié, Toinon Vigier (2017): “Towards HMD-based Immersive Analytics”. Immersive analytics Workshop, IEEE VIS 2017, Phoenix, United States.
[9] Jorge A. Wagner Filho, Marina F. Rey, Carla M. D. S. Freitas, Luciana Nedel (2018): “Immersive Visualization of Abstract Information: An Evaluation on Dimensionally-Reduced Data Scatterplots”. Proceedings 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
[10] Joseph J. LaViola Jr., Prabhat, Andrew S. Forsberg, David H. Laidlaw and Andries van Dam (2009): Chapter “Virtual Reality based Interactive Scientific Visualization Environments” in “Trends in Interactive Visualization, State of the Art Survey”, Springer.
[11] T. Chandler, M. Cordeil, T. Czauderna, T. Dwyer, J. Glowacki, C. Goncu, M. Klapperstueck, K. Klein, K. Marriott, F. Schreiber, et al. (2015), “Immersive analytics”. In 2015 Big Data Visual Analytics (BDVA), pp.1–8. IEEE.
[12] Wasay, Abdul, Athanassoulis, Manos and Idreos, Stratos (2015). “Queriosity: Automated Data Exploration”. In Proceedings of the 2015 IEEE International Congress on Big Data (BIGDATACONGRESS '15). IEEE Computer Society, Washington, DC, USA.
[13] Shaw, Jeffrey: “mARChive”. URL: https://www.jeffreyshawcompendium.com/portfolio/marchive, https://vimeo.com/137801550
[14] Google Arts and Culture: “Open Heritage”. URL: https://artsandculture.google.com/project/cyark, https://artsexperiments.withgoogle.com/bagan
[15] Lithodomos: “Ancient World Virtual Reality Content“. URL: https://lithodomosvr.com
[16] CNBC: “VR to make couch potato tours of the world's landmarks a click away”. URL: https://www.cnbc.com/2016/12/07/hiverlab-creating-a-virtual-heritage-of-the-worlds-most-historic-sites.html
[17] Frédéric Kaplan (2015): “The Venice Time Machine”. In Proceedings of the 2015 ACM Symposium on Document Engineering (DocEng '15). ACM, New York, NY, USA.
[18] Into the minds: "Visit to EPFL ArtLab featuring leading Big Data projects". URL: https://www.intotheminds.com/blog/en/visit-to-epfl-artlab-featuring-leading-big-data-projects
[19] Kraak, Menno-Jan and MacEachren, Alan (1999): “Visualization for exploration of spatial data”. In International Journal of Geographical Information Science, Volume 13, No 4.
[20] MacEachren, Alan M., Wachowicz, Monica, Edsall, Robert, Haug, Daniel and Masters, Raymon (1999): “Constructing knowledge from multivariate spatiotemporal data: integrating geographical visualization with knowledge discovery in database methods”. In International Journal of Geographical Information Science, Volume 13, No 4.
[21] Andrienko, Gennady L. and Andrienko, Natalia V. (1999): “Interactive maps for visual data exploration.” In International Journal of Geographical Information Science Volume 13.
[22] Ishantha Lokuge and Suguru Ishizaki (1995): “GeoSpace: an interactive visualization system for exploring complex information spaces”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95), Irvin R. Katz, Robert Mack, Linn Marks, Mary Beth Rosson, and Jakob Nielsen (Eds.). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA.
[23] UNESCO World Heritage Convention: “Interactive Map”. URL: https://whc.unesco.org/en/interactive-map
[24] Google Earth VR: “Your world awaits”. URL: g.co/earthvr
[25] Google: “Google Earth VR — Bringing the whole wide world to virtual reality” URL: https://youtu.be/SCrkZOx5Q1M
[26] M. Soldati, M. Doulis, A. Csillaghy (2007): “SphereViz. Data Exploration in a Virtual Reality Environment”. In Information Visualization, 2007. IV'07. 11th International Conference (pp. 680-683). IEEE.
[27] C. Donalek et al. (2014): “Immersive and collaborative data visualization using virtual reality platforms”. IEEE International Conference on Big Data. S. 609-614, IEEE.
[28] Virtualitics: “Empower your business through AI+VR”. URL: https://www.virtualitics.com
[29] Reski, N. and Alissandrakis, A. (2016): “Change your perspective: exploration of a 3D network created from open data in an immersive virtual reality environment”. In ACHI 2016: The Ninth International Conference on Advances in Computer-Human Interactions (pp. 403-410). International Academy, Research and Industry Association (IARIA).
[30] Linnaeus University: “Project: Open Data Exploration in Virtual Reality (ODxVR)”. URL
https://lnu.se/en/research/searchresearch/forskningsprojekt/project-open-data-exploration-in-virtual-reality-odxvr
[31] University of Bristol. “Virtual reality helps make sense of complex scientific data”.
URL: http://www.bristol.ac.uk/alspac/news/2015/vr-big-data-prize.html
[32] University of Bristol. “ALSPAC: Seeing is Believing”. URL: http://www.bristol.ac.uk/population-health-sciences/research/groups/data2knowledge/current-research-projects/alspac-seeing-is-believing
[33] Masters of Pie: "Big Data VR Challenge". URL: http://www.mastersofpie.com/big-data-vr-challenge