Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
  • Biennale experiment
  • Experiment billboard%26paintings
  • Experiment billboard
  • Experiment doubleline marker
  • Experiment frame%26paintings5
  • Experiment frame%26paintings
  • Experiment frame%26paintings4
  • Experiment simple marker billboard

Current State and Outlook

The result of our research so far is a functional prototype, implemented with Unity using the VRTK framework, optimized for the HTC Vive headset and controllers. Following the Visual Information Seeking Mantra it contains a map offering an overview of the full dataset, a body referenced on demand menu for filtering data points, and visualization concepts for direct integration of filter results in the map representation in the form of a Choropleth map encoding in combination with volume height adjustments. This immediate spatial representation of search results offers rapid insight on the contents of the dataset to the user. Thoughtful attention was given in respect to level of detail and visualization fidelity in the design and implementation of exemplary samples of collection objects. Applying the “details on demand” principle, a detail view with additional information is displayed by pointing at an item.

The application provides system control by the means of a perspective controller, allowing zooming in and out to shift the user’s perspective back and forth between bird’s eye view and ground level view. Spatial movement on the map is offered by physical locomotion and teleportation. And data management is provided through a JSON interface.

User tests have resulted in generally positive feedbacks, revealing that this form of spatial visualization and interaction with data provides benefits for the data exploration task and indeed offers a fresh perspective on data.

some impressions

prototype with control center
prototype closer


Figure 1 & 2: bird's eye view offering an overview of the full dataset


filter menu
world map prototype with filtes


Figure 3 & 4: body referenced on demand menu for filtering data points


visualization concept
visualization concept
visualization concept


Figure 5, 6 & 7: visualization concept for direct integration of filter results in the map representation in the form of a Choropleth map (no selection, Africa selected, Asia selected)


detail view
detail view


Figure 8 & 9: detail view of collection items, additional information is displayed on demand


world map prototype
world map prototype
world map prototype


Figure 10, 11 & 12: more impressions of the prototype


outlook

Future work includes refinement of menu design and introduction of more elaborated search and filtering options and giving a deeper look into orientation. For the detail view the display of additional attributes or behaviors of objects such as its provenance journey, plus interactions like marking favorites or comparing the behavior of multiple selected objects are envisioned.

Furthermore, freehand system control should be considered. The benefit of gesture input lies in the natural characteristics of direct mapping of user movement which are easier to learn than virtual control performed by an input device. Studies have shown that bimanual input in particular has the potential to allow users to perform tasks faster, leverage existing skills, and increase expressiveness. A drawback is the need for clear delimiters to indicate initialization and determination of gesture, otherwise normal human motions may be interpreted as gestures while not intended as such.

Recognizing that spatial visual representation, filtering and interaction capabilities provided by this virtual data exploration space result in more rapid understanding and unique insight on data in general, applying these findings to other datasets with geographical reference should be considered. Museums in particular have a significant amount of reusable data about their collections that often remains largely untapped, yet has the potential to create real value if presented in a meaningful way and made accessible for interaction, exploration and discovery, eventually allowing the documentation layer to become an actual experience. For this reason, every endeavor should be made towards reusability. This implies in particular advancements in the data management interface and modularity of the application architecture as well as on the elaboration of concepts to embed such a VR experience into the context of an online presence or as part of an exhibition.

No comments

Testing the Experience

Because VR is still a rather new technology and there are not many established interaction patterns yet, it is important to continuously get user feedback during the development process and apply the learnings immediately to the next iteration of the prototype. This frequent lean feedback loop happens in informal test settings as opposed to periodical more formal user and usability testing procedure, which may include a variety of aspects such as comprehensibility, effectiveness, efficiency, satisfaction, learnability, retainability, ease of use, speed of performance, rate of errors and user comfort.

Testing procedure

In our testing scenario the context, overarching idea and features of the application were briefly explained to the probands prior to launching the test user into the VR experience where they had the opportunity to explore the virtual data space applying the thinking aloud method. Then they were given specific tasks to solve, for instance changing the perspective, retrieving detail information about a collection item, applying filters to the dataset, or sharing their interpretation of the visualization concept. Following the test the probands were surveyed about their experience by the means of a questionnaire asking questions on the usability of specific features of the application, how they estimate their own performance in the given test tasks as well as their overall user experience.

Test subjects and summary of results

In total a sample of 8 users participated in this more comprehensive testing procedure. The probands were 75% male, 25% female, aged from 22 to 35 years with diverse occupational backgrounds. Most of them had little experience with VR stating that they had only tried it once or twice, or a few times before, yet some were more experienced with computer games.

More than the majority found the experience easy or rather easy to use and most had no problems with locomotion in the virtual space. The operation of the perspective controller was a little bit more challenging. The application of filters, obtaining detail information about collection items and comprehensibility of the data visualisation were perceived differently by the users resulting in mixed feedback. Nevertheless all users found the given task easy to solve. 7 out of 8 considered the experience to be very enjoyable or enjoyable, however there were mixed reactions concerning the insight provided, yet all reported that it offered them a fresh perspective on the dataset.

Insights from Testing

The testing observations resulted in many valuable findings. The most important learnings are summarized below:

  • Regarding the perspective controller, many test users commented that plus and minus icons (as in the previous version of the prototype) would be more intuitive for them than the arrow icons for up and down. In addition, seamless zooming instead of stepwise adjustment of zoom levels for shifting the perspective between overview and detail was suggested. And the user experience of this feature might be further improved if the the user would be moved along the line of sight instead of the vertical axis. It was also mentioned that it is difficult to keep orientation when on ground level. Stronger orientation support could probably be provided by the control center or additional orientation aid such as a compass.
  • Concerning filter interactions the consensus was that filters should be applied instantly. An additional option to select all or deselect all would be desired by some. And the default setting has to be a filter combination that provides an interesting selection of items.
  • Most users intuitively discovered how supplementary information about collection objects can be retrieved on demand and for those who did not discover this feature on their own, it was easy to apply the corresponding interaction pattern once a hint was given. Due to the orientation challenges on ground level perspective some users suggested that it would be convenient to find out more about a collection item by the means of magnification of the selected item, triggered by simply pointing at it instead of having to approach it for this purpose. However, we hold the view that the spatial movement intensifies the exploration process.
  • No attention was given to the control center. The reason for this is that it was almost invisible for most users, for some it was out of the field of view or illegible due to too small font size and others just suppressed it because the information on display was not considered to be relevant. A better solution might be to display the information only on demand, though larger in size and more prominent in position.
  • The abstract visualisation style of the application was well received by the majority of the test users, yet a visual differentiation between countries and surrounding might increase clarity. Only one person expressed the desire for a higher level of detail of geographical features such as rivers, mountains and so on. Additional landmarks might offer help for orientation in the virtual world. The higher-level data visualization by the means of the choropleth map was not obvious for the majority of test user, yet after being pointed out the feature was well received and comprehensible for all.
  • What may constitute legitimate reasons for the mixed reactions concerning the insight provided by the experience could be the fact that on one hand side the tested prototype only included a small portion of the original data set because the data owner hasn’t provided full access to the archive database yet and on the other hand side the intended filtering options have not yet fully been implemented. It is assumed that more data will lead to more connections and correlations between individual data points and more interaction possibilities will result in a more interesting exploration space. And the combination will eventually lead to the discovery of more insight.
  • Last but not least, most users of the target group are inexperienced with VR and therefore need clear instructions about the use of the hardware (the controllers in particular). Explanation on how to navigation in the virtual space as well as how to make use of the features provided by the application are required to take full advantage of the proposed data exploration experience. The best way to learn this would be by the means of an interactive tutorial in the way it is often done in games. To compensate for the lack of such an introduction, clear instructions need to be given to the user and the operative rules have to be explained by the host. In addition it can be useful to guide the user through a specific usecase to become familiar with the application.
    A way to demonstrate usage and interaction principles in one specific use case is by applying a narrative pattern called Martini glass structure, which begins with a single-path author driven approach following a tight narrative path (representing the stem of the glass). Once the intended narrative is complete, the experience is opening up to a user driven explorative stage where the user is free to interactively explore the data (representing the body of the glass). [1]

Testing has provided many valuable insights that will feed into the further development of the prototype. Although pointing out some minor issues it has revealed that this form of spatial visualization and interaction with data provides benefits for the data exploration task, that the application is engaging and enjoyable and that the experience indeed offers a fresh perspective on the given dataset.

[1] Edward Segel and Jeffrey Heer (2010): “Narrative Visualization: Telling Stories with Data” in IEEE Transactions on Visualization & Computer Graphics.

No comments

Orientation in the Virtual Space

Jeremy Bailenson, founding director of Stanford University’s Virtual Human Interaction Lab, states in his recent publication that “VR is about exploration whereas storytelling is about control” [1]. Yet even while striving for offering the highest degree of freedom by handing full system control over to the user and allowing him/her to explore the virtual (data) space at is own discretion, the user needs some support for orientation, navigation and wayfinding to take advantage of this freedom, so that the exploration process may not solely be based on serendipity and the vastness of the data space doesn’t feel like a maze. .

localization, wayfinding and navigation

The authors of “Understanding Virtual Reality” define navigation in VR as “the way of traveling through the space” and wayfinding as “knowing where you want to go” [2]. In the context of exploration one might need go even one step further back, namely knowing where your are. The term localization is used to determine the user’s current position within the virtual space and give him/her a system of reference that will be an assistance for the continuation of the exploration journey.

Some practical wayfinding aids for localization and navigation include

  • (world) maps provide a mental picture that is deeply rooted in the user’s mind. Information on the current position such as the country the user is located in serve as a trigger to activate existing knowledge about the surrounding.
  • offering a system of reference such as geographical coordinates.
  • orientation aid via the indication of true north or placing a compass at the user’s disposal.
  • hints such as leaving bread crumbs, some kind of guideposts or punctual instrumented guidance.
  • marking spots that have been popular with other users, because insight on an individual level can be gained from information that might not be new to other members of the community.

The purpose of such measures is to foster the discovery of insight by offering light guidance in the individual exploration process without reducing the freedom of choice of the user.

localization in the context of this project

While the user observes the data space from the bird’s eye perspective in our scenario, orientation is obvious, yet while flying over the world, the precise corresponding location on the ground might not be so obvious. Plus the real challenge is to keep orientation when the user switches perspective to ground level view. Support in this matter can either be offered directly within the virtual world, for instance by indicating the current location by the means of an avatar or marker on the ground, visually highlighting the underlying corresponding country or placing a label on it. Another approach that separates this additional information layer from the actual data space would be some kind of cockpit or control center, that provides the user with facts such as the geo coordinates of the current position, corresponding country, “flight” altitude, etcetera. In addition to localization related facts a summary of further relevant information could be offered on demand. This could include what filters have been applied to the currents selection and how many items are included as a result.

Prototype with Control Center

Figure 1: The prototype of the worldmap data exploration prototype offers a control center with localization related facts and a summary of other relevant information to the user.


[1] Bailenson, Jeremy (2018): “Experience on Demand: What Virtual Reality is, how it works and what it can do”. W. W. Norton & Company, New York, USA.
[2] William R. Sherman, Alan B. Craig (2003): “Understanding Virtual Reality”. Morgan Kaufmann Publishers, San Francisco, CA, USA.

No comments

Design Considerations regarding Display of Collection Items

In the previous article we explained the reasoning for some of the overall design decisions of our worldmap data exploration prototype. In this article aspects of consideration for the display of individual collection items such as the form of representation, their position on the map, the display of meta information and so forth are explained.

form of representation of the data points

Even though the collection mainly consists of physical objects, the database contains no 3D models but only photographs and metadata of the objects. On one hand side, this limits the forms of representation of the objects in virtual 3D space, yet on the other hand it poses a new challenge of finding an adequate display mode for 2D artefacts in a 3D environment. In order to create a spatial representation of the item, the photograph can be applied as a texture to a cube or a cylinder or a more common practice for this purpose is placing pictures on camera facing billboards. Concerning the camera facing mechanism, it has to be decided if all axes of the object should get aligned with the users line of sight or if the object should simple rotate around the z axis. With regard to the various perspectives the user can take from bird’s eye view to ground level view and everything in between, full adjustment of the object’s orientation is recommended.

An alternative approach would be to visualize merely the item’s meta data in the overview by the means of some abstract form of representation applying the item’s attributes to various dimensions such as size, shape, colour, transparency, etc. of a basic 3D object as it is done in many spatial data analytics tools, and only show the photograph of the item on demand. However this idea was not further pursued due to the fact that the attributes of the dataset contain mainly textual descriptions and categorical or discrete values, yet very few continuous values. Consequently there is no meaningful mapping of attributes to visual variables possible with the data at hand.

placement of objects

There are many aspects for consideration regarding to the placement of the the collection item objects and most of them relate to the issue of uncertainty visualisation because the dataset is rather imprecise in regards to the location of origin of the individual objects. For many of them only the country or broader region of origin is known, but there is no specific geographical information available, that could be translated into precise geo coordinates. This issue has been brought up previously and we have also discussed various approaches for the visualization of uncertainty.

In regards to visualization fidelity it is not advised to anchor the items in a precise location by the means of a flag pole, a pin or another kind of marker because that level of information about the location of origin simply is not known. Let the objects float over a certain area is a better expression of the level of accuracy. In addition, a random distribution factor could be added on purpose to the position. This would also prevent the probability that multiple items are located at exactly the same location. Another way of dealing with multiple objects in proximity would be some form of clustering or stacking of objects.

display of meta information

While the photograph of the item is considered to attract the highest interest of the target audience and therefore is displayed permanently, following the “details on demand principle” additional meta information should be available if requested. An intuitive interaction to express this request is simply pointing at an object.

Aspects of consideration for the display of meta information include which information should be displayed as well as the position where this information should be displayed, for example on top of the item itself, in close proximity, on some kind of menu floating in the virtual world or attached to the controller or in a fixed position on the user’s display. Further aspects include font size, font colour, background and text orientation. One has to balance these factors between readability, ensuring that the reference to the object is unambiguous, usability and aesthetics.

And on a side note, spoken text would for various reasons such as the general legibility of text in VR due to the limited resolution of current headsets for instance be a better option than text display, although audio presents its own challenges like leveling up with the noise level of the surrounding (taking into account that this application might be used in public spaces) or accessibility for users with hearing problems.

detail view
detail view
detail view

detail view


Figures 1, 3, 3 & 4: various drafts for the display of meta information

The data management of the meta information is provided through an interface, which enables automated import and export of data by the means of JSON files. Museum management database offers for each entry an object an identification number, name, short description, place and time of origin (both in textual format), material and dimension of the object as well as information about its provenance. This information is transformed into more structured data by adding additional attributes such as country code, start and end time of the time period (as numbers) to enable the filtering mechanisms offered by our prototype.

visual and interaction design

There are a few more points to mention in regards to the visual and interaction design of the collection items. But first of all it has to be acknowledged that the mARChive project by Sarah Kenderdine and Jeffrey Shaw which allows users to explore the database of the collection of Museum Victoria in Melbourne [1], has been a great inspiration for the visual design of the collection items. Even though this project is not realized in full immersive VR, learnings are applicable because they work with similar types of content and their experience is directed towards a comparable target audience.

One remaining point for discussion is the size of the collection item objects. A question of principle is whether the object’s dimensions should be fix or if it should dynamically change depending on the distance of the viewer. It can be argued that a set size is more realistic and reinforces the discovery approach as items get larger when the user approaches it. On the contrary, if the size adapts to the user’s perspective in the same way as the width of the country outlines, it can be ensured that items are recognizable at all times and a more consistent impression is conveyed. Moreover, it has to be decided if the object should have some kind of frame or border around the photo in order to stand out against the background. Such a measure for a clearer distinction seems expedient, yet the width and style should match with the surrounding.

When it comes to the filtering interaction with the objects, an additional aspect for visualization would be to somehow visually display the degree to which an item matches the selection criterions. And last but not least, some additional general interactions with the object could include to mark items as favorites, share them with others and eventually display which items are most popular among all users.

[1] Shaw, Jeffrey: “mARChive”. URL: https://www.jeffreyshawcompendium.com/portfolio/marchive, https://vimeo.com/137801550

No comments

Overall Design Decisions for the Worldmap Data Exploration Prototype

Because maps are something most users are familiar with and therefore offer a natural interface to access and explore the given data set, we chose to pursue this scenario for our initial prototype. In this article we give reason for the design decisions that were taken over the iterative implementation process of this prototype.

map representation

We have discussed various forms of representation of the world in VR space in a previous article and observed that it depends on the objective of a VR experience whether virtual globes are the best way to display global data in immersive environments or if other representations are more eligible. A significant limitation of globes is that only one hemisphere can be observed at once and another observation is that the unfamiliarity with this perspective leads to disorientation of users. Given the objective of fast perception, we concluded that the presentation of data on a flat map offering a comprehensive overview is more suitable for the given use case.

navigation in the virtual world

A further general design question regarding the map representation is whether the map should be positioned below, in front of or tilted towards the user. This question is closely related to how the user should navigate around the virtual world. Having the map on the ground allows for more natural forms of spatial movement such as free walking (thanks to the support of physical locomotion in room-scale VR setups) and locomotion by the means of teleportation. And zooming between overview and close-up view can happen by allowing the user to be lifted up and down. Flying up naturally offers a bird’s view perspective whereas lowering down brings him/her to ground level view.
In contrast, when the map is in front of or tilted towards the user, he/she would float in space and other less natural and therefore presumably less intuitive navigation concepts – for example via indirect navigation through an input device such as motion controller or joystick – would be required for all directions of locomotion, specifically the vertical and horizontal movement in front of the map as well as for the “zoom factor” which is realized by moving closer towards or further away of the map.
Some of the benefits of simpler navigation and interaction techniques in regards to usability and user experience are that they are faster perceived and mastered particularly by inexperienced VR users such as the target group in our usecase, that they require less cognitive effort in the execution and consequently offer a more pleasurable spatial experience. Additionally, being in the world and being able to freely move around offers a higher degree of immersion and overall supports better the explorative approach pursued in this project, allowing the users to gain spatial memory and understanding about the dataset and eventually discover their own insights.

design of the map

There are a number of design question in regards to the visual style of the map:

  • What is an appropriate level of detail for country shapes? On a side note, it has to be considered that this is not only a design but also a performance question…
  • Should the countries have outlines or gaps to indicate the borders? And if so, how do we keep a consistent visual impression as users are looking at the map from different perspectives? A gap could look immense from a close up perspective, but be hardly recognizable from far away and the same applies to outlines. To deal with this challenge a concept was applied that is similar to the zoom level technique of geo information systems where, depending on the current zoom level, cartographical material or orthophotos with a different levels of detail are displayed. Country outlines with different line thickness tailored to the viewing distance of the user are dynamically loaded to convey a consistent visual language from every perspective.
  • Will the shape alone be sufficient for the user to recognize the country or is additional information such as a label with the country name, country code or a flag needed? And in case of a label, additional points of discussion include aspects such as if the full country name should be displayed and if so, in what language should the names be displayed (e.g. english or local language)? How to deal with autonomous regions like Tibet for example? Where to display the label in the case of countries with a rather small surface? How can the reference to the corresponding country be ensured if placed next to the area or would the country code alone be understandable for the majority of users? How can the legibility be ensured given that the user looks at it from different angles, distances and changing background conditions. Some approaches to deal with this issue is to to adapt the font size to the users distance and to adjust the orientation of the text towards the viewing direction of the user, which is a technique that is often used in VR and commonly referred to as camera facing billboards.
  • Additionally it needs to be determined what is the right measure of information density in each situation and for the given use case in general. For example should country labels only be displayed for countries that are of particular interest for a user at that moment (for example those closest to his/her position or those with the largest amount of collection items)? Or should collection items which are close to each other be displayed as clusters in the overview perspective in order to to prevent information overload?

data visualisation and interaction with the dataset

Our data visualisation concept aims to combine the visualization of information about the dataset as a whole with detailed information about specific data points and therefore operates on two layers: the former is incorporated into the map visualization and the latter is represented in the display of the individual collection objects as shown in the following figure.

choropleth map

Figure 1: the two layers of the data visualisation and interaction concept of the Worldmap Data Exploration Prototype.

In order to provide a fast overall impression about the contents of a given dataset, aggregated general information about the dataset such as the prevalence of objects matching certain criterias is directly embedded into the map representation itself in the form of a Choropleth map colour encoding in combination with volume height adjustments. While visualizing quantities through the height dimension might seem to be the most obvious form of spatial data visualisation, it should be noted that this form of expression alone might be difficult to read because scale variation makes judging distances, heights and areas in 3D space harder, and occlusion might at times remove relevant information from the field of view. Therefore, a combination of spatial data representation together with a visual encoding in the texture of the surface of a geographic object provide a more effective solution allowing fast interpretation of data. In concrete terms, this means that the texture of countries with many collection items on display is highlighted and the height of the 3D models of these countries is elevated.

choropleth map

Figure 2: A choropleth map is a thematic map in which administrative areas are colored or shaded according to the range in which the aggregated statistic of interest falls. The example shows the length of countrynames, lighter shades are indicating fewer characters, darker shades a greater amount of characters in the countryname. [1]

Additionally, interaction with the dataset is an essential factor for gaining a better understanding of its content. Our approach is to allow the user to filter the dataset by various criterions such as time span of creation, material, geographical origin, etc. and apply the filter results in real time to the global data visualization rendering concept explained above. And in addition to the overall presentation, collection items that are included in the selection are highlighted by being expanded. This immediate spatial representation of filter results offers rapid insight on the contents of the dataset to the user.

choropleth map implementation

Figure 3: screenshot of the prototype showing the Choropleth map implementation

visual design

And last but not least there are the obvious design questions regarding the general aesthetic such as overall style, colour schema, lighting mood and so forth which exhibit a certain degree of subjectivity and can be rather hard to measure. Filonik and Baur have researched the topic of measuring aesthetics for information visualization and conclude that “aesthetics is an unsolved problem of information visualization, because there is no satisfactory understanding of what constitutes aesthetic effect”. Yet they also point out that empirical studies have shown a correlation between perceived aesthetics and usability. [2] And Katja Kwastek states in Interaction in Digital Art that “materiality and interpretability should be understood as complementary components of aesthetic experience” [3] confirming that the choice of material and style can contribute to the comprehensibility of the data on display.

The leading design principles for our VR experience are simplicity, information density, focus and comprehensibility.

Simplicity in a broader sense applies to being intentional on how much visual detail information is really required. Lindeman and Beckhaus who coined the term “experimental fidelity” encourage VR creators to carefully curate the user experience yet not by increasing the level of detail, but rather by reducing it to the essential components [4]. And Salen and Zimmerman address this issue in the domain of computer games and use the term “immersive fallacy” to denote the widespread misbelief that illusions should be as realistic as possible [5]. For this reason we intend to apply an adequate level of detail in the visual design of the components. For instance not every minor detail of the world map is needed, simplified shapes and outlines fulfill their purpose (and might improve the performance as a side effect).

Simplicity concerns not only the visual aspect but also an adequate level of information density. According to Claudia Giannetti, the theory of apperception states that a “oversupply” of information causes irritation of the user, whereas “undersupply” can lead to boredom. [6] Dealing with larger amounts of data, the lower bound is never an issue, the challenge is to show just the right amount of visual information to the user to prevent a cognitive overload. It is interesting that the author subsequently recommends to use a multilayer model to keep the right balance between the two extremes which is exactly the approach taken with this prototype.

The virtual environment should not take away the user’s attention from the actual content and colour can plays an important role in guiding the user’s focus. Therefore only the collection items are represented in their original colour to be in the center of attention whereas the chromaticity of the surrounding has deliberately been reduced to grayscale with low luminosity for countries with little or no prevalence of collection items and increasing brightness for countries with higher concentration of collection items.

Last but not least emphasis should be given to comprehensibility. It is important that the chosen forms of visualisation can be interpreted by the target audience, which are by the majority not expert users nor data scientists. While we strive for simplicity and higher degree of abstraction might offer interesting visual appeal, employing familiar forms of representations allow to tap into prior knowledge. For this reason a conventional world map was chosen to convey the geographical context.

[1] GisGeography (2018): “Choropleth Maps – A Guide to Data Classification”. URL: https://gisgeography.com/choropleth-maps-data-classification/
[2] Filonik, Daniel and Baur, Dominikus (2009): “Measuring Aesthetics for Information Visualization”. Information Visualisation, 13th International Conference. 579 - 584.
[3] Kwastek, Katja (2013): “Aesthetics of Interaction in Digital Art”. MIT Press, Cambridge, Massachusetts.
[4] Lindeman, Robert W. and Beckhaus, Steffi (2009): “Crafting memorable VR experiences using experiential fidelity”. Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, Seiten 187-190. ACM, New York.
[5] Katie Salen and Eric Zimmerman (2004): “Rules of Play - Game Design Fundamentals” MIT Press, Cambridge, Massachusetts.
[6] Giannetti, Claudia (2004): “Ästhetik des Digitalen: Ein intermediärer Beitrag zu Wissenschaft, Medien- und Kunstsystemen”. Springer Verlag, Wien/New York.

No comments

Interaction with Data in VR and 3D UI Design

Interaction and UI design for virtual environments are still a rather novel discipline where not many standard design patterns have been established yet. This leaves much room for exploration and therefore present an interesting challenge. In this article we will look into theory and current research on this topic and investigate if proven principles from 2D interaction and UI design can be transferred to virtual space.

VR pioneer LaViola et al. write in the introduction of their reference book on 3D User interfaces that „although we live and act in a 3D world, the physical world contains many more cues for understanding and constraints and affordances for action that cannot currently be represented accurately in computer simulation. Therefore great care must go into the design of UIs and interaction techniques for 3D applications. It is clear that simply adapting traditional WIMP interaction styles to 3D does not provide a complete solution to this problem. Rather, novel 3D UIs based on real-world interaction or other metaphors must be developed.“ [1] With this statement the authors point out the importance of developing adequate interaction patterns and menu designs for virtual environments.

Basics of spatial interaction design

Concerning types of interaction, 3D interaction theory distinguishes between direct user control, virtual control and agent control. In the first case, interface gestures that mimic real world interaction are applied meaning that the user directly interacts with a virtual object as he would with a physical object. Interaction fidelity is a term that is frequently used in this context to describe the objective degree of exactness with real world actions [2]. An example for high interaction fidelity is the isomorphic outlook, where strict geometrical, one-to-one correspondence between hand motions in the physical and virtual worlds is suggested, reasoning that more natural interaction patterns are better for users. Yet this is often impractical and ineffective because of user’s limitations such as arm length meaning he/she can only reach items within that limited radius. In the second case of virtual control, system control is effected by forms of indirect manipulation, for instance by the means of menus, commands, physical controllers or laser pointers among many others.

Another general question regarding interaction design is whether interactions should be effectuated single handed (unimanually) or with both hands (bi-manually) and in the latter case, if the actions of the two happen synchronously or asynchronously and an additional distinction according to Guiard’s framework for classification is made depending on the role of the dominant and the non-dominant hand. Studies have shown that bimanual input in particular has the potential to allow users to perform tasks faster, leverage existing skills, and increase expressiveness [3]. However it is also more difficult and time-consuming to acquire that skill and input gestures may be prone to errors if they are not carried out properly. Consequently, bi-manual control is recommended for VR experiences targeted to expert users who will presumably use that application recurringly, whereas for single usage and target groups with little experience in VR unimanual interaction patterns are more sensible.

abstract, real and magical 3D user interfaces

When designing 3D user interfaces, the degree of abstraction is a big question. Some school of thoughts believe that the closer an interface or a virtual environment resembles the real world, the easier the usage. The grounds for this assumption are that in a more realistic setting, novice users can transfer precognition from real life to the virtual environment and recognize affordances from their everyday life experience. However if the virtual representation differs in some minor aspect from the user’s expectation, the transfer learning effect is lost.

In fact, it has long been argued that the real power of 3D UIs lies not only in imitating physical reality and simulating real-world features, but also in creating a “better” reality by utilizing magical interaction techniques. By creating mappings and interactions that are specifically tailored to 3D environments, actions and processes can be made easier than in real life and an advantage of VR is that it allows users to overcome many human limitations that are so prominent in the real world such as limitations of our cognitive, perceptual, physical and motor capabilities [4].
In some case this is done by enhancing the user’s natural capabilities (e.g. offering the ability to fly to allow the user to view the world from bird’s eyes perspective). Yet in other cases, constrained interfaces that are designed to be simpler than the real world by restricting movement, limiting interface action and keeping interface objects in a plane are more suitable for a given task. An intriguing possibility is that enhanced 3D interfaces might offer simpler navigation, more compelling functionality, safer movements and less occlusion, than 3D reality, especially for information exploration and visualisation tasks. In conclusion, clever 3D designs prioritize facilitation and simplification of user tasks over mimicking reality [5].

Placement of Menus in VR

According to the initial frame of reference for the design of virtual menus offered by Jacoby and Ellis, invocation, location, and reference frame are considered as the most important aspects [6]. In the virtual space, there is no menu bar as it is the standard in desktop applications nor are there any borders or corners to which the menu could be attached to. The term spatial reference is used to describe the placement of the menu in virtual space. The distinction is made between world referenced (attached to a specific location in the virtual space), object referenced (attached to an object), head referenced (showing the menu in a constant ratio to the users position, for instance on a “head up display” at a fixed position of the users field of view) and last but not least body or device reference (for example attached to the hand controller or wrist of the user).

Related to menu placement is the issue of occlusion. Is a menu permanently shown at a fixed position of the screen, it might hide other relevant information from the user’s field of view. The challenge lies in finding the right balance between visibility of the content and accessibility of the menu. For this reason, it is recommended to display menus prominently when needed, for example by evocation on demand. Although a concern with on demand is the following question: how will the user know about the functionality and remember the evocation mechanism when requested? The distinction is made between recall and recognition. Recall refers to knowledge in the head and recognition refers to knowledge in the world. In the context of this issue recall would mean that the display of the menu is evoked by pressing a certain button, applying a gesture or voice control command whereas recognition means that there is some memory aid in the virtual environment such as an icon, object or another visual representation, that indicates the on-demand availability of the menu. The latter is recommended because it lowers the cognitive load placed on the user.

3D Menu design

While over the past decades of software development interaction patterns for application control techniques in 2D applications were established, there is not yet a proven set of 3D menu techniques available to designers and developers. Oftentimes, well-known desktop interaction techniques are adapted to virtual environments, although menu solutions integrating 2D approaches into space face problems such as the greater skills required in reaching a menu item in space as well as the lack of haptic constraint and tactile feedback. Freehand menu selection in particular is an inherently difficult task, especially with increasing menu breath, and free moving hands in space cannot achieve the same precision levels as with physical input devices such as mouse or stylus. For this reason effectiveness is severely reduced when adapting a traditional 2D drop-down menu to virtual environments, plus it leads to exhaustion.

Dachselt and Hübner devised a taxonomy with classification criterias for 3D menus which includes the following aspects [7]:

  • intention of use: number of displayed items (e.g. limited or not, range or definite value), hierarchical nature
  • appearance and structure: geometric structure, structural layout, type of displayed data (e.g. text, image, 3D object, combination), size and spacing of items
  • placement: frame of reference, orientation (e.g. always facing user), repositioning
  • invocation and availability: visibility (e.g. whole time, temporarily, user-dependent), invocation, animation, collapsibility
  • interaction and I/O setting: interaction device dependence, application type and setting, dimensionality, feedback (e.g. highlighting selection, audio cue, etc.), visualization of selection path
  • usability: evaluation criterions such as selection speed, error rate, efficiency, user comfort, ease of use and learning as well as comparison with other menu solutions
  • combinability

Some prominent early examples of menus that were deliberately designed for immersive virtual environments include inter alia the TULIP Menu and rapMenu.

TULIP is a menu system using Pinch Gloves for tracking the position of the user’s hand. It’s name stands for “Three-Up, Labels In Palm”, meaning that three items are active at one time while the rest of the menu items are arranged in columns of three along the palm of the user’s hand. To access an active item, the user simply pinches the thumb to the appropriate finger. To access other items, the user pinches the thumb to the little finger until the desired item appears on one of the fingers [8, 9].
A further evolution of the TULIP menu is Leap Motion’s Hovercast menu, which radiates from the palm of the user’s hand. A wide arc of menu items extend his/her fingertips and follow the hand movement. The user can interact with menu items using the index finger of the opposite hand. On rotation of the palm towards the eyes, the menu fades into respectively out of view. Hovercast is highly customizable, and can include many nested levels of selectors, toggles, triggers, and sliders. [10]

The rapMenu (roll and pinch menu) developed by Ni, McMahan and Bowman is a hybrid of the best features of the ring and the TULIP menu. It is based on gestural commands, by rotating the wrist the user makes a pre-selection of a sector with four items and then selects one out of them via pinch gesture. The menu works for up to 16 options (constrained to groups of four) and can be extended by the means of nesting of hierarchies. To prevent unintended menu invocation, gestural commands are only interpreted when the user’s hand is point at the display which reveals, that this menu was not designed for immersive VR environments. However, this “effective zone” could be substituted by another mechanism. User studies comparing pie menus to linear menus have shown, that radial placement reduces target selection time and lowers error rate. [11]

TULIP menus
Hovercast Slider
RapMenuMain


Figures 1, 2 & 3: Examples of menus that were deliberately designed for immersive virtual environments: the TULIP Menu [8] (on the left), Leap Motion’s Hovercast Menu [10] (in the middle) and the rapMenu [11] (on the right).

Guidelines for 3D UI design

Shneiderman, who is also the author of the originator of the Visual Information Seeking Mantra, points out that advanced 3D UI designs are marked by their support of (1) rapid situation awareness through effective overviews, (2) reduced numbers of actions to accomplish tasks, and (3) prompt, meaningful feedback for user actions. He offers the following guidelines for 3D UI design [5]:

  • minimizing the number of navigation steps for users to accomplish a task
  • concern for readability of text, in particular good contrast with background and minimization of tilt orientation
  • avoidance of unnecessary visual clutter
  • simplification of user and object movement
  • error prevention
  • organisation of information or groups of items in structures that allow rapid visual search and enable spatial recall
  • enriching interfaces with haptic feedback and audio cues might provide additional benefits
  • overviews to provide users with big picture of an application
  • permit user actions on objects
  • give user control over amount of information, offer details on demand
  • implement dynamic queries to rapidly filter out unneeded items
  • recognizable and memorable icon designs

Applying these twelve guidelines to the implementation of a 3D UI increases the usability and user experience of a spatial environment.

Application to the Worldmap Data Exploration Prototype

Applying these learnings to the context of our prototype, that has been built on the hypothesis that real user value is created when data is presented in a meaningful way and made accessible for exploration and discovery, notably if the user is able to interact with it, and combining it with the previously presented Visual Information Seeking Mantra “overview first, zoom and filter, then details on demand” which also points out important aspects for interaction with data, leads to the following interaction concepts:

  • perspective control: facilitating a mechanism to switch between overview and detail, allowing to shift the user’s perspective back and forth between bird’s eye view (presentation of the dataset as a whole) and ground level view (individual collection item and its surrounding)
  • importance of filtering mechanisms: filtering is essential in order to make sense out of large amounts of data, therefore the application should allows to apply various filter combinations such as category, material, technique, purpose, origin, epoche, objects on display in current exhibition, etc. to the dataset
  • providing details on demand: allowing to interact with specific data points and displaying additional information for selected items. Further interaction possibilities could include marking favorites or comparing the behavior of multiple selected objects

Concerning interaction control, it is estimated that body referenced controls attached to the handles are the best solution for this use case as they minimize occlusion of view yet remain available anytime and are therefore easy to recall. And for menu design, given the many benefits we came to know through the research above, radial menus should be explored.
In any case, rapid prototyping should be applied and great attention be given to ongoing testing while iteratively devising interactions and menus suitable for the given task.

[1] Joseph J. LaViola, Ernst Kruijff, Ryan P. McMahan, Doug A. Bowman, Ivan Poupyrev (?): „3D User Interfaces - Theory and practice“. Addison-Wesley Educational Publishers Inc, New Jersey, USA.
[2] McMahan, Ryan Patrick (2011): ”Exploring the Effects of Higher-Fidelity Display and Interaction for Virtual Reality Games”. Virginia Polytechnic Institute and State University, Blacksburg, VA.
[3] Russell Owen, Gordon Kurtenbach, George Fitzmaurice, Thomas Baudel, Bill Buxton (2005): “When It Gets More Difficult, Use Both Hands – Exploring Bimanual Curve Manipulation”. GI 2005 Conference proceedings: Graphics Interface Conference, pp. 17-24.
[4] William R. Sherman, Alan B. Craig (2003): “Understanding Virtual Reality”. Morgan Kaufmann Publishers, San Francisco, CA, USA.
[5] Shneiderman, Ben (2003): „Why not make Interfaces better than 3D reality?“. IEEE Computer Graphics and Applications, Volume: 23, Issue 6.
[6] Jacoby, Richard H., Ellis, Stephen (1992): „Using virtual menus in a virtual environment“. Proceedings of SPIE - The International Society for Optical Engineering.
[7] Dachselt, Raimund and Hübner, Anett (2007): “Three-dimensional menus: A survey and taxonomy”, Computers & Graphics, Volume 31, Issue 1, Pages 53-65.
[8] D. A. Bowman and C. A. Wingrave (2001): "Design and evaluation of menu systems for immersive virtual environments". Proceedings IEEE Virtual Reality 2001, Yokohama, Japan, pp. 149-156.
[9] D. A. Bowman, C. A. Wingrave, J. M. Campbell, V. Q. Ly, C. J. Rhoton (2002): “Novel Uses of Pinch Gloves™ for Virtual Environment Interaction Techniques”. In “Virtual Reality”, Volume 6, Issue 3, pp 122–129. Springer, London, UK.
[10] Kinstner, Zach (2015): “Hovercast VR Menu: Power at Your Fingertips”, URL: http://blog.leapmotion.com/hovercast-vr-menu-power-fingertips
[11] Ni, Tao, McMahan, Ryan, Bowman, Doug (2008): “Tech-note: rapMenu: Remote Menu Selection Using Freehand Gestural Input”. 3DUI - IEEE Symposium on 3D User Interfaces 2008.

No comments

Visualization of Uncertainty

It is in the nature of cultural heritage data that datasets are often not complete or information is imprecise. We have briefly covered some of the challenges related to this issue in a previous blog post and will discuss this topic now more thoroughly.

information uncertainty

In natural science the term “uncertainty” is used to describe doubt about the validity of the result of a measurement. This could be due to for example an error in measurement, transmission or conversion. In the context of cultural heritage uncertainty is rather referring to missing information or imprecision of data. And more generally, information uncertainty refers to the impreciseness, vagueness, fuzziness, inconsistency, doubt or likelihood present in information. According to Pham et al. some of the sources of information uncertainty include limited accuracy in data entry or transformation (for example limited precision), missing data, incomplete definition, imperfect realisation of a definition, inadequate knowledge about the effects of the change in environment, personal bias, ambiguity in linguistic description, approximation or assumption embedded in model design methods or procedures [1]. Consequently, visualization fidelity describes the accuracy and exactness in the visual representation of such information uncertainties.

uncertainty visualization

According to Pang et al., uncertainty visualization is the “strive to present data together with auxiliary uncertainty information. These visualizations present a more complete and accurate rendition of data for users to analyze. [...] The ultimate goal of uncertainty visualization is to provide users with visualizations that incorporate and reflect uncertainty information to aid in data analysis and decision making. [2] Although information uncertainty brings more complexity to the visualisation problem, it is crucial to consider this issue in data visualisation and exploration. Brunet and Andújar state that “most immersive (and non-immersive) visualizations nowadays are fundamentally biased and can be even unreliable“ because “data is usually presented as realistic and plausible 3D information” and “uncertainty is largely ignored by present applications and virtual environments” [3]. And MacEachren stresses that despite the long history in uncertainty visualization research, there is no generally accepted strategy for leveraging visualization to cope with uncertainty. He suggests to take a visual analytics approach to tackle the larger challenge which is not just to present data accurately but ultimately to enable reasoning under uncertainty in all its forms. [4]
All sources agree that a visual distinction is required for portions of a dataset or model that contain aspects of uncertainty. Let’s have a look at the domain of archeology for learnings and inspiration how this can be done.

Learnings from Archaeology

Dealing with uncertainty and incompleteness is by nature a common task in the field of archaeology. Yet the emergence of 3D visualization tools resulted in a strive for an increasingly higher level of detail in digital reconstructions of historical and archaeological artefacts. Though the resulting almost photorealistic visualisations did according to Roussou and Drettakis not improve the experience but rather led to an information overload and diverted the users attention to irrelevant details. And additionally, such hypothetical renderings carry a deceptive connotation of a higher level of information instead of being truthful about the actual fragmented knowledge. In the meanwhile, the trend has shifted towards less photorealistic, yet more credible forms of visualization that place greater emphasis on the differentiation of facts and assumptions. this evolution has been labeled by the authors as “realness factor” [5]. Also Gershon puts emphasis on the importance of an accurate representation of the degree of incompleteness and uncertainty of information. He recommends to demonstrate this by purposefully choosing a lower level of detail or using visual metaphors conveying this notion of incompleteness [6]. In «Fantastic reconstructions or reconstructions of the fantastic? Tracking and presenting ambiguity, alternatives, and documentation in virtual worlds» [7] the authors describe a set of dynamically adjustable variables including color schemes and opacity levels to render images that are more transparent about the presence of ambiguity, evidence, and alternatives in virtual reconstructions. And Strothotte, Masuch and Isenberg make the concrete proposal to visualize the degree of uncertainty in reconstructions by visual variables such as line width, ductus or sketchy layout style and suggest to purposefully leave out details [8].

Visualizing Knowledge
Fantastic Reconstructions


Figures 1 & 2: examples for visualization of uncertainty given by Strohotte et al. [8] on the left and Kensek et al. [7] on the right.

Robert W. Lindeman and Steffi Beckhaus share the viewpoint of reduction to the essential and introduce the term «Experimental Fidelity» which they define as follows: «Experiential Fidelity is an attempt to create a deeper sense of presence by carefully designing the user experience. We suggest to guide the user‘s frame of mind in a way that their expectations, attitude, and attention are aligned with the actual VR experience, and that the user‘s own imagination is stimulated to complete the experience.» [9]
All sources agree on the importance of visualization fidelity, and some also emphasize to focus on the essential which leads to better comprehension and prevents cognitive information overload.

Transfer into practice

How can the principles and learnings listed above be applied to projects in practice? It first starts with being conscious of the issue of information uncertainty, then carefully examining the dataset and questioning the apparent. A next step is searching for suitable forms of visualization in respect to this. Some techniques suggested by Pang et al. to make users aware of locations and degree of uncertainties in their data include adding glyphs, adding geometry, modifying geometry, modifying attributes, animation, sonification, and psychovisual approaches. [2] However there is no uniform solution and what visualisation strategies will be most expedient depends largely depends on the dataset, the presentation medium, goals to be achieved and last but not least the target audience. For this reason it is crucial to finally test and measure if the visualization concepts are understood by the target audience.
Some specific aspects where the topic of uncertainty visualisation needs to be considered in this project:

  • geographical origin of objects: there are no geo-coordinates provided for the origin of the collection items and the textual description of the location of origin given in the dataset is rather vague. In many cases only the name of a region, province or even just the country of origin is stated. Anchoring items with a precise location marker such as a pin or a flag onto the map would imply a higher level of information than there is available. Some approaches to ensure visualization fidelity in this scenario would be to use a diffuse material (for example smoke, vapor or some kind of a particle system) or a rough sketched line for the anchor or leave out the fixture entirely and let the item float over an area.
  • age determination: for most objects the dating is not a specific point in time but a textual description of a rather vague time interval such as an estimated time span, a century or an epoque. In order to dynamically process this information it has to be converted into an interval of numeric values. As in the example above, this implies a higher level of information than what is actually available. And when positioning items on a timeline it has to be considered if they will be displayed as a point and if so where this point would be located (start date, end date, mean value...) or rather as a vector with diffuse beginning and end points to reflect properly the information uncertainty aspect of this attribute.

Conclusion

There is not the one and only model solution how to deal with the issue of visualization of uncertainty, it always depends on the context, the objective and the target audience. Though it always starts with careful examination of the data at hand and challenging the obvious. And whilst aiming at the highest possible level of visualization fidelity one should at the same time be sensible to the user’s cognitive load. The goal is to find intuitive ways for the visualisation of uncertainty we are not just throwing even more information at the user.

[1] Binh Pham, Alex Streit and Ross Brown (2009): Chapter “Visualization of Information Uncertainty: Progress and Challenges” in "Trends in Interactive Visualization, State of the Art Survey".
[2] Pang, A., Wittenbrink, C., and Lodh, S. (1997). Approaches to uncertainty visualization. Visual Comput. 13, 370–390.
[3] Brunet, Pere and Andújar, Carlos (2015): “Immersive Data Comprehension: Visualizing Uncertainty in Measurable Models”. Frontiers in Robotics and AI. URL: https://www.frontiersin.org/articles/10.3389/frobt.2015.00022/full
[4] MacEachren, Alan M. (2015): “Visual Analytics and Uncertainty: Its Not About the Data”. In Proceedings EuroVis Workshop on Visual Analytics (EuroVA). The Eurographics Association.
[5] Roussou, Maria and Drettakis, George (2003): “Photorealism and Non-Photorealism in Virtual Heritage Representation”. Proceedings of the International Symposium on Virtual Reality, Archeology and Cultural Heritage. Eurographics, Brighton, UK.
[6] Gershon, Nahum D. (1998): Visualization of an Imperfect World. IEEE Computer Graphics and Applications, p.43-45.
[7] Kensek, Karen M., Swartz Dodd, Lynn and Cipolla, Nicholas (2004): “Fantastic reconstructions or reconstructions of the fantastic? Tracking and presenting ambiguity, alternatives, and documentation in virtual worlds”.. Automation in Construction 13, Seiten 175-186. Elsevier, Amsterdam.
[8] Strothotte, Thomas, Masuch, Maic and Isenberg, Tobias (1999): “Visualizing Knowledge about Virtual Reconstructions of Ancient Architecture”. IEEE Computer Society. Proceedings Computer Graphics International, Seiten 36-43. The Computer Graphics Society, Los Alamitos, CA.
[9] Lindeman, Robert W. and Beckhaus, Steffi (2009): “Crafting memorable VR experiences using experiential fidelity”. Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, Seiten 187-190. ACM, New York.

No comments

Mapping Data

To visualize data means to form a mental model or image of something in order to gain insight. One particular way for doing this is by mapping, which is defined as “creating a graphic representations of information using spatial relationships within the graphic to represent some relationships within the data. The common and original practice of mapping is the scaled portrayal of geographical features, that is, cartography. In the contemporary sense of data visualization, it includes metaphorical extensions of geographical map conventions and literacies to other kinds of data, as well as innovative ways of visualizing data not clearly related to the geographical archetype. In popular vernacular, mapping can just mean organizing or systematizing information” [1]. Hans Ulrich Obrist further defines maps as “an abstraction of the physical or conceptual world – a symbolic depiction of a space or idea that allows one to understand and navigate an unfamiliar topography or complex topology. But while most conventional charts, plans and diagrams claim to offer an accurate, even objective picture of the world, each one is bound by the specific agendas of its creators and users” [2]. He points out that there is no neutral way for mapping data and stresses on the importance to carefully consider and choose factors such as visualisation methods, styles, type, medium and context when representing data because they influence how the data is perceived. Jay David Bolter writes in the context of medium and materiality that “the computer is not a neutral space for conveying information. It shapes the information it conveys and is shaped in turn by the physical and cultural worlds in which it functions” [3].

Geographical Maps

Maps and globes are widely used for visualizing qualitative or quantitative data with a spatial aspect. The choice of map projection type is one of the initial steps in the quest to represent the world, a sphere, on a flat planar surface. No projection is neutral, natural or given, they are all constructed, configured and underpinned by various and quite arbitrary conventions. The main distinction is made between conformal maps like Mercator’s projection and equal area maps like the Gall-Peters projection. The former has the appealing property of faithfully representing the shapes of regions on the map and has served as standard for atlases for centuries. It is fine around the equatorial areas, but it greatly exaggerates the sizes of features as it nears the polar regions. [4] The later depicts sizes accurately, the area covered by a feature on the map, is proportional to the true land area of the feature in real life, but now the shapes are distorted, again this distortion is greatest near the poles.

In between the two of them lies another common choice, the equidistant cylindrical projection (plate carrée). This projection gets neither the areas nor the shapes of map features correct, but adopts a compromise solution that distorts neither areas as badly as Mercator nor shapes as badly as Gall-Peters. [5] There are many other projections of the world, but all of them inevitably distort our view of the world because it is not possible to draw the surface of the earth on a 2D surface without significant spatial distortion.

Maps and Globes in VR

In contrast to the challenges of representing the world on a 2D medium outlined above, VR would allow to represent the world correctly in its original spherical shape. A key question however is if virtual globes are the best way to show global geographic data in immersive environments or whether maps or some other visualisations may be better. Some of the drawbacks are that a globe can only show one hemisphere which is a limitation for many global visualisations and another observation is that the unfamiliarity with this perspective leads to disorientation of users. In their paper “Maps and Globes in VR” the authors explore different ways to render world-wide geographic maps in VR. They compare the pros and cons of the 3D exocentric globe (user’s viewpoint outside the globe), the egocentric globe (viewpoint inside the globe) and the flat map and it’s variant the curved map (rendering map on a plane or section of a sphere). [6]

Mapping the world

Figure 1: Different apporaches for mapping the world in VR proposed by Yang et al. in "Maps and Globes in Virtual Reality" [6]: 3D exocentric globe, flat map, 3D egocentric globe and curved map (from left to right). Additionally, the authors present their work in a short video .

Other Types of Maps

As stated in the introduction, mapping can refer to any kind of systematic information visualization where data attributes are bound to visual encodings. Erik Champion states that “maps are not just instrumental artefacts, but also epistemic ones with a long history that have helped to organize our knowledge of the world for millennia.” [7] Besides of geographical maps there is a wide variety of other map types. For instance cartograms, a rather recent invention, are digitally modified maps which depict the areas and countries of the world not by their physical size, but by their demographic importance on a vast range of subjects. Each territory on a map displays its data graphically, becoming larger or smaller in proportion to other areas, which are in turn scaled according to their own data. On a population cartogram a country with twice as many people as another is drawn twice as large. This approach creates unpredicted views of the world and allows new perspectives for mapping the social dimension of our planet. [5, 8] In the context of the Serpentine Gallery Map Marathon Obrist even goes as far as denoting mapping as a curatorial device. [9]

Cartogram on Adult Literacy in 2015 by worldmapper.com
Cartogram on Absolute Poverty Inequality in 2016 by worldmapper.com

Figure 2: Worldmapper offers a collection of cartograms where territories are re-sized on each map according to the subject of interest. The subject in the examples shown above are Adult Literacy in 2015 and Absolute Poverty Inequality in 2016 [10]

Cultural mapping is a term introduced by organisations like UNESCO concerned about safeguarding cultural diversity to map distinct people’s tangible and intangible cultural assets with locals landscapes around the world. Other types of maps that have no geographical reference at all include but are not confined to the mapping of networks (often in the form of graphs), thematic maps, story maps, mind maps, tag clouds and various forms of text visualisation.

Visual Information Seeking Mantra for Information Visualization

In one of the foundational papers on information visualisation the UI and HCI pioneer Ben Shneiderman introduces the Visual Information Seeking Mantra “overview first, zoom and filter, then details on demand”. He emphasizes the importance of letting the user gain an overview of the entire collection by offering a zoomed out view at first and then allowing him/her to zoom in on items of interest. Moreover he/she should have the capability to filter out uninteresting items in real time by dynamic queries applied to the collection. He stresses that advanced filtering mechanisms should accommodate highly varied user needs regarding filtering features. And finally, allowing the selection of an item or group and get details on demand when needed. He recommends the overview plus detail view strategy, which he also calls context plus focus, and points out that as a consequence there is a need for navigation tools to pan and scroll through the collection [11].

The principles of Shneiderman’s Visual Information Seeking Mantra are applicable to information visualisation in any medium including new technologies such as VR and provide a useful strategy for presenting and interacting with data that is worth pursuing in this project. With regards to the importance of offering a comprehensive overview it is presumed that even though a spatial environment offers features such as a 3D globe, the mapping and presentation of data on a flat map would be more suitable for our use case. Evidently user tests will be needed to verify if this mapping approach contributes to the intended outcome of discovery of insights through exploration of dataspaces by the means of interaction and data visualisation.

[1] Bleisch, Susanne (2018): "TSM Information Visualization – Mapping".
[2] Obrist, Hans Ulrich (2014): "Mapping it out: an alternative Atlas of Contemporary Cartographies". Thames & Hudson, London UK.
[3] Bolter, Jay David and Gromala, Diane (2003): "Windows and Mirrors: Interaction Design, Digital Art and the Myth of Transparency". MIT Press, Cambridge, Massachusetts.
[4] McCarthy, Tom (2014): Foreword to "Mapping it out: an alternative Atlas of Contemporary Cartographies". Thames & Hudson, London UK.
[5] Dorling, Daniel, Newman, Mark, Barford, Anna (2008): "The Atlas of the real World: Mapping the way we live". Thames & Hudson, London UK.
[6] Yang, Yalong, Jenny, Bernhard, Dywer, Tim, Marriott, Kim, Chen, Haohui, Cordeil, Maxime (2018): "Maps and Globes in Virtual Reality". Eurographics Conference on Visualization (EuroVis).
[7] Champion, Erik (2015): "Digital and Non-Digital Cultural Methods For Mapping the World Around Us". Malta Workshop Knowledge Maps and Access to Digital Archives. URL: https://de.slideshare.net/nzerik/digital-and-nondigital-cultural-methods-for-mapping-the-world-around-us
[8] Hennig, Benjamin D. (2014): Chapter "Redrawn Territories" in "Mapping it out: an alternative Atlas of Contemporary Cartographies". Thames & Hudson, London UK.
[9] Serpentine Galleries: "Map Marathon". URL: https://www.serpentinegalleries.org/exhibitions-events/map-marathon
[10] Gotthardt, Tina and Hennig, Benjamin: "World Mapper". URL: https://worldmapper.org/
[11] Shneiderman, Ben (1996): "The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations". Proceedings 1996 IEEE Symposium on Visual Languages.

No comments

Some Challenges to Overcome when Working with Museological Cultural Heritage Data

Working with museological and cultural heritage data evidently brings some challenges along as this kind of data differs in its nature from data of other fields. Here are some ideas and approaches of how to tackle them.

Data structure and format

In order to allow any kind of automated data processing, structured data is required with unified scales of measurement. However many archives contain a mixture between quantitative and qualitative description of data, since information derived from expert knowledge is mostly qualitative in nature. To make use of textual descriptions, for example, requires to identify keywords and attribute them to an object in form of some kind of tags. Also for other scales of measurement harmonization might be needed.

Coping with the lack of standards

There is no global standard how archival or museological systems should be structured, each collection has its own structure. While in the past collection databases functioned as internal museum tools for the almost exclusive use of collection managers and curators, there was no obvious reason for standardisation and consequently little motivation to alter formats, classifications, and terminologies that characterized collections documentation. However nowadays, as Fiona Cameron points out, the presence of collection records on museum Internet sites has opened up database access for a wide range of potential users. Although this move has been touted in the linchpin solution to facilitating expanded public access to museum resources, the exact process for rendering collection databases truly useful and engaging to public users has remained undefined [1]. In consequence, it is difficult to combine data from multiple archives in one application and the reusability of an app developed for one particular dataset is limited.
Oliver Grau has written much on this subject, in particular in regards to the documentation of new media art [2] [3]. In addition, Hauck and Kuroczynski have made a proposition for a Cultural Heritage Markup Language, although their primary focus is on the creation and preservation of 3D assets of digital reconstructions [4]. Furthermore, the use of a museum collection management software like MuseumPlus or ArtPlus from zetcom for instance helps with the definition of data structure and documentation of collections and collection-related processes and could be an enabler of standards for this industry [5].

Dealing with incompleteness and imprecision

It is in the nature of cultural heritage data that datasets are often not complete or information is imprecise. An example for this is the geographical origin of objects. Instead of having a precise geolocation the given dataset only provides the name of a region, province or even just the country of origin. In such a case, where should the marker of such an object be placed on a world map – in the geometrical center of that province area or the capital city? Both proposition are based on assumptions and might not reflect the truth origin. When visualizing data it is important to be transparent about gaps in datasets. Careful consideration must be given not to imply a higher level of detail (LoD) in the data transformation process than the original level of information (LoI). In cultural heritage reconstruction the difference between LoD and LoI is referred to as level of hypothesis (LoH) [4]. Learnings from the field of of archaeology can be applied to this issue. Gershon puts emphasis on the importance of an accurate representation of incompleteness and uncertainty and recommends to purposefully choose a lower level of detail to demonstrate this [6].

dealing with polysemy

In Theorizing Digital Cultural Heritage Cameron writes that “collection objects can be seen as inherently polysemic. That is, they possess the potential to be interpreted in a variety of way, depending on the nature of their incarnation as museum objects, and are subject to perceptual fluctuations in meaning dependent on various factors, whether cultural, theoretical, disciplinary, institutional, or individual.” She gives an example of a silver teaspoon that can be classified as Industrial Art, Decorative Art, Silver, Industry or anything else depending on the context and concludes that “an object’s meaning or its classification, is not self-evident or singular, but is imposed on it depending on the position and aims of the museum.” [1]
A similar issue regarding the categorisation of collection items occurs in the given dataset. As a concrete example, the material attribute contains almost as many different types and variants as there are objects in the collection. For meaningful interaction and usability an aggregation will be needed to a manageable amount of superordinate categories. Careful considerations need to be made as this classification process can be highly subjective.

sensible choices and tradeoffs regarding visualisation

There are many factors to consider when making choices about the visual representation style starting with obvious factors such as color, shape, style, perspective, size and amount of data on display, medium, context, etc. It is evident that there is no neutral way to visually represent data points, each form of visualisation carries its own underlying message. Bolter writes in this context that “the computer is not a neutral space for conveying information. It shapes the information it conveys and is shaped in turn by the physical and cultural worlds in which it functions” [7]. Yet there are also many more less obvious design decisions that need to be considered. One of them is visualizing temporal evolution as the objects in the collection cover a long period of time over which many geopolitical changes have occured. Some collection items contain descriptions of historical geographical locations such as “Maya lowland” for example. How can the change of political structures over time be represented appropriately? The challenge posed regarding the visualisation of the geographical context is to find the right balance between historical fidelity on one side of the spectrum and orientation, comprehensibility and usability on the other side. An additional aspect that is also related to the map representation is a sensible approach to political issues such as disputed country borders and autonomous or independent regions.

dealing with a documentation layer

Another challenge lies in the fact that archival databases typically consist of artefacts such as photographs and metadata representing the actual object. Yet this documentation layer is merely an image of the original and it should be taken into account that the reproduction might not be complete nor fully representing its substantiality and it can to some degree be an interpretation of the source object [8]. And an additional constraint is that in most cases there are only 2D images of objects available which limits the forms of representation in the virtual 3D space. While 3D scanning has become more accessible in recent years some of the reasons for not taking advantage thereof are the lack of knowhow and resources to create 3D models as well as a certain school of thought buzzing around that in the context of cultural heritage and art some objects should only be looked at from a particular perspective. For these reasons adequate display modes for 2D artefacts in a 3D environment have to be considered.

We will propose concepts of more in-depth solutions approaches to deal with the given constraints of museological cultural heritage data over the course of this project.

[1] Cameron, Fiona and Robinson, Helena (2007): Chapter "Digital Knowledgescapes: Cultural, Theoretical, Practical and Usage Issues Facing Museum Collection Databases in a Digital Epoch" in "Theorizing Digital Cultural Heritage. A critical Discourse" edited by Cameron, Fiona and Kenderdine, Sarah. MIT Press.
[2] Grau, Oliver (2013): "The Database of Virtual Art: for an expanded concept of documentation". ICHIM 03 – Art Access & Visual Education. Archives & Museum Informatics Europe. URL: http://www.archimuse.com/publishing/ichim03/016C.pdf
[3] Grau, Oliver and Haller, Sebastian and Hoth, Janina and Rühse, Viola and Schiller, Devon and Seiser, Michaela (2017): "Documenting Media Art: A WEB 2.0-Archive and Bridging Thesaurus for MediaArtHistories". Leonardo.
[4] Hauck, Oliver and Kuroczynski, Piotr (2015): "Cultural Heritage Markup Language: How to Record and Preserve 3D Assets of Digital Reconstruction". Proceedings of the 20th International Conference on Cultural Heritage and New Technologies 2015 (CHNT 20).
[5] Zetcom. URL: https://www.zetcom.com
[6] Gershon, Nahum D. (1998): Visualization of an Imperfect World. IEEE Computer Graphics and Applications, p.43-45.
[7] Bolter, Jay David and Gromala, Diane (2003): "Windows and Mirrors: Interaction Design, Digital Art and the Myth of Transparency". MIT Press, Cambridge, Massachusetts.
[8] K. Koebel, D. Agotai, S. Arisona and M. Oberli (2017): "Biennale 4D — A Journey in time: Virtual reality experience to explore the archives of the Swiss pavilion at the “Biennale di Venezia” art exhibition," 23rd International Conference on Virtual System & Multimedia (VSMM), Dublin.

No comments

Scenarios for the Exploration of Cultural Heritage Collection Data in VR

Based on the insight and inspiration through the State of the Art research we developed some conceptual scenarios for our use case. These scenarios propose three different interface metaphors for possible solution approaches how the given archival dataset could be presented in the VR space and what forms of interaction would benefit the targeted user group. The overarching goal of all scenarios is to offer new perspectives and give more context to the data.

Worldmap

focus on giving context to the geographical aspects of the dataset

data points will be positioned in the area of their geographical place of origin on a world map, possibly visualize the provenance journey of individual collection objects

Timeline

focus on giving context to chronological aspect of the dataset

data points will be positioned on a timeline offering a virtual time journey, possibly offer additional information on significant events and incidents on the timeline

Exploration Space

focus on understanding correlations between different attributes of the dataset

dimensions assigned to the axes can be changed in real time affecting the positioning of data points, possibly use other visual variables (e.g. size, colour, shape, transparency) to visualize additional attributes



While the proposed scenarios offer three different access approaches that serve as a starting points, they all aim to encourage users to launch on an exploration journey into the given data set. All approaches are specifically concerned with the presentation of the data overview. Following Shneiderman’s principles proposed in the Visual Information Seeking Mantra “overview first, zoom and filter, then details on demand” [1] all scenarios should additionally offer some kind of filtering mechanism as well as detail views for the individual data points.

Also interaction and navigation concepts need to be further examined and refined as well as design questions related to the issue of visualisation fidelity, which concerns all scenarios because the level of information for many attributes including the location of origin as well as creation date are not very precise in the given data set. In particular for the last scenario this will be a major challenge.

Over an iterative evolution process interactive prototypes will be developed that will also enable the testing of the usability and effectiveness.

[1] Shneiderman, Ben (1996): "The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations". Proceedings 1996 IEEE Symposium on Visual Languages.

No comments

State of the Art for Exploration of Dataspaces in VR

What is data exploration and what is the state of the art for exploration and visualization of dataspaces in Virtual Reality in general, and for data related to cultural heritage archives in particular? This article aims to define the terminology, show the benefits of VR for this use and present selected examples of related work in this field.

Terminology and definitions

Data exploration is defined by Idreos et al. as “efficiently extracting knowledge from data even if we do not know exactly what we are looking for”. Some key facets to achieve this objective are advanced data visualization and alternative exploration interfaces that help users navigate the underlying data space. [1]

A related term from the field of statistics is exploratory data analysis, which is an approach for data analysis that often employs visual methods with the objective of maximizing insight into a data set, uncovering underlying structures, extracting important variables, detecting outliers and anomalies, testing underlying assumptions, developing parsimonious models, and determining optimal factor settings. Seminal work on this topic was done by Tukey. [2]

Another term that is closely linked to the above is visual analytics, an approach that integrates visual and automatic data analysis methods. Thomas and Cook define visual analytics in their foundational work on this topic rather broadly as “the science of analytical reasoning facilitated by interactive visual interfaces”. [3] Kiem et al. give the following, more specific definition: “Visual analytics combines automated analysis techniques with interactive visualisations for an effective understanding, reasoning and decision making on the basis of very large and complex datasets.” [4] The goal of visual analytics is the creation of tools and techniques to enable people to (1) synthesize information and derive insight from massive, dynamic, ambiguous, and often conflicting data, (2) detect the expected and discover the unexpected, (3) provide timely, defensible, and understandable assessments and (4) communicate assessment effectively for action. [5] Visual data exploration is considered a subfield of visual analytics.

Visual Data Exploration

Figure 1: Visual data exploration a subfield of visual analytics. Figure based on the visual analytics process model by Kiem et al. [6]

As seen above, data visualization plays an important role in the data exploration task. Elena Zuidlova-Seinstra et al. write in their State of the Art Survey on Trends in Interactive Visualization that “the purpose of interactive visualization is to develop new methods to increase a person's abilities to explore and understand the data, so that an increased awareness of meaning in the data is possible. Interactive visualization techniques not only provide users with the possibility of viewing data but also permit them to use interaction capabilities to interrogate and navigate through datasets and communicate these insights with others.” [7] This affirms our hypothesis that interaction with the data set in combination with contextualized data visualisation will provide a better understanding of the data at hand und lead to the discovery of insights.

Benefits of VR for data exploration

Regarding the use of Virtual Reality technology in the field of data visualisation and exploration, a number of researcher have shown benefits in the use of VR for helping people better understand their data from a perceptual point of view. They notably have demonstrated that, with the use of appropriate depth cues, 3D perception can improve the intelligibility of the data and allow to disambiguate complex abstract representations. [8] As a matter of fact, virtual environments such as the CAVE have since their inception been used for immersive data visualization and analytics and the rise of inexpensive stereoscopic head mounted displays increased the use of VR for this purpose. [9] Joseph J. LaViola Jr. et al. state that “there is a mounting evidence that immersive Virtual Reality’s ability to let users be “inside” their data or model and interact directly with the data through body-centric interaction (e.g. by moving their heads, bodies and hands) speeds up the processes of perception and interpretation”. The authors explain that such direct manipulation interactions offered by immersive Virtual Reality environments can be more fluid and more efficient in comparison to non-immersive data displays such as the standard desktop environment with interaction via keyboard and mouse-driven GUI, and thus reduce the cognitive load of the user. [10] Ongoing research in the field of immersive analytics is investigating how new interaction and display technologies can be used to support analytical reasoning and decision making. A particular focus of Chandler et al. is on bringing attention to higher-level usability and design issues in creating effective user interfaces for data analytics in immersive environments. [11]

Brief digression on automatic exploration

Modern data systems are generally designed for data retrieval rather than exploration. This can restrict knowledge discovery to expert users like data scientists. However curiosity and playfulness is what enables exploration, learning, creativity and eventually making sense of ever growing amounts of data. Exploration and curiosity are widely studied notions in artificial intelligence and machine learning within computer science, as well as in other fields such as neuroscience and behavioral psychology. Wasay et al propose a paradigm shift in data exploration with their concept of Queriosity, a system that autonomously proposes items of interest to the user. [12] We expect that novel approaches such as spatial user interfaces and interactive 3D data visualisation will likewise foster curiosity to never stop exploring and in return reward the user with insights and understanding.

Related Work

Research revealed that many museums and collections have an online presence offering search and filtering features to explore their inventory, interactive web-based maps are likewise widespread applications and VR technology has been employed to present cultural heritage objects in exhibitions, yet the use of VR for data exploration is still largely uncharted in this domain. Hereafter we present some selected notable examples we discovered in our investigation.

Presentation of Cultural Heritage Data by the means of Interactive and Immersive Systems

mARChive is a stereoscopic data browser that contains 100’000 objects of the collection of Museum Victoria in Melbourne. This navigable interactive 360-degree data landscape developed by Sarah Kenderdine and Jeffrey Shaw offers users an intuitive platform to engage with the objects found at the museum. The applications allows to select items from 18 topic fields and subsequently displays an image and meta data for selected data point. [13]

mARCHive
mARCHive
mARCHive
mARCHive

Figures 2-5: Museum visitor interacting with mARChive [13]

Open Heritage by Google Arts & Culture in collaboration with CyArk allows to explore iconic cultural heritage locations in 3D. Starting from a map offering an overview of heritage preservation around the world, the user is then launched into interactive scenes. In the presentation of the Bagan temple of Myanmar for in instance then another local map is offered as red thread for the user’s journey through this interactive narrated sequence. The focus of the Open Heritage project is on digital digital preservation of cultural heritage sites for future generations in light of destruction by natural disasters or human conflicts. [14]

Open Heritage
Open Heritage

Figures 6-7: Screenshots of Open Heritage's presentation of the Bagan temple of Myanmar where a local map is offered as red thread for the user’s journey through this interactive narrated sequence [14]

In the domain of historical VR, Lithodomos VR offers to experience and explore ancient worlds by the means of Virtual Reality for tourism, entertainment and education with high commitment to archaeological accuracy. [15] And the Singapore-based start-up Hiverlab is working to preserve heritage sites in a virtual world where users can get a close look at these famed structures and learn about the histories through interactive links within their VR headsets. Cultural heritage will eventually decline, therefore digitalisation is a way of conservation. The ambition of Ender Jiang, founder of Hiverlab, is to amass enough VR content to create kind of an archive or digital library of an expansive history of mankind.

The Venice Time Machine is an international scientific programme launched by the EPFL and the University Ca’Foscari of Venice. It aims at building a multidimensional model of Venice and its evolution covering a period of more than 1000 years. Kilometers of archives will be digitized, transcribed and indexed setting the base of the largest database ever created on Venetian documents. The information extracted from these sources will be organized in a semantic graph of linked data and unfolded in space and time in an historical geographical information system, both a “Facebook” and a “Google map” of the past. [17]

The Venice Time Machine project

Figure 8: the Venice Time Machine project [18]

Exploration of Spatial Data by the means of Interactive Maps

In a special issue of the International Journal of Geographical Information Science devoted to visualization for exploration of spatial data the authors highlight the research by MacEachren et al. that proposes an approach for constructing knowledge from large spatiotemporal data sets by combining methods and techniques from knowledge discovery in databases and geographical exploratory visualization. [19] [20]
Besides that some fundamental research in the field of interactive maps for visual data exploration was accomplished by Andrienko and Andrienko. Their software system Descartes is designed to support visual exploration of spatially referenced data. Descartes offers two integrated services: automated presentation of data on maps, and facilities to interactively manipulate these maps. Because exploratory data analysis requires highly interactive, dynamic data displays, Descartes strives to develop various interactive techniques for map manipulation that could enhance the expressiveness of maps and thus promote data exploration. [21]

An other early contribution to research in interactive visualization of geo data is offered by GeoSpace. This interactive visualization system allows information seekers to explore complex information spaces. By putting strong emphasis on visual clarity, GeoSpace allows users to rapidly identify information in a dense display and it can guide a users' attention in a fluid manner while preserving overall context. [22]

GeoSpace
GeoSpace


Figure 9 & 10: Examples of 2D and 3D data visualization in GeoSpace [22]

In the field of cultural heritage, the UNESCO World Heritage Centre offers an interactive, yet not immersive map that gives an overview of cultural heritage sites around the world and emphasizes in particular on world heritage sites that are in danger.

More recently, Google Earth VR helps the world to see the world by bringing the whole wide world to almost photorealistic Virtual Reality. Users can walk around, fly over an area or even soar into space. Moreover the app allows to browse on a miniature globe and teleport yourself to any location. [24] [25]

VR Tools for Data Visualisation and Exploration

SphereViz is a 3D interface for the visual exploration of multi-dimensional image data sets. It allows users to freely walk through the data space and interact with the data objects through natural actions like grabbing or moving. By applying additional advanced interaction capabilities like interacting with the dataset’s dimensions directly within the VR environment it allows to visually search an archive and enables the discovery of relations between parameters and grouping of images with similar properties. The authors conclude that VR increases the legibility of large data sets and provides more intuitive techniques for browsing through data. [26]

SphereViz

Figure 11: Photo archive in SphereViz: items are positioned in the sphere according to their colour components. [26]

Substantial fundamental work is presented with iViz, a tool for scientific data visualization in VR, that allows to experiment with different approaches for multidimensional data representation. The user can easily select and shuffle which data parameters are mapped to which graphical variable (xyz axis, position, color, shape size, transparency, texture, etc.) in order to determine the optimal mapping choice for a given scientific application like clustering of different object classes or search for outliers. iViz has proven that immersion provides benefits beyond traditional visualization and leads to a demonstrably better perception of datascapes, more intuitive data understanding and a better retention of perceived relationships in data. [27]

Virtualitics is a data analytics platform in a collaborative virtual environment targeting business customers. It is developed by Prof. George Djorgovski and based on his previous research at the California Institute for Technology – where he worked inter alia on the iViz project mentioned above – and NASA. The tool offers better understanding of data through machine learning, immersive visualisation and collaborative analysis. In one of their showcases they demonstrate the exploration of data using innovative interactive 2D maps and 3D globes. [28]

Virtualitics

Figure 12: Data exploration showcase by Virtualitics using interactive 3D globes. [28]

Open Data Exploration in Virtual Reality (ODxVR) is a research project by Linnaeus University that proposes novel interface design approaches that enable interactive visualization of Open Data within immersive VR environments. A particular focus is on natural interaction for information exploration. [29] [30]

ODxVR

Figure 13: The ODxVR project is concerned with the visualization and interaction with Open Data in immersive Virtual Reality environments. [30]

A recent landmark for data visualisation in VR is the Avon Longitudinal Study of Parents and Children (ALSPAC). University of Bristol’s ALSPAC Children of the 90s research on health and wellbeing holds an enormous amount of data, which scientists all over the world use to ask and answer important public-health questions. As part of the Big Data VR Challenge set up by the Wellcome Trust and Epic Games in 2015, LumaPie successfully created a fully functional, scalable VR visualization environment built up from the study’s data itself. This immersive multiplayer VR space allows researchers to intuitively interact with and manipulate the data doing remote collaborative analysis. The system employs custom designed visualisation methods that tap into the unique human ability to quickly recognise patterns in colour, size, movement and 3D spatial position. Researchers have the freedom to explore the data directly using 3D hand-tracking technology to point, click, slide and drag the data all around them. [31] [32]

ALSPAC
ALSPAC

Figures 14 & 15: interactive immersive data visualization environment for the ALSPAC data. [33]

conclusion

While the examples presented above show many interesting facets about the presentation of cultural heritage data by the means of interactive, immersive systems and serve as inspiration for data visualization in general and the exploration of spatial data by the means of interactive map in particular, the aspect of dealing with incompleteness and imprecision is hardly not addressed. A particular focus of this project is in the application of learnings from spatial data visualization and exploration techniques to cultural heritage data collections and their special demands regarding visualization fidelity.

[1] Idreos, Stratos, Papaemmanouil, Olga and Chaudhuri, Surajit (2015): “Overview of Data Exploration Techniques”. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD '15). ACM, New York, NY, USA
[2] Tukey, John W. (1977): “Exploratory Data Analysis”. Pearson.
[3] Thomas, J.J., Cook, K.A. (2005): “Illuminating the Path. The Research and Development Agenda for Visual Analytics”. IEEE Computer Society Press, Los Alamitos.
[4] Daniel Keim, Jörn Kohlhammer, Geoffrey Ellis and Florian Mansmann (2010): “Mastering the Information Age: Solving Problems with Visual Analytics”. Eurographics Association.
[5] Daniel Keim, Gennady Andrienko, Jean-Daniel Fekete, Carsten Görg, Jörn Kohlhammer, et al. (2008): “Visual Analytics: Definition, Process and Challenges”. In Andreas Kerren, John T. Stasko, Jean Daniel Fekete, Chris North. Information Visualization - Human-Centered Issues and Perspectives, Springer, pp.154-175, LNCS.
[6] Keim, D.A., Bak, P., Bertini, E., Oelke, D., Spretke, D., Ziegler, H. (2010): "Advancedvisual analytics interfaces". In: Proceedings of the International Conference on Ad-vanced Visual Interfaces. pp. 3–10.
[7] Zudilova-Seinstra, Elena, Adriaansen, Tony and van Liere, Robert (2009): Chapter “Overview of Interactive Visualization” in “Trends in Interactive Visualization, State of the Art Survey”, Springer.
[8] Grégoire Cliquet, Matthieu Perreira, Fabien Picarougne, Yannick Prié, Toinon Vigier (2017): “Towards HMD-based Immersive Analytics”. Immersive analytics Workshop, IEEE VIS 2017, Phoenix, United States.
[9] Jorge A. Wagner Filho, Marina F. Rey, Carla M. D. S. Freitas, Luciana Nedel (2018): “Immersive Visualization of Abstract Information: An Evaluation on Dimensionally-Reduced Data Scatterplots”. Proceedings 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
[10] Joseph J. LaViola Jr., Prabhat, Andrew S. Forsberg, David H. Laidlaw and Andries van Dam (2009): Chapter “Virtual Reality based Interactive Scientific Visualization Environments” in “Trends in Interactive Visualization, State of the Art Survey”, Springer.
[11] T. Chandler, M. Cordeil, T. Czauderna, T. Dwyer, J. Glowacki, C. Goncu, M. Klapperstueck, K. Klein, K. Marriott, F. Schreiber, et al. (2015), “Immersive analytics”. In 2015 Big Data Visual Analytics (BDVA), pp.1–8. IEEE.
[12] Wasay, Abdul, Athanassoulis, Manos and Idreos, Stratos (2015). “Queriosity: Automated Data Exploration”. In Proceedings of the 2015 IEEE International Congress on Big Data (BIGDATACONGRESS '15). IEEE Computer Society, Washington, DC, USA.
[13] Shaw, Jeffrey: “mARChive”. URL: https://www.jeffreyshawcompendium.com/portfolio/marchive, https://vimeo.com/137801550
[14] Google Arts and Culture: “Open Heritage”. URL: https://artsandculture.google.com/project/cyark, https://artsexperiments.withgoogle.com/bagan
[15] Lithodomos: “Ancient World Virtual Reality Content“. URL: https://lithodomosvr.com
[16] CNBC: “VR to make couch potato tours of the world's landmarks a click away”. URL: https://www.cnbc.com/2016/12/07/hiverlab-creating-a-virtual-heritage-of-the-worlds-most-historic-sites.html
[17] Frédéric Kaplan (2015): “The Venice Time Machine”. In Proceedings of the 2015 ACM Symposium on Document Engineering (DocEng '15). ACM, New York, NY, USA.
[18] Into the minds: "Visit to EPFL ArtLab featuring leading Big Data projects". URL: https://www.intotheminds.com/blog/en/visit-to-epfl-artlab-featuring-leading-big-data-projects
[19] Kraak, Menno-Jan and MacEachren, Alan (1999): “Visualization for exploration of spatial data”. In International Journal of Geographical Information Science, Volume 13, No 4.
[20] MacEachren, Alan M., Wachowicz, Monica, Edsall, Robert, Haug, Daniel and Masters, Raymon (1999): “Constructing knowledge from multivariate spatiotemporal data: integrating geographical visualization with knowledge discovery in database methods”. In International Journal of Geographical Information Science, Volume 13, No 4.
[21] Andrienko, Gennady L. and Andrienko, Natalia V. (1999): “Interactive maps for visual data exploration.” In International Journal of Geographical Information Science Volume 13.
[22] Ishantha Lokuge and Suguru Ishizaki (1995): “GeoSpace: an interactive visualization system for exploring complex information spaces”. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95), Irvin R. Katz, Robert Mack, Linn Marks, Mary Beth Rosson, and Jakob Nielsen (Eds.). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA.
[23] UNESCO World Heritage Convention: “Interactive Map”. URL: https://whc.unesco.org/en/interactive-map
[24] Google Earth VR: “Your world awaits”. URL: g.co/earthvr
[25] Google: “Google Earth VR — Bringing the whole wide world to virtual reality” URL: https://youtu.be/SCrkZOx5Q1M
[26] M. Soldati, M. Doulis, A. Csillaghy (2007): “SphereViz. Data Exploration in a Virtual Reality Environment”. In Information Visualization, 2007. IV'07. 11th International Conference (pp. 680-683). IEEE.
[27] C. Donalek et al. (2014): “Immersive and collaborative data visualization using virtual reality platforms”. IEEE International Conference on Big Data. S. 609-614, IEEE.
[28] Virtualitics: “Empower your business through AI+VR”. URL: https://www.virtualitics.com
[29] Reski, N. and Alissandrakis, A. (2016): “Change your perspective: exploration of a 3D network created from open data in an immersive virtual reality environment”. In ACHI 2016: The Ninth International Conference on Advances in Computer-Human Interactions (pp. 403-410). International Academy, Research and Industry Association (IARIA).
[30] Linnaeus University: “Project: Open Data Exploration in Virtual Reality (ODxVR)”. URL https://lnu.se/en/research/searchresearch/forskningsprojekt/project-open-data-exploration-in-virtual-reality-odxvr
[31] University of Bristol. “Virtual reality helps make sense of complex scientific data”. URL: http://www.bristol.ac.uk/alspac/news/2015/vr-big-data-prize.html
[32] University of Bristol. “ALSPAC: Seeing is Believing”. URL: http://www.bristol.ac.uk/population-health-sciences/research/groups/data2knowledge/current-research-projects/alspac-seeing-is-believing
[33] Masters of Pie: "Big Data VR Challenge". URL: http://www.mastersofpie.com/big-data-vr-challenge

No comments

Who will benefit from this?

The purpose of the intended VR application for exploration of dataspaces in the context of museological cultural heritage collection is to enable users to gain a better understanding and insight about the data at hand by offering novel ways of representation and interaction with the data by the means of Virtual Reality. Allowing the exploration of dataspaces by a non-guided discovery approach, the intended VR experience will be of informative, self-educative as well as entertaining character. This follows Viola Rühse’s recommendation that “online archives and digital collections that combine an educational and an entertaining approach seem to be more socially valuable” [1].

Users and Use Cases

One might ask the question who would make use of an online collection or even such an virtual space for exploration of data sets? Johannes Beltz, who is in charge of curation and deputy director of Museum Rietberg pointed out in an interview that the existing web based online collection serves on one hand side exhibition visitors in the preparation of their on-site visit, and on the other hand art historians for instance for provenance research or curators and professionals of other cultural institutions for example in the search of loan objects when planning an upcoming exhibition and lastly, it’s a documentation of cultural heritage in general.

Following the Lean UX approach we designed some proto personas reflecting the anticipated categories of our target audience which by and large overlap with the four key user groups curators, collection managers, educators, and non specialists that were identified by Fiona Cameron in the Themescaping Virtual Collections research project. [2] According to Lean UX, “Proto-Personas are our best guess as to who is using (or will use) our product and why. Then, as we learn from our ongoing research, we quickly find out how accurate our initial guesses are, and how we’ll need to adjust our target audience (and persona) – and thus our design.” [3]

Protot Persona Peet Protot Persona Sara Protot Persona Joana

Figures 1-3: Three Proto Personas. Pictograms by Freepik and Flaticon under Creative Commons License.

Evidently more user research will be required to identify all user groups, learn more about their context and use cases and eventually understand underlying motives and needs. Also in-depth analysis of the use of the existing online collection would provide useful insights on current user behaviour and allow to see trends and tendencies in the use of the existing web platform.

However, the goal of this project is also to engage with new user groups. For this reason especially, only surveying the current user base and analyzing existing tools won’t lead to innovation. Serial entrepreneur Mark Cuban states: “Your customers can tell you the things that are broken and how they want to be made happy. Listen to them. Make them happy. But don't rely on them to create the future road map for your product or service. That's your job.” [4] And he further reinforces the importance of offering innovative, unconventional approaches to users: “Customers didn't really see the value or need until they saw the new product. When they tried it, they loved it.” [5] His statement goes in line with Steve Jobs famous quote “It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them." [6] Applying this argument to the project implies that only once the advantages of a VR tool for the exploration for dataspaces in the context of museological cultural heritage collections are demonstrated to potential user groups in form of a functional prototype, they will apprehend the benefits of this offer.

To sum up, user tests on the prototypes of our intended experience will be needed to verify the hypothesis that exploration of dataspaces in VR by the means of spatial interaction and contextualized 3D data visualisation will not only add value to the user by providing a better understanding of the data and leading to the discovery of insights but also that the exploration process will be regarded as an engaging and unique user experience that is associated with great joy of use.

[1] Rühse, Viola (2017): Chapter "The Digital Collection of the Rijksmuseum" in "Museum and Archive on the move: Changing cultural Institutions in the digital era". De Gruyter, Berlin.
[2] Cameron, Fiona and Robinson, Helena (2007): Chapter "Digital Knowledgescapes: Cultural, Theoretical, Practical and Usage Issues Facing Museum Collection Databases in a Digital Epoch" in "Theorizing Digital Cultural Heritage. A critical Discourse" edited by Cameron, Fiona and Kenderdine, Sarah. MIT Press.
[3] Gothelf, Jeff (2016): “Lean UX: Applying Lean Principles to Improve User Experience”. O’Reilly Media Inc., Sebastopol.
[4] Sturt, David and Nordstrom, Todd : “Delight Your Customers By Giving Them What They Didn't Ask For”. URL: https://www.forbes.com/sites/davidsturt/2014/01/03/delight-your-customers-by-giving-them-what-they-didnt-ask-for/#730220593ad5
[5] Cuban, Mark: “Mark Cuban on Why You Should Never Listen to Your Customers”. URL: https://www.entrepreneur.com/article/222501
[6] Wikiquote: "Quotes about Jobs". URL: https://en.wikiquote.org/wiki/Steve_Jobs, originally in BusinessWeek (25 May 1998)

No comments

Exploration of Dataspaces in the Context of Museological Cultural Heritage Collections

Goal of the Project

The goal of the project “Exploration of dataspaces in Virtual Reality in the context of museological cultural heritage collections” is to enable the user to gain a better understanding and insight of a given data set (such as it’s contents, size, dimension, etc.) and identify correlations, dependencies and patterns among the data entries. This is achieved by providing intuitive forms of interaction with the data and offering spatial visual representation. We aim to give more context to the data and develop suitable forms for visualisation that will give reference to the geographical and chronological aspects of the data points. In addition, the use of VR technology will enable higher immersion and open up new perspectives on the data on display for the viewer and as a result increase user engagement. For this reason strong emphasis is placed on user experience and usability.

In this project we will be working with data from the online collection of Museum Rietberg Zurich, which is the only art museum for non-European cultures in Switzerland exhibiting an internationally renowned collection of art from Asia, Africa and Ancient America. Their extensive online collection comprises in addition to the objects in the art collection (including sculptures, paintings, ceramics, textiles) of the museum also numerous historical photographs out of estates and therefore offers an interesting data set as use case for this project. The key findings of this project though should be applicable to other archives or collections in the domain of cultural heritage and beyond.

Definition, Function and Purpose of Collections and Archives

As this project is concerned with data exploration in the context of museological cultural heritage collections, we first define what is considered with the term collection and secondly, what purpose and function collections fulfill.

According to Oxford Dictionary the origin of the word “collection” is defined as “a group of accumulated items of a particular kind” and goes back to the Latin “collectio(n-)” which is rooted in the verb “colligere” and means to “gather together” [1]. In our research we observed that the term collection is often used interchangeably with archive which is not surprisingly defined as “a collection of historical documents or records providing information about a place, institution, or group of people” [2]. Both terms are used to describe the activity of collecting and preserving artefacts. While the term archive gives a stronger reference to the modality how collection items are documented and the infrastructure (for example a database) that holds the information, the use of the word collection often indicates that there exists a kind of outlet that displays its contents, such as a museum or exhibition.

While museums, archives and libraries are all a sort of repositories, Dietz et al. distinguishes them by their motive. Museums are curator-driven and provide only limited access to their holdings, usually through a particular interpretative exhibition context provided by curatorial and educational staff. Archives tend to be research driven and are often only accessible by appointment. And libraries are user-driven and their role is to provide access to a vast amount of material, where the user can freely roam and make his/her own connections between works. [3]

Youngjin Lee writes in “Museum and Archive on the move” on the significance of archives that “archiving means systemizing and preserving documents of tangible and intangible information” and “provide these information repositories for further use. [...] Archives coincide with museums and libraries in exhibiting and preserving or providing search facility services.” [4] With this statement he/she is highlighting the affordance of such dataspaces for search, research and exploration. And Okwui Enwezor, director of the “Haus der Kunst” in Munich, compares in the same publication archival systems with museological systems and collections by stating “while the archival system focuses primarily on the accumulation and preservation of documents, the museological system devotes itself to the collection and exhibition of artefacts, objects and artworks.” [5] He is making the same distinction as observed in the definition of the two terms above and putting emphasis on the presentation aspect of collections. He further cites Douglas Crimp who developed the connection of the museum to the function of the archive in his essay “The Postmodern Museum” and explains that “the semantic link between the two [museums and archives] is owing to their capacities to decisively nominate what is important and valuable. [...] In order to properly understand the function of the archive and the museum [...] we perhaps need to think about them on a meta-critical level, as sites for the accumulation and interpretation of signs and representation, rather than in their prosaic forms as containers of documents and artefacts.” [5] While validating that archives are a collection of large amounts of (semi-) structured data and acknowledging that there might be a bias how in the selection process of an archive or collection, he puts emphasis on the importance of interpretation of data and the insight that can be gained through the way it is represented.

Abhay Adhikari reinforces this line of thought: “Museums have a significant amount of reusable data about their collections, and this information can be combined with other sources to encourage discovery and learning. [...] data can offer unique insights and in the process create new ways of working, as well as new forms of storytelling that allow cultural institutions to develop new models of participation with audiences online.” [6] He puts particular emphasis on the fact that real value is created when data is presented in a meaningful way and made accessible for exploration and discovery, allowing users to interact with it. Also discussing the approach how information is presented and the user experience that is created in this process, Annet Dekker finally raises the question in “Collecting and conserving net art: moving beyond conventional methods“ whether “documentation can potentially become an actual experience?” [7]. This leads to the follow-up question if creating that kind of experience should be considered as ultimate purpose of archives and collections. It can be summarized that the function and purpose of collections and archives goes far beyond just the accumulation of data. Their real value lies in the manner how this wealth of information is presented and how the user can explore and interact with the contents to gain insight and understanding.

Looking at Museum Rietberg, their goals declared in the collection policy are (1) collection, (2) preservation, (3) presentation, (4) documentation and (5) cooperation [8]. While the first two objectives, namely collection and preservation, lay a foundation, the online collection is one of the means to pursues the other three targets of the overarching goal which are presentation, documentation and cooperation.


Purpose%20of%20online%20collection

Figure 1: Purpose and impact of the online collection.

Exploring dataspaces through interaction with and visualisation of data

The main emphasis of the work in this project will be on facilitating the exploration of dataspaces in VR by the means of (1) interaction with the data set and (2) its visualisual representation. We are planning to investigate the following aspects for interaction and information visualisation of the whole data set as well as individual data points.

interaction with data

  • filter and search data (for example type/category, technique, purpose, origin, epoche, objects on display in current exhibition...)
  • sort data by certain attributes or criterias (e.g. number of exhibition an object has been on display, most views in online collection, …)
  • compare the behaviour of specific data points
  • select, mark or manipulate an individual object
  • switch between predefined views

visualisation of data

  • geographic reference
  • chronological reference
  • visualisation of occurrence (any or none), quantity or frequency (amount, size), dimensions of data set
  • grouping/aggregation of similar objects, point out clusters, distribution and range
  • display of other attributes or behaviours of an object, such as provenance or other meta information

Why using VR for the exploration of such dataspaces?

We are convinced that in contrast to traditional 2D representation of data by existing web applications and archival software an immersive VR application – allowing novel forms of spatial interaction with the data set and providing more context to the data through 3D visualisation – will open up new perspectives and lead to a better understanding of the data and eventually to the discovery of fresh insight.

[1] Oxford Dictionary: “Definition of collection in English”. URL: https://en.oxforddictionaries.com/definition/collection
[2] Oxford Dictionary: “Definition of archive in English”. URL: https://en.oxforddictionaries.com/definition/archive
[3] Steve Dietz, Howard Besser, Ann Borda, and Kati Geber with Pierre Lévy (2014): “Virtual Museum (of Canada): The Next Generation”. Canadian Heritage Information Network, Ottawa.
[4] Lee, Youngjin (2017): Chapter "Asia as a Methodology" in "Museum and Archive on the move: Changing cultural Institutions in the digital era". De Gruyter, Berlin.
[5] Okwui, Enwezor (2017): Chapter "The death of the African Archive and the birth of the museum" in "Museum and Archive on the move: Changing cultural Institutions in the digital era". De Gruyter, Berlin.
[6] Adhikarim, Abhay: “The arts and culture sector must think about data … but differently“. URL: https://www.theguardian.com/culture-professionals-network/culture-professionals-blog/2014/mar/28/arts-culture-sector-data-impact
[7] Dekker, Annet (2018): “Collecting and Conserving Net Art Moving beyond Conventional Methods, 1st Edition“. Routledge, London.
[8] Museum Rietberg: “Die Sammlungspolitik des Museums Rietberg”. URL: http://www.rietberg.ch/de-ch/sammlung/sammlungspolitik.aspx

No comments

Exploration of Dataspaces in Virtual Reality

After working for an extended time on the Biennale 4D project I’ve been assigned with a new challenge: “Exploration of dataspaces in Virtual Reality in the context of museological cultural heritage collections”. The aim of this new project is to devise strategies for filtering, search and exploration of objects as well as developing suitable interaction patterns and forms of visualisation with reference to the geographical, chronological and contextual aspect. The subsequent development of interactive prototypes for VR experiences based on the dataset of the online collection of Museum Rietberg will showcase the findings. Furthermore we will investigate how the use of VR technology can add value to visitors of cultural institutions, either during their on-site visit or while browsing through the content of their online presence.

In the near future, you will read hear about our insights and discoveries concerning exploration of dataspaces in VR, in particular in regards to strategies for filtering and search, interaction design and visualisation, storytelling and narratives as well as user experience and usability.

No comments

More experiments for displaying archival artefacts within an immersive VR environment

In a previous post the challenges of displaying 2D objects within a 3D space were discussed and texture mapping was introduced as a possible approach to display archival photos within an immersive VR experience. In this article insights gained on this matter through further experiments are shared. One of the most common practice for this purpose is placing pictures on billboards. Aspects such as the design of these billboards, their positioning, the reconstruction of observation points as well as the visual markers that guide the user to these viewpoints and last but not least the user interaction, that triggers the display of the additional content in the proper context, were investigated.

Aesthetics and functionality of billboards

Various experiments were made to find an adequate visual expression for billboards fitting with the chosen aesthetics of the virtual environment. In addition to the prevelant billboards, that merely consist of a plane or thin object in the size of the image which then is applied to its surface as a texture, different styles were tested such as semiopaque billboards or billboards surrounded with a frame matching the grid aesthetics of the virtual environment [see figures 5 and 8]. Also questions concerning the interaction with the billboards were examined, for instance if a billboard should be visible at any time or only when an user is positioned within proximity of an adequate observation point of this artefact. And if so, whether it should appear instantly, fade-in over time or emerge by the means of another transformation. These considerations resulted in the following concept for the display of the billboards: when a user approaches the specified viewpoint (technically speaking if he enters into a given perimeter of the precise position), it triggers a slow fade-in transition of the artefact and reversely the item disappears by the inverse operation, when the user moves away from this position. In case of the experiments with framed billboards though, this surrounding structure remains visible at all times having an indicative value for stimulating the user's interest to further explore the area and discover the supplementary content provided [see figures 4 and 7]. Aspects like interaction radius and transition time of this feature were evaluated through user testing.

Reconstruction of original artefact position

In order to provide the best immersion experience to the user, the archive photo needs to be seamlessly integrated into the scene [see figures 2 and 9]. The prerequisite for this envisaged blend between artefact and virtual environment is two-fold: on one hand side, the image needs to be placed in the right position and on the other hand, the user needs to observe the experiment from the correct position, which is precisely the viewpoint from where the image was originally taken. It is crucial that vanishing lines and horizon in the shot are matching with the perspective lines of the building and that distance, scale and angles are fitting. Some approaches tested in our experiments to this effect include the attempt to reverse engineer the perspective by calculating vanishing points, camera height and scale [1]. However the attempt to reconstruct the correct position turned out to be a rather complex procedure and some parameters like the focal distance of the camera lens used for the original photograph had to be estimated. As a matter of fact, the authors of the paper "Tour into the picture: using a spidery mesh interface to make animation from a single image" state in the context of a similar endeavour as a precondition for an input image to be regenerated from a sence model, that the virtual camera position in 3D space is known [2]. The same requisite applies to the integration of an image into an existing scene. Another approach for the positioning was to test the photo matching feature provided in the software Sketch Up [3]. Both attempts did not render satisfactory results.

Eventually the visual approach of approximating the original position with the support of some self-made helper functions, that allowed to reduce the opacity of the image while adjusting the location of the billboard, produced the best result for the reconstruction. This approach was also confirmed by Horry, Anjyo and Arai in the paper mentioned above. While, according to these authors, rigid approaches such as computer vision techniques are not always applicable to achieve the objective in this context, they state that "it is relatively easy for us to roughly specify the vanishing point by manually drawing guide lines for perspecitve viewing" and advise that "the best possible approach currently available to making animation from a single image therefore depends largely on the skill, sense, and eye of the animators." [2]

Guiding the user to the viewpoint

Since showcasing complementary artefacts on billboards only adds value to the user experience if being observed from a particular position, the design decision was made to only display these artefacts when the user is within a given perimeter of the perfect location. Markers were added to the VR experience as a guiding tool to offer an indication to the user to look out for additional content in that area [see figure 1].

Many experiments were conducted regarding the functionality, visual design and interaction features of these markers. While particle systems, a technique frequently used in VR game physics using a large number of tiny graphic objects to simulate a certain kind of diffuse phenomena that was already introduced in the initial Biennale 4D Project, draw the users attention to a given spot, this effect was not suitable with the new rendering aesthetics. The pursuit of developing an appropriate visual expression for these markers followed an iterative process that lead to an extensive exploration of the solution space. This included experiments with a variety of subtile gradients and bands with stepwise increasing colour intensity around the epicentre, single and multiple stroke outlines in various colours and styles, both with and without supplementary text hints, experiments with flat textures as well as three-dimensional objects such as small cubes or cylinders. The latter experiment of modeling objects with a smooth circular shape turned out to be a rather challenging endeavour with the Unity game engine as well as with other tools such as Blender for example, because with the objective to optimise performance, by default 3D objects are reduced to the minimal number of polygon faces required. For an impeccable circle representation thought, the highest possible number of segments would be needed. Unity’s LineRenderer feature [4], programmatically added to the marker object at run-time, eventually turned out to be a great workaround to draw clean circular outlines around the markers [see figure 10].

Enhancing the viewpoint marker

When reconstructing the original viewpoints it became apparent that for many shots the camera lens must have been positioned above the eye level of the user. For this reason experiments were conducted with physical marker objects providing an elevated platform for the user to reach the exact hight level that would allow to observe the picture from the correct perspective [see figure 12]. In addition, step elements and ramps were introduced offering affordances that these platforms are walkable and inviting the user to climb up to these vantage points [see figure 11]. Although the result was a perfect viewpoint, that approach was rejected because these additional elements appeared as foreign matters within the virtual environment. This lead to the idea of implementing a lifting feature within the marker: in the moment the user steps onto the surface, he is steadily elevated until he reaches the proper level [see figure 6]. While the upward movement worked surprisingly well in regards to user experience, even though it technically violates the principle that the user should always have full control over his movements to prevent motion sickness, the lowering down movement caused discomfort to the test subjects. This lead to the middle-ground solution, where the user is slowly elevated as soon as he enters the specified region, yet is immediately lowered to the original hight level once he leaves the given area. Additionally, the marker changes its colour when the user enters the triggered space in order to provide a visual feedback to the user regarding his position [see figure 3]. This is particularly important to compensate the lack of full-body avatars.

The experiments conducted have shown that there are many factors that need to be considered in order to integrate archival artefacts in an immersive fashion within a virtual environment. The screenshots below offer an impression of the final outcome of the experiments.


Screenshots of Scenario 1
Experiment doubleline marker

Figure 1: Experimental setting with multiple markers that indicate to the visitor where supplementary content can be observed.

Experiment billboard

Figure 2: Experiment with plain billboard. A historic photograph is placed in the original position of the virtual exhibition space, however perspective is not perfectly fitting because the observation point is to low.

Experiment marker highlighted with billboard

Figure 3: The marker changes its colour to provide the user with a visual feedback that his invisible avatar has entered the interaction radius of this viewpoint.

Screenshots of Scenario 2
Experiment frames simple markers

Figure 4: Experimental setting with frames as placeholder for the space where archival artefacts will be revealed upon user interaction.

Experiment frame photo

Figure 5: The placeholder frames in this scenario are matching the grid aesthetics of the virtual environment.

Experiment simple marker billboard

Figure 6: By the means of the lifting feature of the marker, the artefact blends perfectly into the scene, among other things vanishing lines are matching with the perspective lines of the grid.

Screenshots of Scenario 3
Experiment frame%26paintings

Figure 7: Experimental setting combining artworks with supplementary artefacts. The frame serves as a clue that there is more content to be discovered.

Experiment frame%26paintings5

Figure 8: Experimental setting combining artworks with supplementary artefact embedded into the virtual environment.

Experiment billboard%26paintings

Figure 9: Variation of the experiment shown in figure 8, though without surrounding frame aorund the artefact.

additional Screenshots

experiment with simple marker

Figure 10: Basic marker indicating that additional content can be discovered in this area. Thanks to the application of a LineRenderer, the marker has a smooth outline.

threedimensional marker

Figure 11: Three-dimensional marker object with step as affordance indicating that this object is walkable.

Experiment frames lift markers

Figure 12: Scene with placeholder frames and three-dimensional markers, yet without steps. This experiment lead to the idea of introducing platforms with a lifting feature as marker points.


[1] Gozali, Faustinus Kevin: "Two-Point Perspective 3D Modeling from a Single Image: A Tour into the Picture Experience".
[2] Horry, Y., Anjyo K.-I., Arai, K. (1997): "Tour into the picture: using a spidery mesh interface to make animation from a single image", Proceedings of the 24th annual conference on Computer graphics and interactive techniques.
[3] Sketch Up: "Matching a Photo to a Model (or a Model to a Photo)". URL: https://help.sketchup.com/en/sketchup/matching-photo-model-or-model-photo
[4] Unity Documentation: "LineRenderer". URL: https://docs.unity3d.com/Manual/class-LineRenderer.html

No comments

Embedding archive photos into an immersive VR experience

Biennale 4D is an immersive Virtual Reality experience that allows users to explore the archives of past art exhibitions. In addition to digital documentation of the individual art works in form of images and metadata, the Biennale archives contain additional artefacts such as expressive historic shots taken during the actual exhibitions. These pictures offer unique insights and share more of the spirit and atmosphere of the past events. The quest is to make these artefacts available to the user in a suitable way within an immersive VR experience.

Context

While some of these artefacts have been published previously on the web and in printed products, Virtual Reality offers an unparalleled opportunity to display these pictures in the original context of the exhibition space. The virtual environment allows the user to see an archive photo from exactly the same point of view from where the photographer originally captured the shot. This additional context added to the photograph offers a richer, more immersive experience to the user and lets him dive into that scene. Figuratively speaking, it breaths new life into these old shots and lets them become alive.

2D vs. 3D

The design challenge posed is contriving an appropriate way to display two-dimensional archive photos in a three-dimensional world without loosing the sense of immersion and presence provided by the application. Some of the common practices to do this include inserting the photo directly into the virtual room, either by projecting it onto on an complemented artificial element like some kind of partition wall or a head-up display or displaying it “free-floating” in the virtual space. The first approach works well as long as the user is standing directly in front of the picture, once he is positioned sidewise, the picture disappears and becomes merely a line with the width of the element. A solution that is often applied to fix this issue is displaying the picture on a “camera-facing” billboard, meaning that the billboard always aligns itself to the viewpoint of the user. However, by employing this technique, the photo is no more longer displayed precisely in the original position. And additionally, the notion of being within the scene is destroyed by the constant wiggle movements of the photo always aligning itself to the user’s viewpoint.

Texture Mapping

An impressive example for a completely different approach is offered by the Russian artist Alexey Zakharov in the photo-based animation project “The Old New World” [1], where he animated pictures from the early 20th century. He does this by reconstructing the scene with animated wireframes and mapping the historic photo onto these structures. Inspired by Zakharov’s work the idea of mapping the archive photos as textures onto the walls of the model evolved. This concept has many parallels with projection mapping, a light projection technique applied inter alia in many urban installation works of the famous Spanish artist Pablo Valbuena [2] such as “augmented sculpture” or “quadratura”, where a pattern of shadows and lights is projected onto a surface which then creates the illusion of depth.

For texture mapping as well as projection mapping the image needs to be transformed to match with the shape of the correlative surface and in order to result into an undistorted view. For texture mapping specifically, a transformation of the perspective view from the original photo into Bird’s eye view perspective textures for the individual surfaces is required. For this purpose the photo has to be divided into separate perspective segments and each patch needs to be normalized before it then can be applied as textures to the according faces. Yet again, the full immersion of this effect only works from one specific view point: the spot that corresponds to the original camera position [see figure 1]. The further the users moves his position away from this particular spot, the more of the distortion is revealed [see figure 2] until the intended illusion effect is completely gone. With this in mind we introduced frames [see figures 3 and 4] in our experiment to indicate to the user the correct view point and by this means optimising the user experience.


Screenshots of texture mapping experiment
Mapping experiment

Figure 1: Texture mapping seen from original viewpoint.

Mapping experiment with distortion

Figure 2: Texture mapping seen from other view point revealing distortions.



frame indicating the viewpoint of Texture Mapping experiment

Figure 3: The picture frame serves as a hint for the user where to position himself to discover view points with the best user experience.

Mapping experiment frame%26mapping

Figure 4: As the user approaches the frame and lines his point of view up with it, the historic shots get revealed within the original context.


[1] Zakharov, Alexey (2016): “The Old New World”. URL: https://vimeo.com/160024074
[2] Valbuena, Pablo. URL: http://www.pablovalbuena.com/

No comments

How to guide the focus of the user to what really matters

VR today is heavily driven by technology and the design of many applications is influenced by the technology’s strong roots and prevalence in the gaming industry. Thanks to advancements in rendering and increase in computation power, photorealism is nowadays technically possible, however a higher degree of realism is not advisable for every application. It is important to critically examine what is the most appropriate rendering aesthetics for a given application context. A more realistic representation style indirectly transmits a notion of certainty, it takes much of the users attention and last but not least in the context of the Biennale 4D project it would present a big contrast to the quality and character of the archival artefacts.

Level of detail and level of hypothesis

The Biennale 4D project deals with the archives of past art exhibitions and as a matter of course incompleteness and uncertainty are in the nature of such a case. It is impossible to reconstruct the pavilion or past exhibitions with a high level of detail without making many assumptions to cover documentation gaps. In literature about 3D reconstruction within the domain of Cultural Heritage there is often made reference to the distinction between the semantic level of detail (LoD) and the level of information (LoI). While the LoD refers to the detail degree of the graphical content of a model, the LoI refers to the density of information contained and the credibility of the data sources. In “Cultural Heritage Markup Language: How to Record and Preserve 3D Assets of Digital Reconstruction” [1], Oliver Hauck and Piotr Kuroczynski offer the simple formula “LoI – LoD = LoH” to compute the hypothesis ratio within a reconstructed object, where the difference between the two values leads to the level of hypothesis (LoH). In consideration of this, a deliberate choice of a lower level of detail for the whole building is more appropriate, because it demonstrates credibility and transparency in regards to the actual level of information and still allows a continuous visual style.

Experimental Fidelity

Robert W. Lindeman and Steffi Beckhaus introduce the term “Experimental Fidelity” in their paper “Crafting Memorable VR Experiences using Experiential Fidelity” [2] which they define as follows: “Experiential Fidelity is an attempt to create a deeper sense of presence by carefully designing the user experience. We suggest to guide the user's frame of mind in a way that their expectations, attitude, and attention are aligned with the actual VR experience, and that the user‘s own imagination is stimulated to complete the experience." The authors encourage VR creators to carefully curate the user experience from end to end. However this shouldn’t be done by increasing the level of detail, but rather by reducing the experience to the essential components, guiding the user’s attention towards these items and giving just the right cues to frame the expectations and let the brain fill in the gaps. For the Biennale 4D project, the focus is on the exploration of archival artefacts. The reconstruction of the pavilion is nothing but a structure to hold these artefacts and its purpose is to add context to the artefacts. For this reason we reduced this structure to the fullest extent [see figure 1].

Rendering Aesthetics

Another challenge lies in defining an appropriate aesthetics for a VR application. The visual style of the virtual environment should match with all components of the application without taking away the user’s attention from the actual content. Dealing with cultural heritage archives in the context of the Biennale 4D project, which combines the three material layers of historical content, documentation of the archival artefacts and the virtual reconstruction, this means creating a continuous visual expression which complements the distinct materiality features of each layer.

For these reasons we decided to make major adjustments to the aesthetics of the Swiss pavilion in the Biennale 4D project. Various experiments lead to the new representation style, where a higher degree of abstraction was applied to the architectural reconstruction of the building. All textures and reflections of the material of the walls have been removed and the colour scheme has been inverted reducing the chromaticity of the pavilion to the highest degree: utmost black. The structure is merely indicated by wireframes [see figure 2 and 3]. Also the environment has been reduced to the absolute minimum through a reduction of the saturation of the environment to grey tones. Through these changes the building has deliberately been visually moved into the background and the archival documents receive more attention.

Swiss pavillion outside grids

Figure 1: New rendering aesthetics applied to the model of the Swiss pavillion in the Biennale 4D VR experience, view from outside.

Malerei inside grids2

Figure 2: View into the "Malerei" hall.

Malerei inside grids2

Figure 3: View from within the "Malerei" hall towards the "Passaggio".

[1] O. Hauck, P. Kuroczynski, "Cultural Heritage Markup Language: How to Record and Preserve 3D Assets of Digital Reconstruction", 2015 Proceedings of the 20th International Conference on Cultural Heritage and New Technologies 2015 (CHNT 20).
[2] R. W. Lindeman, S. Beckhaus, "Crafting memorable VR experiences using experiential fidelity", Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology, pp. 187-190, 2009.

No comments

What makes great UX for VR?

12 things I’ve learned on the journey so far...

Here are the main points from today's presentation @ UXcamp Zurich:

1 - Technology

The basics: accurate tracking, no latency, minimize lagging & delay, handling of cables...

2 - Ergonomics

Make sure the user has a comfortable body position while using the application: e.g. holding the hands high up can be very tiring. Avoiding motion sickness!

3 - Psychology

Design of the virtual environment: no narrow spaces, spooky lighting (unless you want that on purpose), choice of colours... Avoid to trigger common phobias.

4 - Interaction

Choose user interactions carefully, allowing the user to control his movements (eg. teleportation) vs. guided tour. General rule of thumb: more interaction, more comfort, more fun :)

5 - Feedback

Importance of providing appropriate feedback to substitute the lack of haptic feedback, for example through audio cue or light effects.

6 - Text

Avoid extended text blocks! Reading in VR (with the current resolution of VR headsets) is just a pain and doesn’t contributes to a great UX. When text is necessary, pay attention to the positioning (field of view, angles, contrast).

7 - User Interfaces

Design for 3D space, do not simple copy the 2D paradigmas from film/web/software without questioning it.

8 - Reduction

Abstraction: decide what is really needed & let the brain fill in the gaps. Realism: if you can’t do it right, then don’t do it. Avoid the uncanny valley!

9 - Intuitivness

Intuitive, self-evident design of gestures and interactions, giving affordances through means such as light, sound, etc. Offer on the job training to the user, for example by offering a demo scene at the beginning where he/she learns how to interact with the virtual environment.

10 - Flow

Importance of "Onboarding Experience": goal is a fluid, seamless transition from reality to virtuality. Consider aspects such as design of the installation, embedding it into a suitable environment, preparation of the frame of mind of the user. Create and frame expectations.

11 - Storytelling

Don’t use technology just for the sake of technology: the use of VR should bring an unique benefit to the user, if not, let it be. Carefully consider to narrative, flow, user guidance. Consciously apply emotions and surprise for higher user engagement.

12 - Testing

Test early & continuously. Don’t take yourself and your friends as standard, you quickly become to familiar with your own projects. Use probands with no gaming experience unless gamers are your target group.

Your experience

What do you think makes great UX in VR? Would love to hear your thoughts and learn from your experience!

No comments

Impressions from VR World London

Quick facts about the event

  • Four tracks (Vision, Experience, Innovation, Developer)
  • Presentations und panels with 150 speakers from of various industries
  • Expo with 60 exhibitors

SOME HIGHLIGHTS AND TRENDS

We can only see a short distance ahead, but we can see plenty there that needs to be done. – Alan Turing

In 2016 the question about VR changed from “IF” to “WHEN”:

  • 2007: 6.2 Mio iPhones were sold in the first year
  • 2016: 6.3 Mio VR Headsets sold (HTC VIVE, Oculus Rift, Samsung Gear VR… not including Google Cardboard!)

Motion Tracking and Haptics

  • Motion tracking = analysing how we interact with the world
  • Haptics provides a better understanding of that interaction of the environment with us

Creating 3D Models for VR

Capture the world in 3D with your mobile device anywhere - anytime - in real-time with astrivis (ETH SpinOff!). watch https://www.youtube.com/watch?v=ChfwKRfxwkU

Storytelling in VR

VR is the next evolution in Storytelling, it will be as transformative for storytelling as the leap from radio to TV or from PC to Smartphone. – Dominic Collins, JAUNT

The success of VR in the future is not so much tech but content. Technology will transcend, good content is timeless. – Matt Beveridge, Pebble Studios

Immersion is often created by what you don’t see. – Matt Beveridge, Pebble Studios

The only rule for VR is that old rules of storytelling don’t apply. VR is not film without edges, we need alternative paradigms. – Matt Simmonds, Telegraph Media Group

First ever BAFTA awarded for a VR Film

Last month, David Attenborough’s Great Barrier Reef Dive VR produced by Alchemy VR won the first ever BAFTA Award 
for a VR experience in the Digital Creativity category. Watch https://www.youtube.com/watch?v=5mA07ZnSPVg and https://www.youtube.com/watch?v=rvqAoPFpO2s

VR and the arts

Panel "The place of VR in Arts, Culture and Heritage": "Curating" the user experience, draw in theatre experience. Go back to the roots of VR? Is VR a tool or an art expression?

Reality is merely an illusion, albeit a very persistent one. – Albert Einstein

The best way to predict the future is to create it. – Alan Kay

No comments