3VR: Vice Versa Virtual Reality Algorithm to Track and Map User Experience

This digital era governs our lives in all segments including the needs for cultural upliftment and progress in this sphere of life. The protection and preservation of cultural heritage are not excluded from this general trend. New generations are becoming digital and virtual, which should not be ignored by cultural and research institutions. Moreover, all structures of society have to adapt their operations and improve the ways of promotion and presentation to effectively reach their digital consumers.

The increase in the level and comprehensiveness of digital transformation of societies around the world is evident in the last decade [1]. This is the consequence mostly of the improvements in fundamental technologies, hardware/software development, and an increase in the Internet speed. Unfortunately, the COVID-19 pandemic has significantly affected our entire life, including cultural enlightenment and the social life melioration. According to the United Nations Educational, Scientific and Cultural Organization (UNESCO) document [2], the COVID-19 crisis resulted in the closure of \(90\%\) of museums whereas \(10\%\) of museums may never reopen. Opposing this, we are constantly looking for new ways to overcome all negative influences and by using the advantages of technological progress, we are trying to create a new environment in which we will continue our progress [3].

The museum sector responded rapidly in developing virtual experiences for their visitors [4]. Institutions are actively involving digital platforms and many of them opted to try out new digital software, previously untapped platforms, and channels [5, 6]. Chronologically important milestones toward cultural heritage digitization include the launching of Europeana prototype in 2008 [7], the establishment of Digital Agenda for Europe in 2010 [8], and as a part of it, the publishing of “The New Renaissance” by the Comité des Sages in 2011 on bringing Europe’s cultural heritage online [9] and the “Commission Recommendation on the digitization and online accessibility of cultural material and digital preservation” in 2011 [10], the establishment of the new scenarios during the European Year of Cultural Heritage in 2018 [11], “Digital4Culture” strategy defined by the New European Agenda for Culture in 2018 [11], and in the same year, Leeuwarden Declaration on adaptive re-use of the built heritage [12].

The digital trend continues even nowadays and virtual visits to museums are permanently rising since virtual tours proved to be effective, entertaining, and educational [13]. Google Arts & Culture platform [14] is an example of providing easy online access to digital cultural heritage and virtual museums. Museum directors and curators are more aware of the importance of having content online to overcome a dramatic loss of visitors. Thus, the importance of collecting and analyzing data from the user activities is needed. In response to the necessity of such information, many investigations have been performed including User Behavior (UB) tracking both in real and virtual environments.

Some relevant research that encompasses electronic tracking in the physical environment includes the usage of RGB and depth cameras to capture visitors and deep learning algorithms for data extraction [15]; sensors for facial recognition with a convolutional neural network for defining the interests of visitors [16]; WiFi-based head counting framework for defining the type of visitors [17]; Bluetooth proximity detection for the analysis of visitors’ behavior in the Louvre museum [18]; Bluetooth Low-Energy beacons for the UB assessment and the interaction with the artworks in the Ateneo Art Gallery in the Philippines [19]; and Signal sensors for social behavior patterns of visitor pairs [20]. Despite the above-mentioned trends shedding light on the importance of understanding the dynamics that occur in a real environment, the same cannot be said for virtual ones. Indeed, to our knowledge, there are no well-established methodologies that can enable developers and insiders to infer statistics and/or behavioral patterns within the “virtual compartments” of a museum exhibition.

Our research focuses on human behavior tracking solely in the virtual panoramic environment and strives to determine the most important region in the museum observed by an individual. The input of our algorithm is the virtual online interactive platform (Virtual Museum of the Civic Art Gallery of Ascoli Piceno in Italy) [21] with 81 16,386 \(\times\) 8,192 pixels panoramic images and several interactive features including maps, thumbnails, and menus. The tracking model operates as a reverse virtualization process that assigns particular descriptive functions (descriptors) to each interactive feature, such as viewing multimedia content and observing the panoramic environment. The tracking experiment was performed online, and the web virtual museum key study attracted the attention of 171 visitors around the world. The tracking output is then categorized and measured to define common patterns of UBs. Our primary research contribution includes (1) the development of the virtual web museum, and (2) the novel approach for users’ data collection through the “Vice Versa” Virtual Reality (3VR) algorithm. By collecting and analyzing such data, we proved that the correlation between the equirectangular projection of the view position and the value of the Field of View (FOV) parameter is crucial for the salient region definition, i.e., the most perceptually marked region of the observed panoramas. Our secondary research contribution is the tracking of the events that fire functions when specific buttons are activated. The contributions thus face the challenge of understanding the patterns and preferences of users as they interact within the Virtual Reality (VR) environment. This is made possible through the use of the “salient data repository” and a connected dashboard, allowing museum curators and experts to assess the effectiveness of their decisions. They can then enhance the online museum by introducing new features and diverse content to engage users and maintain their interest. In our approach, the digital web museum is seen as lively content that can be updated according to the user suggestions and inclinations.

The article is organized as follows. In Section 2, we give an overview of the relevant work focusing on UB tracking in virtual environments. Section 3 describes the concepts, hypotheses, and challenges in our user study. Within the same section, we describe the case study of the Civic Art Gallery of Ascoli that we exploit for the interactive virtual tour development and the design of the virtual museum web platform. Our algorithm is explained in detail in Section 4 along with the data collection protocol, the description of fundamental geometrical and procedural features used for the calculations in the virtualization process, and definitions of each segment of the reverse virtualization process. We show results and analyze the properties of selected Regions of Interest (ROIs) in Section 5. The conclusion with a brief recap of our research achievements and goals for future work is given in Section 6.

UB tracking is implemented in almost all applications and websites used on smart devices in various fields including education, economy, medicine, retail, gaming, and cultural heritage. Data collection from the users used to be performed with the conventional technique, surveys that usually come in the form of questionnaires. Lorenzo et al. assessed UB and specifically user aesthetics and emotions in the virtual 360° videos using different technologies [22]. To evaluate user satisfaction with x-commerce for fashion retail with the integrated vocal dialogues with Amazon Alexa virtual assistant, two VR-based immersive experiences are developed by the authors in [23] and subjected thoroughly to the questionnaire-based evaluation. Sarraf performed surveys to collect data on the demographic of average museum Website visitors and their perceptions [24]. Peacock suggested a framework for testing the user experience of the museum based on analysis of web log data [25]. Accessibility, functionality, attractiveness, and usefulness of the website were the main questions. Barbieri et al. discuss the feedback on learnability, device, and system efficacy through a performed user study using the trackball and touch-screen system of the virtual museum [26]. The focus was on comparing the effect of these technologies on UB and performance and defining which technology participants prefer and enjoy more than others.

To capture users’ interaction across VR sessions in the Three-Dimensional (3D) space, an image-based encoding model was introduced in [27] whereas quantitative and qualitative measurements were performed to investigate the satisfaction across users that have tested web and mobile applications and those who were engaged in an on-place visit to the museum in [28]. To record interactions within two different scenarios, virtual and real visits, the authors performed several questionnaires and used remote analytic tools and cameras coupled with computer vision algorithms that detect human motions. The authors of [29] have proposed a method for assessing the effectiveness of 360° virtual tours using an analytical tool. Interactions of the users were measured through embedded JavaScript codes and the final results presented the quantity and types of performed actions including clicks on the button, the way users accessed different web links, visits duration, and so forth. One notable technique for analyzing UB in virtual environments is the use of eye-tracking technology when using VR headsets [30]. For example, in [31] Google Analytics is featured along with HTTP server logs and ClickStream logs to gather information on stickiness and site loyalty, social media referrals, and virtual exhibition usage of the Europeana portal. Google Analytics combined with eye-tracking for user interaction observing in the museology field is leveraged by [32].

In line with the latest research achievements, our study follows the previously mentioned directions of research, but with a specific focus on the ROI detection during the panoramic virtual tour rather than the developed platform assessment that has been mostly performed in the related works. Besides the salient regions, this study, for the first time, aims to assess the interactions with digitized High-Definition (HD) paintings and 3D models defined as the most important content of the museum.

Additionally, the tracking algorithm is specifically tailored to capture user interactions within the panoramic interactive environment. This differs from the more typical visual tracking approaches on the web, which rely on tracking parameters through the content displayed by the browser, often leading to outcomes associated with a Two-Dimensional (2D)-rendered image. Therefore, the homogeneous nature of the panorama ensures that, even as the viewing angle is shifted (and thus the image is rendered in the browser), the exact position of the object within the 3D space is consistently known. This uniformity across the panoramic view obviates the need to collect numerous 2D fragments of images and to locate those pinpoints across different images for behavior tracking. When a user navigates within the panorama, the algorithm can maintain tracking without having to save all possible views or recalculate the object position within a single panorama, which would require significantly more effort than monitoring a specific artifact in a homogenous environment such as a panorama.

Unlike some comparable work in literature, we did not want to consider any personal information of the visitors (name, email addresses, date of birth, and so on) because they might not behave naturally with the feeling of being observed (tracked). Moreover, such a way ensures compliance with the General Data Protection Regulation by exposing this information neither throughout the article nor externally.

The motivation driving the development of our algorithm is to exhibit the collections of museum institutions, particularly aiming to highlight specific pieces of art according to the needs of the public. To achieve this, a connective framework is essential to bridge the gap between two fundamentally different entities: museum experts, such as curators and historians, and visitors. The vice versa method in all its aspects facilitates the gathering of valuable information from both groups. In this collaborative approach, experts from the Civic Art Gallery initially select the pieces, which we then digitally render into 3D models. These chosen pieces are semantically enriched by museum professionals through textual descriptions, allowing for a deeper examination by visitors, whereas the remaining pieces serve as visual elements within the panoramas. Conversely, our algorithm has identified ROIs that visitors are drawn to, embodying the reciprocal purpose of our algorithm. This data, captured by tracking visitor interactions, is invaluable for museum institutions. It enables them to tailor the content of the virtual tour based on visitor interactions. Furthermore, this insight allows the institution to craft varied tours, including specialized guided experiences, by analyzing this interaction data.

3VR aims to comprehensively analyze and accurately differentiate obtained data. Although laboratory user study gives explicit data from the trained and monitored subjects, our goal was to perform the experiment on a real-life scenario, hence we designed a comprehensive web page for the virtual museum without technically demanding requirements. An important task for such a large and complex VR museum exhibition of over 81 panoramic views is to involve a sufficient number of visitors. To extract captured interactions in the different rooms of the museum, collecting enough data from the user interaction is essential. To this end, we leveraged a variety of social media platforms to disseminate the web link of the virtual museum, aiming to attract a substantial number of visitors to our user study.

Figure 1 illustrates the design scheme of the developed tracking system and includes common and separate output data for each panorama region and click events collecting, respectively.

Panoramas are subjected to two types of tracking functions: (1) JavaScript click events and (2) angles \(\theta_{x}\) and \(\theta_{y}\) which in equirectangular space correspond to the horizontal and vertical movement, respectively. Their outputs are generated in the local spreadsheet simultaneously upon each user interaction. The common output of these two functions is locally generated data including timestamp of the interaction (date and time), unique user ID, user IP address, user’s web browser, panorama ID, and width and height of the body website element. Additionally, horizontal and vertical pixel values, \(p_{x}\) and \(p_{y}\) , and vertical FOV of the panorama are generated for the panorama region calculation, and click events data named interactive elements for the content and user interface assessment, which we explain in the Section 4.

The accumulated dataset from different visitors allows the investigation of a multitude of questions of interest in cultural heritage and perceptive psychology. We were inspired by the approach of [34] that defines salient regions on the 3D meshes based on several geometrical properties of the selected points by a large number of users. More precisely, it asked several basic psychological questions that we adapted and leveraged to correspond to our scope of study. Although the subjects of our article (panoramic images) and those from [34] (3D models) are completely different, there is a surprisingly strong bond in the visual and psychological aspects, the salience in any visually represented matter. The questions that we opt to answer include “How consistently do people observe regions on the same panorama?” “How are observed regions distributed on the panorama?” and “What features or contents on the panorama are prominent at the observed regions?”

This section introduces the methodology that we have applied to get answers to previously determined questions. The 3VR algorithm with a flow of all operation processes is depicted in Figure 3.

Our experiment included a total of 171 visitors and was conducted from 23 November 2022 at 11:28:04 Central European Time (CET) to 11 December 2022 at 20:11:10 CET, which is approximately 18 days in total. The virtual tour was accessed by visitors from 19 countries, including Austria, Belgium, Bulgaria, Canada, Croatia, France, Germany, Ireland, Italy, Malta, New Zealand, Poland, Portugal, Serbia, Spain, Sweden, Switzerland, United Arab Emirates, and the United States.

To ensure clear and meaningful information when defining ROIs, filters were applied to eliminate from consideration the user IDs, panoramas, and short views that have been unidentified during data processing. Our analysis revealed that the most visited panoramas were located in the courtyard of a museum, where actually the virtual tour starts (Pa \({}_{1}\) , Pa \({}_{2}\) , and Pa \({}_{3}\) ) and that the viewing frequency of remaining panoramas showed a gradual decline which is evident in Figure 5. It represents the values of the panorama visits frequency over the active time in seconds.

To highlight the flow of the curve while also conserving the space on the page, only a subset of the results is shown in the figure, i.e., the first 26 most viewed panoramas out of 81. The reason why the frequency decreases extremely after the third panorama can be interpreted as a loss of interest and/or a decrease in attention. In addition, the remaining panoramas are less accessible, resulting in more scattered distribution of views. Our analysis has confirmed that the key information and exhibits should be highlighted at the beginning of the virtual tour. Therefore, it is important to incorporate an interactive or dynamic element following the first panoramas to maintain visitor engagement. Utilizing such findings, curatorial planning for the museum interface can be enhanced or even personalized according to different groups of visitors. This, paired with traditional methods for measuring user satisfaction [37], can provide a more comprehensive understanding of visitor behavior in virtual environments.

To investigate the consistency of the observed ROI, we extracted regions for each panorama with the obtained data from all users. For each panorama Pa \({}_{\text{F}}\) we sorted its horizontal and vertical pixel values \(p_{x}\) , \(p_{y}\) accordingly which are obtained from Equation (4), FOV by ascending values along the x-axis (panorama width \(\mathrm{W}\) ). To cluster the most densely populated points, we calculated the difference between neighboring elements of Pa_F along the x-axis array dimension and stored them into the vector Pa’_F. The dense groups of points along with the ROIs are manually selected by intuitively choosing the groups of the smallest consecutive values within Pa’_F. Figure 6 presents ROIs extracted from the visitors’ observation of panorama Pa₂₀.

The values of the elements calculated in Pa’_F differ among various panoramas, depending on the observed content, the variety of exhibits, the distribution of the exhibits within the built environment, and the initial parameters including horizontal pixel value ( \(p_{x}\) ), vertical pixel value ( \(p_{y}\) ), and vertical FOV.

As illustrated in Figure 6, ROIs are located in the central area of the panorama which is one of the paintings that was considered essential. The figure indicates that the visitors were indeed interested in the painting indicated by the high concentration of regions and points, as well as the presence of small squares in multiple areas, suggesting a desire for closer inspection. In the process of identifying dense points on the panorama, our primary assumption was that the ROIs are those where movements and zoom are observed, and our goal was to uncover such regions in each Pa’ \({}_{\text{F}}\) . Moreover, we excluded from the further calculation the groups of unchanging \(x\) -axis coordinates because that could indicate that a region is not relevant enough.

It is necessary to state that \(p_{x}\) , \(p_{y}\) , and vertical FOV are positioned to ensure that the initial view is directed toward the exhibits when reached through their icons from the menu. Moreover, when the interactive artwork is reached by its icon in the menu, the focus is always on the selected artwork, which is the case of the painting represented in Figure 6. Additionally, when navigating through the scene hotspots, the values of \(p_{x}\) , \(p_{y}\) , and vertical FOV remain unchanged when transitioning between panoramas, whereas navigating through the map and thumbnail hotspots results in a predefined view. Due to the described variations in initial views, additional analysis is required to enhance the validity of the ROIs obtained on the panoramas such as Pa \({}_{20}\) . The selection of other 21 panoramas that were defined as suitable to showcase the validity of our study is depicted in Figure 11. It indicates a different distribution of ROIs over panoramas and the number of clustered points in several cases.

To assess the most frequently occurring descriptors during the visiting course, we extracted 24 functions that we defined as crucial for our investigation shown previously in Figure 2. To facilitate the examination of data, we clustered functions of the same type. For example, when the visitor selects the panorama Pa \({}_{\text{F}}\) utilizing the scene hotspot or presses the interactive HD painting “Passeggiata Amorosa,” the corresponding function “Scene hotspot—panorama Pa’ \({}_{\text{F}}\) ” or “Panel HD Painting” which is dedicated to the selected painting is fired respectively. As opposed to analyzing the frequency value over a single panorama over time that was discussed previously, we direct our attention toward calculating the number of interactions with the hotspot elements from the scene (D \({}_{9}\) ), map element (D \({}_{1}\) ), and thumbnail menu (D \({}_{17}\) ). Furthermore, interactions with the menus that contain HD paintings and 3D models are grouped and defined as “Icon HD Painting” and “Icon 3D model.” In a similar fashion, the interactions with these exhibits that trigger the appearance of the related panels are restricted to “Panel HD Painting” and “Panel 3D model,” respectively. Buttons for interactions with a 3D model, a closer look at an HD painting, exhibit description, and information about an artist are also grouped as single functions, regardless of the artwork they are associated with. In this context, Figure 7 presents the frequency of interaction with 24 descriptors whereas the exhaustive list with the corresponding JavaScript codes and descriptions is included in Appendix A.1.

It is clear that the interaction with the hotspots placed within the scene (viewport) of the panoramas is significantly more frequent than with other features. Subsequently, the next highest frequency usage is observed for the map element and thumbnail menu used for switching between panoramas. It is worth mentioning that the low frequency of descriptor (D \({}_{21}\) ) that activates the description of a specific HD painting is a result of it already being activated at the same time as when the interactive HD painting is pressed. In the same context, we omitted the descriptor assignment pertaining to the 3D models as it is the only feature displayed in the corresponding panel. The obtained chart implies the incorporation of an extra functionality when interacting with the hotspots, apart from the mere transition between panoramas. This is another strategy for highlighting the crucial content of the museum.

Furthermore, it is easy for museum experts and curators to obtain statistics on visitor interactions based on selected variables from the tabular data. Selected results are presented in Figure 8, which illustrates statistics related to the top 10 most engaged users ranked by their number of interactions, and in Figure 9, which depicts statistics for the top 10 most engaged panoramas among the top 10 users with the highest number of interactions. Lastly, Figure 10 demonstrates the interactions of the 10 most engaged users with the 5 most engaged panoramas ranked by the highest number of interactions. These figures highlight the pragmatic potential of our method for museum institutions.

In this article, we aimed to present a novel analytic method for UB data analysis in the virtual tour. Our algorithm achieved a satisfactory level in tracking and mapping different kinds of data obtained from user interactions. The variations in the shapes of ROIs, their concentration, and the distribution of the grouped regions across all panoramas confirmed our approach used to identify user interest. The method’s efficiency is clearly proven through the numerical and visual representations and can serve as a useful tool for developers and curators in creating a more engaging exhibit presentation.

The contribution of this article is two-fold: (1) it presents an algorithm that extracts the importance of ROIs determining patterns of the visitor virtual movements and interest and (2) it provides statistics of captured browser activity both of which are crucial for assessing overall user interest. In conclusion, our method addresses an important question: Might non-expert users show interest in artworks that the museum did not initially deem as most important, thus not highlighting them? Our approach provides a detailed analysis of the viewing regions, along with interactions with other virtual museum features, offering experts and curators a clearer understanding of how exhibits are viewed by users. Our method goes beyond just identifying the most viewed panorama or artwork; it allows for exploration into the detailed interactions with specific parts of the artwork. For example, it can reveal what specific element of a painting is observed, for how long, and by how many users, answering deeper questions about visitor engagement patterns.

The limitation of our study is the relatively insufficient attendance of the virtual tour, which may have impacted the precision of the calculated ROIs. Given that some descriptors received a small number of interactions, a larger sample size of visitors is needed to further validate the results, especially for a museum of the capacity of the Civic Art Gallery of Ascoli.

Nevertheless, the final output of our algorithm provides a general overview of UB patterns in the virtual museum, as it combines data from the interactions obtained from all users across single panoramas in determining ROIs and descriptors that were frequently used. To gain more in-depth insights, the next step would be to conduct a per-user and per-panorama analysis as well as to introduce questionnaires to obtain more accurate information. In addition, the use of heat maps derived from user mouse movement within the identified ROIs would also provide valuable insights.

Our future work would include expanding the research by extracting additional metrics such as visitor length of stay, sequence of visited panoramas and interactions with descriptors, as well as consideration of the visitor feedback data. This will enhance the understanding of UB in virtual environments. The next step in our research would be to gather a larger dataset to train a recommender system that will provide tailored and personalized virtual museum experiences for visitors.

The authors would like to thank Emanuele Frontoni, Eva S. Malinverni, and Paolo Clini for the scientific boosting and continuous stimulus to the present research, and they would also like to extend gratitude to Goran Stojic for his assistance in implementing the various JavaScript codes used in this study. The research is also framed by the E+ project DC Box Digital Curator Training & Tool box, 2021-1-IT02-KA220-HED-000032253.