Tangible User Interfaces for Enhancing User Experience of Virtual Reality Cultural Heritage Applications for Utilization in Educational Environment

Cultural Heritage represents an expression of the ways of living developed by a community and passed on from generation to generation. It is commonly divided and referred to as either intangible or tangible cultural heritage [31]. The former includes oral traditions and expressions, performing arts, social practices, rituals and festive events, knowledge and practices concerning nature and the universe, and traditional craftsmanship [61], as well as political and ideological beliefs that influence cultural practices and history, cyber-cultures in the digital world, and emerging cultural practices that could or will become the heritage of the future [4]. Tangible cultural heritage mainly focuses on places, buildings, monuments, objects, clothing, artwork, and other physical things that can be touched. However, this division is no longer taken for granted, and researchers increasingly focus on the ways they are mutually constituted. The Athena Plus [3], a practice network, whose main objectives are to support digitization and make cultural heritage more widely accessible, stimulate collaboration and facilitate tourism initiatives and services, recommends involving volunteers in the digitization process; E-Inclusion, taking into account the usability requirements and user feedback; analyzing user needs and satisfaction; augmenting traditional information and education tools by using innovative technologies to make heritage more engaging; and establishing synergies with all relevant public and private stakeholders at the local, regional, national, and international levels. Having in mind this continual and necessary reframing of heritage, this work centers participatory design as a necessary methodology in the design, evaluation, and refinement of the SHVM, which is planned to become available in the future physical School House Museum in Prishtina, Kosovo.

Museums and virtual museums are also considered learning environments and are commonly used for non-formal education [19, 58], be it through active learning [9] allowing enhanced engagement with the content and the learning process, thus resulting in enhanced comprehension and participation [14], or through situated learning that takes place in specific environments and contexts in which the learning materials are used, and which are meaningful to the learner [41]. The former can be successfully utilized in educational [6, 30] and cultural heritage serious games [1], delivered on standard platforms and through immersive technologies, such as virtual, mixed, and augmented reality [50, 59, 63]. In all these media, active learning is promoted as the player is required to play, engage, and interact with the content. Similarly, situated learning is intrinsic to VEs and therefore has the capacity to educate the user through engagement and direct participation in contextual and situational experiences.

An additional, intersecting, layer to the engagement and immersion that comes with VR is the potential of participatory-action research principles for generating content, curating collections, and designing active learning tools for a diverse group of users (e.g., museum visitors, students, educators, artists, curators). This is especially true for learning and training aimed at promoting the values of active citizenship, intercultural dialogue, and civic participation. A recent study has shown that high levels of presence, engagement, and immersion can be sustained in archeological VR applications [54] when using 360-degree video storytelling. Forte [21], for example, argues for participatory research in cyber-archeology, which creates “a possible ecosystem ... able to host top-down and bottom-up processes of knowledge and communication” and influences “in a decisive way all the subsequent phases of interpretation and communication.” Specifically, museums, whether physical or digital, are sites of learning that allow engaged and combined methods of formal and non-formal learning. Evidence suggests that co-creative, cross-disciplinary, and cross-sectorial educational models [42] ensure plurality and multiplicity in representation and recognition of narratives and histories that might otherwise be omitted in official historiographies and traditional museum exhibitions. Such a research process underpins the outcomes of the VR experience and the learning it generates and is a possible ground for redistribution of knowledge production relevant to underpinning pillars of social justice, which are increasingly central in both education and museums [38, 66].

Virtual Reality is a technology and a medium that is capable of providing rich experiences through audio-visual computer-generated content reproduced and typically delivered through a VR headset. These experiences and the immersion within them can be further enhanced through embodiment, flow, engagement, as well as by inducing place illusion—a sensation of “being there”—and plausibility illusion—a sensation of the presented scenario actually occurring [55]. Most of the interaction with the environment in real life happens using our hands, by means of touching, grasping, moving, deforming the objects around us. When using computers and interactive systems, this typically happens through input devices, such as mouse, keyboard, game controllers, and tracking systems. Similarly, VR allows us to experience and interact with a virtual environment through different interaction techniques either by using VR controllers or through hand gestures. Typically, four interaction modes are considered: locomotion, selection, manipulation, and scaling, which help us navigate the VE, interact with the objects, change their position orientation or shape, and explore the object or environment details, respectively. Using VR controllers, a certain tactile sensation is obtained. However, controller’s shape, texture, temperature, and other properties often do not correspond to those of virtual objects we interact with. Likewise, when interacting with a virtual environment using hand gestures, we might see a virtual proxy of our hands or even the whole body, but we do not get any haptic feedback from the virtual objects we might be interacting with.

Besides interacting with 3D virtual objects and environments, many computer systems require users to make choices and selections. For making selections in a 2D computer system, menus are a commonly used interaction style [44]. While interaction styles and UI design are rather well established for 2D systems, designing these for 3D applications and environments poses some additional challenges, which has resulted in different techniques being explored and taxonomies proposed [17]. Menu selection and interaction for 3D systems can be utilized by directly mapping 2D menus onto 3D space. For example, pop-up and pull down menus could be used [11, 36] by either fixing them to the viewport of the camera like a heads-up display (HUD), or having them float in 3D space, with predefined position and orientation [11, 16]. Alternatively, spin/ring menus could be implemented [22].

A more natural and immersive alternative would be using diegetic interfaces where the UI elements exist as a part of the VE and are visible to both the player and the virtual characters [50]. Such interfaces combined with passive haptic feedback could result in enhanced and even more immersive experiences in various VR applications. This could be achieved by making the user interface tangible. TEI and TUI are not new concepts [18, 28, 35]. They have been used in different domains, such as education, entertainment, edutainment, information visualization, problem-solving, music [34, 51]. Yu et al. [67] have used tangible interfaces that can be held and squeezed in hands while sensing the user’s heartbeat in an immersive breathing training experience. Their system provides heart-rate variability enhanced breathing guidance in haptic modality that can be used at any time and place for relaxation and stress release. Similarly, TANGAEON system explores how TEI can be used to support people in mindfulness practice [64]. The system uses a water-based interface in a form of real water in a canister placed over a tablet and allows users to interact with it, thus gaining control over the thought-dissolving process. Furthermore, TUIs have been used for learning and edutainment [7, 20, 27, 68] by means of computationally enhanced toys with interactive sensors and dynamic behavior as various types of manipulatives that promote new play possibilities and foster learning in children. TUI can also improve the performance of hand-drawn 3D sketching [2] and be used in collaborative scenarios [48]. Insko [33] has shown that using physical counterparts to virtual objects significantly increases presence and improves cognitive mapping of the environment. This concept of paired virtual and physical environment is also known as Substitutional Reality [53]. These proxies can be made as low-fidelity so the user sees the virtual object in the VE and can touch the physical proxy that provides haptic feedback based on its shape and structure. However, one needs to take particular care of minimizing the mismatch between the physical object and its proxy, being it the shape, size, weight, texture, or something else relevant to the user and the task [40]. In his work, Simeone et al. [53] have shown that those proxies with low and even moderate mismatch, i.e., similarity in material and variations in tactile feedback, temperature, and weight, were found as believable and had a positive impact on the experience, whereas pairings with significant mismatch had a negative effect on the illusion.

In 2018, the Municipality of Prishtina initiated a process to construct the Hertica School House Museum. The museum would be located on the site of the Heritica family house, which was used as the “Sami Frashëri” high school between 1992 and 1999. It was also part of the larger schoolhouse system in Kosovo during the time and grew out of particular, and major, political transformations. In 1989, Kosovo’s autonomy as one of the eight constitutive units of the Socialist Federal Republic of Yugoslavia was forcefully and illegally revoked. Milosevic’s emerging regime in Serbia led this political process, ultimately resulting in the breakup of Yugoslavia. In Kosovo, this disintegration came with a forceful, structural, and systematic political removal of rights. It was accompanied by a violent intervention of the state and the creation of an apartheid like system. The state imposed special measures, whereby schooling became segregated, Albanian primary and high-school directors were fired and school curricula revised. Albanian language instruction at the University of Prishtina was also eliminated. In addition, almost all Albanian public sector workers were expelled from work. The measures included media, healthcare, arts and culture, the police, and the judiciary [24]. With the help of police, school premises, also, became inaccessible. Searching for options, teachers, parents, and community leaders decided on establishing a schoolhouse system. Initially, elementary school teachers began holding classes in their homes but soon all levels of education became organized under this system. Around 3,000 different locations throughout Kosovo, including houses, apartments, basements, and garages were transformed into teaching facilities. The Hertica family house was one of them and saw the graduation of nine generations of high-school students [29]. During the decade, this schooling system came to be considered as the pillar of Kosovo’s civic and peaceful resistance movement. It was part of a massive movement rooted in the belief that civil and peaceful resistance offered an alternative to structural and systematic violence.

The first years of the decade were a time of massive mobilization and protest. They also aligned with demands for plurality and democracy occurring elsewhere in the region. As such, they were characterized by massive social mobilization and solidarity and a movement of civil disobedience towards the regime. Remembrance of these events and their collective experiences, however, have become largely subdued by the tremendous impact of the 1989–1999 war in Kosovo. In particular, post-war history and commemorative practices have contributed to creating a rather narrow recollection, often disregarding the multiple social mobilizations and movements of the decade. However, more recently, there has emerged an interest to explore the 1990s for their socio-cultural politics and practices, to critically account for the experiences and histories of this period and more directly engage what is referred to as the “post-memory generation” [26]. In Kosovo, this generation makes up half of the population. It is a generation that has no personal experiences of the war or the schoolhouse system. Consequently, histories and remembrance from, and of, this period have stood in tension with dominant narratives of the war prevailing in post-war Kosovo.

The SHVM enters this complex space of remembrance, as one site of collective and generational memory, with the aim of documenting, engaging, and animating a cultural heritage that can offer a critical contribution. Whereas heritage sites fall into disarray, conservation lags behind, and digitization is in its infancy, institutions and organizations, as well as independent citizen action, have begun vying and intervening for increased preservation and education around such sites. In addition to a 10-year government strategy for cultural heritage, various organizations and initiatives research, collect, disseminate, archive, and engage, through traditional methods and new digital technologies, to make multiple experiences and memories of the past relevant to practices in cultural heritage and for public interests. The SHVM stands at the intersection of these new practices of cultural heritage, reexamining what their role can be amid old and new erasures and denials of agency attesting to resistance against repression and violence. These have come simultaneously from both within and through the significant international interposition in memory politics in Kosovo. Localized initiatives, however, raise important questions about the roles of future museums—as sites of learning for heritage more broadly—allowing new interventions to take place.

This work is based on several years of collaboration in research and development between the University of Prishtina, Bournemouth University, and the University of Leeds [10, 29, 52, 62]. It encompasses materials collected and designed based on participatory, interdisciplinary, and co-creative methods with students at the University of Prishtina and a larger community of researchers, artists, and practitioners working on heritage and memory. Through a playful, yet genuine, exploration of a difficult and complex past, the design of the SHVM brings to the mix engaged research and learning in a collaborative and co-constructed, pluralist and critical imagining of a site of collective memory. It is based on inclusive modes of representation and interpretation and on increasing understanding of museums as open, living, and networked spaces, where learning and research are transformative and where co-creation and inclusion underpin their vision.

The SHVM provides a curated collection presented as three storylines—Kosovo in former Yugoslavia, Civil Resistance, Schoolhouse (Figure 1)—that build mostly an event based narrative, but one that does not have to be chronologically explored. The chronology presented (“timeline” media in Figure 1) offers a broader context to facilitate an understanding of the schoolhouse system, as well as the contours of the political and social dynamics and power structures that enabled and sustained it. One can enter the journey at SHVM at any storyline in time. The chronology and imagery for the first storyline offers a social history perspective and builds a historical narrative that follows the trajectory of modernization centered on education in Kosovo, in former Yugoslavia. The second storyline offers an overview and exploration of 1990s civil resistance, of the practices and extent of social mobilization, and solidarity, that was enacted in opposition to structural and state violence. It builds a visual account that is based on images with descriptions/captions, a video montage based on collected VHS footage and newspaper clippings from both physical and digital archives, as well as a timeline of events. The third storyline, the schoolhouse, is the most profuse. It includes video-interviews with a diverse group of people (educators, organizers, leaders, and pupils of the school house system), personal and public archival materials (media, school documents, letters, the school newspaper, books, etc.). It is a journey to, and through, a very unique experience.

All three storylines rely on extensive research based on a participatory process using both traditional methodologies and visual and digital research [23]. The research project included over 25 student-researchers who together with the PIs designed the research framework to include the collection of primary sources and archival materials, 15 in-depth interviews, 4 focus groups, and multiple exercises. Also, organized in the form of workshops designed around co-production by students, artists, and researchers, these included: on-site sensory ethnography, found sound and production, filming, mapping, and walking exercises, and video-diaries carried out over a period of three years. The work generated was then designed and presented into two online platforms, the School House Museum (https://shtepiteshkolla.net/) and Respace (https://respace.bournemouth.ac.uk/), which organize all the collected materials and provide toolkits for future projects, respectively.

The process was intentionally collaborative and reflexive, based on grounded exercises through which the social, sensory, and material contexts could be identified and re-constructed [47]. Intentionally, the PIs provided significant contribution in the form of auto-ethnographic description and interpretation, having themselves been pupils at the school. The other interviews, and sources, could then be evaluated by the students all together to raise critical questions about memory production, including the dynamics between more historically dominant narratives and those that become silenced. Participation also involved active involvement in a process of inventing, designing, and imagining sites of memory. Ultimately, the research group indicated the importance of sensory and emotional connection that could be provided by presenting the research through a more embodied experience. This then became translated and imagined into a virtual reality experience, where technology becomes a carrier to possibilities of interconnectedness and empathy. Therefore, the SHVM knowingly addresses the need for an interactive design for a future school house museum and process-based research and design for heritage sites. It provides one example of a full circle research process that might be applied to other sites of memory and heritage and engage participatory learning. We gained a number of key insights through this research process that become translated into the SHVM: Technology can become a means of initiating curiosity, visualization of dense historical and archival materials helps learning, intergenerational and personal recountings can be presented and weighed with factual evidences, the curatorial process of the historical and other materials required specialized knowledge and ability to critically reflected on choices. At the same time, the community that was created around the research and SHVM provided for the necessary critical questioning and nuanced understanding of merging together multiple sources and forms of evidence and representation.

In this experiment, the system has been trialed with the general public in Kosovo trying to validate the usability of the system and investigate UX with it while looking at the two conditions, one of which used a tangible user interface (Desk mode) and the other did not (Walk mode).

Given the importance of immersion for such an experience and the rather high score in Experiment 1, we wanted to further explore the aspects of immersion when using the SHVM. In addition, we wanted to further investigate the potential of using the SHVM in an education setting aimed at learning. In this experiment, the system has been trialed with students and educators at the Faculty of Philosophy, University of Prishtina.