Culture and Computer Science – Physical and Virtual Spaces

Until today, a common understanding of technology is still shaped by the industrialisation as a progress that facilitates our everyday life, making it more predictable, interesting and diverse too. This development since the 19th century was accompanied and also partly reinforced by a multitude of cultural imaginaries. Who does not know the novels by Jules Verne, i. e. 20,000 Leagues Under the Sea or From the Earth to the Moon? Books, films, or imaginaries like these made humans dream and inspired generations of scientists and engineers to invent and build technical devices and systems.

Of course, the cultural imaginaries of science fiction also covered the realm of communication and information. Take Frank Baum’s The Master Key, for example. As early as 1900, he dreamed of a small, flat metal box that could automatically record events and play them back at any time. Or, quite more ambivalent, he envisioned glasses that allowed to recognise and characterise a person as good, clever, wise or bad, cruel or foolish. Accordingly, he wrote: “With these gifts you are now equipped to astound the world and awaken mankind to a realisation of the wonders that may be accomplished by natural forces. See that you employ these powers wisely, in the interests of science, and do not forget your promise to exhibit your electrical marvels only to those who are most capable of comprehending them.”

In recent years, however, this technical development has taken jet another twist. With increasing digitalisation, physical and virtual systems are getting more and more inseparable. Ideas for new products and systems are often conceived, implemented and tested in digital and real environments at the same time. This is especially true for extended reality applications in which code and materiality, virtuality and reality form a unit.

Today, and especially with the background of the Covid 19 pandemic, new digital formats such as streaming, video conference systems, but also virtual and mixed reality and apps are being developed, tested and used in order to compensate the closure of educational and cultural institutions. Some examples include: the virtual tours through the Pergamon Museum Berlin, the #konzertZUhaus stream of the Konzerthaus Berlin, the Virtual Visit of the Hermitage in Sankt Petersburg, the discovery of London’s history through Toto’s eyes of the London museum, the digital school programme of the Salzburg Museum, the 720 ° VR tour of the National Palace Museum in Taipei, the SFMOMA Audio App in San Francisco or the Tata Madiba Virtual Exhibition of the Iziko Museum of South Africa. These actual developments bring new challenges and opportunities for scientists, artists and employees of cultural institutions and, of course too, many ideas and solutions that have been presented and discussed at the this year’s Culture and Computer Science conference in Berlin.

Against the background of the continuously shifting boundaries between physical and virtual spaces, holistic concepts are needed. Cultural studies are making an important contribution here, for example at the Cluster of Excellence Matters of Activity at the Humboldt-Universität zu Berlin. Within the research group Filtering, the interaction of code and materiality is being investigated on a molecular, digital, and cultural level. The underlying techniques of creating physical and virtual environments through filtering processes on these different layers seem to be universal and scale invariant. Hence, every environment consists of materiality and environmental knowledge, i. e. it includes physical reality of the environment and virtual reality in the form of corresponding knowledge of the environment. Physical and virtual reality thus form a unit and cannot be viewed separately from one another.

A new materiality in art and culture as well as in computer science is made possible by connecting physical realities with programmed reality. The creation of new works of art and of computer systems becomes a process that often takes place alternately in physical and virtual reality. Software and computer programs are increasingly becoming tools with which material can be processed. Not only products could be created here, but also works of art. Extended reality applications are an excellent example of this.

Extended Reality combines physical and virtual environments, and human-machine-interfaces such as wearables. It unfolds in a highly interactive, parallel and multimodal manner. As a rapidly expanding area of Computer Science, research in extended reality is characterised by a large degree of interdisciplinarity. The entanglement between the physical world and computer-generated data cuts across and expands disciplines such as human-computer interaction, machine-to-machine communication, computer graphics, sensor systems, but also humanities and artistic sciences such as sound, visual, culture and design studies.

This special issue brings into focus some best practice examples, challenges and future trends in the fields of mixed, extended, augmented, and virtual reality, hybrid systems, 3D technology, modelling, visualisation and interaction. All contributions were presented during the Online Conference Culture and Computer Science in September 2021, which was hosted at the Hochschule für Technik und Wirtschaft Berlin (HTW).

The paper “Immersive Inscribed Spaces – Bringing Virtuality to Written Artefacts for Humanities” describes the ongoing efforts for developing interactive research applications using immersive technologies to explore, catalogue, and experience historical handwritten artifacts. While disciplines like i. e. archaeology has always been highly media but 2D-media related, an important question is how to support, enhance and transform humanities by using immersive 3D tools is still open. Hence, the paper is an excellent example of an interdisciplinary collaboration between humanities and computer sciences.

The paper “Reconstructing the past, enhancing the traces from frescos the case of the St. Venanzio Cathedral in Fabriano” demonstrates how the 3D reconstruction of interior architecture and frescos of a cathedral was done. It mainly describes the manual work that was done in the reconstruction. The research are based on a detailed digital survey of the building and on the following extraction of orthographic representation of the painted walls.

In “Desiderata for a Performative Hybrid Immersive Drawing Platform” the requirements of a software platform for the construction immersive environments through handmade drawings are discussed. It presents also best practices examples with the developed software tools. The digital art practices and the practices of anamorphic handmade spherical drawings are impressive and make the contribution unique.

The authors of the paper “Supporting Self-Embodiment for Non-HMD Users in Asymmetric Social VR Games” discuss the feasibility of supporting Self-Embodiment in an asymmetric VR interaction mode. They demonstrate their research results in an social VR game and support self-embodiment for non-HMD users. The users are represented as realistic full-body avatars and are augmented to increase their sense of agency and presence.

Two best practice example and some technical aspects of Volumetric Videos are described in the contribution “Preserving memories of contemporary witnesses using volumetric video”. Volumetric Video is a new technology which allows the creation of dynamic 3D models of moving objects. In contrast to classical avatar animation, the volumetric videos are more realistic and therefore ideal for the transfer of emotions and gestures.

Within “normal exhibitions”, museums can present only small parts of their collections to the public. Hence curators of museums are using more and more digital solutions to make the accessibility easier, more visitor-centered and non-local. In this context, the paper “Image Garden – Curating Collections and Designing Smart Exhibitions with AI-Based Tools” presents an AI-based toolbox for curating image collections and to design digital exhibitions. The toolbox allows the creation of smart collections based on the interests of the visitors.

The paper “Auralisation of Concert Halls for touristic Purposes” presents the auralisation of three music venues, the Konzerthaus Berlin, the Teatro del Maggio in Florence and the Opera House Lviv. Each building have been digitally recreated and auralised, before being used to conduct case studies. In addition the paper gives insights into current digitalisation and auralisation techniques. Very interesting and new is the use of auralisation for touristic purposes using artificial intelligence for an audience preference analysis.

The contribution “The Virtual Theremin – Designing an Interactive Digital Music Instrument for Film Scene Scoring” describes an implementation of a virtual Theremin as a web-based tool. Important is the relation between the Theremin and the media historical background (movies from Sergei Eisenstein). The guiding idea is the very intuitive use of the Theremin enabling also non experts to realise musical background for movies.

In the paper “XR Art and Culture: Successfull Collaborations in Interdiscplinary Development Processes” the authors present different approaches for interdisciplinary collaborations between engineers, art and culture professionals resp. cultural institutions in interdisciplinary XR developments. They discuss best practices examples and projects from different application areas in art and culture The contribution shows also successful interdisciplinary collaborations at the intersection of culture and computer science and show the potential of such projects for the development of new ways and tools of collaboration in extended reality.

Many people have contributed to this special issue. We would like to thank all authors and reviewers who made it possible to publish this special issue of the Journal of Interactive Media – icom. We hope you will enjoy reading this special issue.

Yours,

Johann Habakuk Israel, Christian Kassung, Jürgen Sieck