Teleimmersive Archaeology

GUEST PAPER TELEIMMERSIVE ARCHAEOLOGY Maurizio Forte, Gregorij Kurillo Teleimmersive archaeology is still in embryonic stage of development but this system is the first one of this kind created worldwide and opens very challenging perspectives in archaeology. The project was supported by the University of California, Merced (School of Social Sciences, Humanities and Arts), and the University of California, Berkeley, CITRIS (Center for Information Technology and Society), where we have started the development of a collaborative system for archaeology, based on Teleimmersive Technology. C yberarchaeology represents a new branch of research aimed at the digital simulation and investigation of the past interpreted as “potential past”, whereas the ecological-cybernetic relations organism-environment and their informative-communicative feedback constitute the core (Forte, 2010). Therefore cyber-archaeology studies the digital codes produced by the simulation processes in virtual environments. Because it depends on interrelationships, by its very nature information cannot be neutral with respect to how it is processed and perceived. It follows that the process of knowledge and communication has to be unified and represented by a consistent digital workflow. 3D information is regarded as the core of the knowledge process, because it creates feedback, then cybernetic difference, among the scientist and the ecosystem. It is argued that Virtual Reality (both offline and online) represents a possible ecosystem, which is able to host top-down and bottom-up processes of knowledge and communication. In these terms, the past is generated and coded by “a simulation process”. The University of California, Merced (School of Social Sciences, Humanities and Arts), and the University of California, Berkeley, thanks to a grant from CITRIS (Center for Information Technology and Society), have started the development of a collaborative system for archaeology, based on Teleimmersive Technology. The collaborative framework is built upon Vrui VR Toolkit, developed by Kreylos (Kreylos, 2008) at University of California, Davis, implemented and further developed by our project. The Vrui VR Tookit aims to support fully scalable and portable applications that run on a wide range of virtual reality systems using different display technologies and various input devices for interaction. The applications built with Vrui can thus run on various clients, from laptops to desktop servers, and support different display technologies, such as 2D displays, stereo displays or fully immersive 3D displays (e.g. CAVE). The framework supports several input devices and trackers with the potential to add custom devices without modifying the developed application. The input device abstraction allows users to attach a virtual tool to each device and assign it with different functionality inside the application. 40 This prototype collaborative application for cyberarcheology, built upon an open source virtual reality framework, is aimed at demonstrating real time collaborative interaction with 3D archeological models in connection with video streaming technologies (including light-weight 3D teleimmersion using stereo cameras). The study and analysis of the interpretation process in archaeology will help the virtual community to re-contextualize and reassemble spatial archaeological data sets, from the first draft version (data not yet interpreted) to the final communicative level. The activity of learning will involve a bottom-up approach - the analyses of the archaeological remains and finds - and a top-down approach - the reconstruction of for example architectural features, artefacts, frescos, styles, materials, shapes, and so on. As all the aspects of this project will pertain to 3D, users will be able to take advantage of the emerging 3D display technologies (e.g. 3D TV) to provide them with a fully immersive experience. At the same time users will be able to continue using more established technologies (e.g. laptops and webcams) to achieve the same level of participation in this environment. Figure 1 - Teleimmersive System at Berkeley, Hearst Mining Memorial Building. ArcheomaticA N° 2 giugno 2011 Tecnologie per i Beni Culturali Figure 2 - Scheme of the Teleimmersive System. TELEIMMERSIVE AND COLLABORATIVE SYSTEM The primary goal of our collaborative framework is to facilitate immersive real-time interaction among distributed users. Collaborative application must provide a communication channel to allow users to verbally communicate and interact with the data (figs 1-2). In case of video conferencing systems, the visual communication is established; however majority of the systems cannot adequately capture gestures, eye contact and other forms of non-verbal communication, which have been shown to increase trust, collaboration and productivity (Fry, Smith, 1975)(Doherty, 1997). When using traditional video conferencing techniques, users are disconnected from the data as the latter is usually presented in a separate window on the screen, resulting in a very low level of immersion or presence. The immersion in a three-dimensional environment can increase the spatial awareness with respect to the data and provide a context for collaboration. Traditional immersive virtual reality systems often use avatars, to represent the human user inside the computer generated environments. The drawback of avatars is that their control is usually unnatural unless users are willing to wear cumbersome motion capture technology. In our work we move further from the synthetic avatars and apply stereo reconstruction to capture 3D representation of users in real time (Vasudevan et al., 2011) to facilitate visual experience similar to reality (e.g. face-to-face meetings). This real-time 3D avatar faithfully represents user’s appearance, gestures and actions. By sending the data to all the remote locations, a virtual presence of each user is established in the collaborative virtual environment. Through this virtual embodiment, the user can now gesture to other users, point at different features, or otherwise communicate via his/her body language. In connection with a 3D display and input device tracking, users can observe their collaborator’s real-time 3D avatar interact with the environment while being able to explore the data in the first person perspective. REAL-TIME 3D AVATARS To generate 3D avatar of a user in real time, we employ multi-camera image-based stereo reconstruction. The stereo framework is extensively presented in [Vasudevan11]. The general idea of the algorithm is to perform accurate and efficient stereo computation of the scene with the user by employing fast stereo matching through an adaptive meshing scheme. The algorithm eliminates the background of the scene, creating a 3D textured mesh from each stereo camera view. By combining several calibrated stereo 41 cameras, larger area can be covered, providing even a fullbody 360 degree reconstruction of the user in real time. The achievable frame-rate is about 25 FPS on images with the resolution of 320x240 pixels or about 15 FPS with the image resolution of 640x480 pixels. The novel meshing scheme also provides high compression ratio when transmitting 3D data of the user to remote locations. A minimum setup for generating 3D video using this framework requires at least one stereo camera which can be mounted above the display. Depending on the camera properties and positioning, the camera may only reconstruct parts of the user’s body, for example the face and upper extremities, while still providing adequate feedback to enhance the communication channel between remote users. For example, user is able to see what part of the scene the remote collaborator is pointing at with his/her hand. Since the algorithm does not assume a human model, user can bring real objects into the scene to showcase them to other users. COLLABORATIVE FRAMEWORK The proposed collaborative system for teleimmersive archaeology has been developed upon OpenGL-based open source Vrui VR Toolkit, developed by Kreylos (Kreylos, 2008) at University of California, Davis. The Vrui Tookit provides abstraction of input devices and display technologies, allowing developed applications to scale from laptop computers to large scale immersive 3D display systems, such as lifesize display walls and CAVE systems. The framework also supports large number of input devices for interaction with ability to add new devices without having to change the applications developed with Vrui. The input device abstraction allows users to attach a virtual tool to each device and assign it with different functionality inside the application. The collaborative extension of Vrui allows linking two or more spatially distributed virtual environments. The clients in the network are connected via three different data streams. The collaboration data stream transmits location of input devices and virtual cameras to all the other clients. The conversation data stream provides communication via audio, video or 3D video conferencing. Finally, the application data stream can be customized to update application states between remote clients and the server (e.g. transmitting object location). In our framework we implemented a centralized scene graph to distribute and synchronize the type and location of spatial data. The scene graph consists of a collection of hierarchically organized, inter-connected nodes with parameterized spatial representation. Each node has one parent and it can have many or no children. The scene graph is managed off the central server which sends clients scene graph changes, 3D position of all users, and video and audio data for communication. This server-based model allows for synchronized interaction in the virtual environment. Any changes made to the scene graph are transmitted to the server in real time while the server sends update of the changes to the connected clients. The clients then render the updated scene. The centralized server model can resolve simultaneous access to the same object node where otherwise inconsistencies in the scene across remote clients could emerge. The scene graph at this point supports the following lowlevel nodes: (a) general node implementing the relationships within the scene graph (i.e. parent class incorporating node organization), (b) data nodes representing the drawable geometries (e.g. triangle mesh, points, polygons, lines), (c) transformation node defining the geometric relationship between connected nodes (i.e. transformation matrix), (d) grid node used for representation of environmental surfaces through grids or height maps, and (d) the root node. Data nodes are currently organized into three data types which allow additional functionality through user interfaces and interactive tools: (1) Wavefront 3D object (OBJ), (2) MeshLab layer files (ALN) and (3) shapefiles with database support (SHP & DBF). In the following sections we describe in more details the individual data nodes. 3D OBJECT NODE Current implementation of the framework supports loading of 3D models in OBJ/Wavefront 3D file format with several texture formats; however, it could be extended to other geometry file formats by adding a new file reading functions. The 3D object node is created from a set of vertices defining the triangles (quads and polygons are automatically converted to triangles for efficiency), the vertex normals and optionally the texture coordinates. For each material, the corresponding vertex buffer objects (VBO) is created. VBOs allow vertex array data to be stored in highperformance graphics memory while allowing subsequent modification of the vertices or their properties. Our current implementation allows for rendering of 1 million triangles with the frame rate of 60 FPS (frames per second) on NVidia GeForce GTX 8800. Due to rather large size of 3D models (in the range of 50-100MB), it is more convenient for the models (i.e. geometry files and textures) to be preloaded to each client instead of downloaded from the server on demand. In the future we plan to incorporate links to models with different levels of detail that could be loaded into the environment by streaming the data from the server or a cloud computing center. This would allow for efficient rendering of complex scenes with ability to examine highly detailed models up-close. Our current prototype application allows users to load, delete, scale, and move 3D objects in the virtual space or attaches them to different parent nodes. When objects in the scene are manipulated (e.g. moving an object, changing scale), a request message linked to the action on the node is sent from the client to the server. If the node is not locked by another client, the parameters of the node get updated and the updates are broadcast from the server to all the clients. 3D LAYER NODE 3D layer nodes are used to combine several 3D objects that share geometrically and contextual properties but are used as a single entity in the environment (e.g. 3D scans of stratigraphic layers of excavation). The framework supports Meshlab (meshlab.sourceforge.net) project format which defines object filenames and their relative geometric relationship. The 3D layer node allows for objects in each layer to be grouped, assigned with different material and color properties, set transparency and visibility levels. Using a slider in the properties dialog, one can easily uncover different stratigraphic layers associated with the corresponding units. GIS DATA NODE The geospatial data is integrated into the framework via shapefiles. The shapefile is a geospatial vector data format for geographic information systems software (e.g. QGIS) with associated attribute database. Our framework currently supports three different vector elements, points, lines and polygons which can be rendered as 2D objects in a geospatial plane or as 3D objects, if depth information is stored in any of the object attributes. In the case of the points, the 3D mode will renderer spheres at different depth locations, while for the polygons, the 3D mode will generate polygonal prisms with depth and thickness pa- 42 rameters. Each element consists of geospatial coordinates and is associated with several attributes which may include the stratigraphic unit number, size information, location, depth, material etc. GIS data property dialog allows user to organize the GIS elements by different attributes. For example, a user can mark all the findings of animal bones with a single color to identify their spatial relationship with respect to other findings or even other models in the scene. Numerical attributes can be further clustered using k-means algorithm to group elements with similar properties by their values. For example, a user can group findings based on the area of the shape and quickly identify large and small clusters of the artifacts. Each group of objects can be assigned with different color, transparency level and visibility parameters. User can work on the GIS data locally (although the same dataset will be loaded for all clients) and once the layout is defined, it can be stored for later use or shared remotely with other collaborators to discuss the findings. NAVIGATION AND TOOLS The proposed framework features a collaborative virtual environment that allows geographically distributed users to navigate in the environment and interact with objects and other users. To provide immersive experience, each user interacts with the application in the first person perspective while being able to observe location of other users through their virtual participation. If the user has the 3D capturing system available, their real-time 3D avatar will appear at their current virtual location. As the remote user moves through the space, his/her 3D avatar travels accordingly through the 3D scene as a part of the model space. If the user has only a webcam, 2D video will appear at their location as a billboard (flat) object to allow some level of visual interaction with other users. The users who have no video acquisition system can still connect and interact in the shared environment while their virtual location is represented by a generic 3D object/avatar. Users can interact with the data independently, although two users cannot move the same object at the same time to prevent inconsistencies in scene. At any time, individual users can also switch to the other user’s point of view or select face-toface mode for direct conversation. The framework features various tools for navigation and interaction which can be linked to wide range of input devices. Inside the environment, user can dynamically assign the tools to different buttons of the mouse or other input device. The Vrui VR toolkit itself provides several virtual tools for navigation and interaction with menus, dialogs and objects: • navigation tools: for navigation through 3D space • graphic user interface tools: for interaction with menus and other on-screen objects • measurement tools: for acquiring object geometry (e.g. dimensional and angular measurements) • annotation and pointing tools: for marking and communicating important features to other remote users In addition to already available tools in Vrui, several custom tools were developed to provide interaction with the virtual objects and data: • draggers: for picking up, moving and rotating objects • screen locators: for rendering mode manipulation (e.g. mesh, texture, point cloud) • object selectors: for selecting objects to obtain metadata ArcheomaticA N° 2 giugno 2011 Tecnologie per i Beni Culturali Figure 3 - A Neolithic house of the archaeological site of Catalhuyuk (Turkey) in the Teleimmersive System. All the archaeological layers are linked with the 3D model and visualized in transparency. In this way it is possible to reproduce virtually the entire archaeological excavation. HARDWARE PLATFORM The proposed framework for the teleimmersive 3D collaborative cyber-archaeology is aimed to be used on various platforms to offer different levels of immersion and interaction. The minimum hardware consist of a laptop with a graphics accelerator, mouse input, microphone and speakers, webcam and wired or wireless connection to establish a 2D video stream from the user into the virtual environment. Such a setup is appropriate also for fieldwork where other technologies are not available. The results presented in this paper were obtained on the teleimmersion platform at University of California, Berkeley (Vasudevan et al., 2011) which consists of several stereo clusters, each connected to a quad core server, to perform 360-degree stereo reconstruction. The system is integrated with a tracking system (TrackIR by NaturalPoint) which tracks position and orientation of a Wii Remote (Nintendo) and active shutter glasses for the 3D TV (Panasonic). The Wii Remote is used for interaction and navigation by tracking its position and orientation. As the user moves his/her head, the rendered image corresponds to the user’s location with respect to the 3D display, providing an immersive experience. The 3D visualization provides more intuitive interaction with various tools (e.g. 3D measurements, positioning of objects) and better recognition of the relative geometric relationship between objects and other data. Furthermore we have connected with a similar system at University of California, Merced, to perform remote experiments between the two sites. Figure 4 - 3D Model of a Neolithic house of Catalhuyuk (B77) reconstructed by laser scanner and now accessible in the Teleimmersive collaborative system. 43 Figure 5 - 3D model of multistratified layers and artifacts from an archaeological trench of Catalhuyuk (East Mound). All the data were recorded with optical scanners and they have a micron accuracy. The combination of 3D layers and artifacts is able to suggest new interpretations. COLLABORATIVE ARCHAEOLOGY The development of the system is still in progress, but nevertheless, we have started different applications according to three very important archaeological case studies: the Neolithic site of Catalhuyuk in Turkey (figs. 3-5), two tombs of the Western Han Dynasty in China with colored wall paintings (Xi’an, figs. 8-9) and the Mayan city of Copan, Figure 6 - Interactive embodied actions (Wii) in the teleimmersive system: query and visualization of spatial layers and artifacts in a Neolithic house. Figure 7 - Collaborative interaction with the Mayan temple of Copan with motion tracking. This digital reconstruction is the result Model made by Raul Maqueda. Honduras (temple 22, fig. 7). The principal scope for any project is the collaborative simulation of different actions and hypotheses of 3D models, dbases and libraries in the cyberspace. In Teleimmersive archaeology the interpretation process is the result of embodied participatory activities whereas multiple users/actors construct a new digital hermeneutics of archaeological research from the fieldwork to virtual reality communication. This cyberspace augments the possibilities to interpret, measure, analyze, compare, illuminate, simulate digital models according to different research perspectives while sharing models and data in the same space. In the case of Catalhuyuk, the Teleimmersive system is aimed to recreate virtually all the archaeological process of excavation, layer-by-layer, artifact by artifact (figs. 3-5). All the data are recorded originally by time-of-flight and optical scanners and then spatially linking them with 3D dbases, alphanumeric and GIS data. In short the 3D interaction can query and investigate 3D models and spatial relations that was not possible to analyze before. Therefore the excavation process becomes digitally reversible and in this way we are able to reproduce new different affordances. In particular the system shows in augmented reality 3D connections between stratigraphies and artifacts not visible in situ. In the above mentioned Chinese tombs, both digitally recorded by laser scanners, the teleimmersion is focused on the study and recontextualization of the funeral objects in the ancient spatial architectural space. Here the iconography of frescos can be reinterpreted by collaborative actions and simulations and 3D cybermaps. Figure 8 - Human avatar inside the virtual tomb M27 of the Western Han Dynasty (Xi’an, China). The cybermap (fig.10) represent the 3D iconic geography of the tomb with the relations between the main subjects; for example: social life, symbolic animals, characters, divinities, etc. In the case of the Mayan city of Copan (Maya Arch 3D Project), we are working on the virtual reconstruction of the temple 22 (fig. 7), studying the model at different stages of reconstruction and comparing it with other architectural models and with the existing archaeological remains. These hybrid forms can be seen as a 3D puzzle, a sort of Lego able to generate potential unexplored possibilities of reconstruction. Assembling and disassembling the model is a necessary starting point for interpreting and understanding architectural features, cultural background and 3D spatial connections of all the components of the model. CONCLUSIONS AND PERSPECTIVES Teleimmersive archaeology is still in embryonic stage of development but this system is the first one of this kind created worldwide and opens very challenging perspectives in archaeology. Collaborative minds at work simultaneously in the same immersive cyberspace can generate new interpretations and simulation scenarios never explored before. This process enhances the feedback of the operators which can develop and share data originally segmented in different domains (layers, units, areas, museums, labs, buildings, databases, archives, repositories, etc.). The collaborative system works as a virtual laboratoy where all the activities are performed in real time and involve teams from different geographical locations. Figure 8 - Human avatar inside the virtual tomb M27 of the Western Han Dynasty (Xi’an, China). ACKNOWLEDGEMENTS Figure 9 - Collaborative work of human avatars inside the virtual tomb M27. 44 Teleimmersive archeology project was supported by Center for Information Technology Research in the Interest of Society (CITRIS) at University of California, Berkeley. We also acknowledge financial support from NSF grants 0703787 and 0724681, HP Labs, The European Aeronautic Defence and Space Company (EADS) for the implementation of the teleimmersion software. We thank Ram Vasudevan and Edgar Lobaton for the stereo reconstruction work at University of California, Berkeley. We also thank Tony Bernardin and Oliver Kreylos from University of California, Davis for the implementation of the 3D video rendering. For the implementation of the archaeological case studies: Maya Arch 3D Project, Digital Technologies for Research in Maya Archaeology (supported by NEH), Catalhuyuk Project, Stanford University, University of New Mexico, Xi’an Jiaotong University, Institute of Archaeology of Xi’an. Special thanks are due to: Ian Hodder, Stanford University, Jennifer Von Schwerin, UNM, Heather Richard, UNM, Fabio Remondino, FBK, Raul Maqueda, Madrid. ArcheomaticA N° 2 giugno 2011 Tecnologie per i Beni Culturali 45 REFERENCES ABSTRACT Forte M. (ed.) (2010), Cyberarchaeology, BAR International Series 2177, Oxford. Vasudevan R., Kurillo G., Lobaton E., Bernardin T., Kreylos O., Bajcsy R., Nahrstedt K. (2011), High Quality Visualization for Geographically Distributed 3D Teleimmersive Applications, IEEE Transactions on Multimedia, vol. 13, no. 3, pp. 573 – 584. O. Kreylos (2008), Environment-independent VR development in G. Bebis, et al. (eds.): Advances in Visual Computing, ISVC 2008, Part I, LNCS 5358, 901–912. Fry R., Smith G.F. (1975), The effects of feedback and eye contact on performance of a digit-coding task. J. Soc. Psychol. 96, pp. 145–146. Doherty-Sneddon G., Anderson A., O'Malley C., Langton S., Garrod S., Bruce V., (1997) Face-to-face and video-mediated communication: A comparison of dialogue structure and task performance, Journal of Experimental Psychology: Applied, Vol 3(2), Jun 1997, pp. 105-125. Archeologia teleimmersiva The system is scalable and low cost. Right now we have two campuses already equipped with Teleimmersive technologies, UC Merced and Berkeley. Other institutions can connect by Web interfaces, simply using standard web cams. As future work we are thinking to extend the system also to outdoors contexts, for example in an archaeological excavation. This would combine and integrate labs and operators (for example scholars and students) with the archaeologists on site. In conclusion, it will be possible in the future to analyze the degree at which immersive collaborative work generates more advanced forms of learning and human interactions. Il progetto di Archeologia teleimmersiva è supportato dal Center for Information Technology Research in the Interest of Society (CITRIS) dell’Università di Berkeley. È ancora in uno stato embrionale ma è il primo sistema creato su scala mondiale che apre prospettive di cambiamento all’archeologia. La Ciberarcheologia rappresenta un nuovo campo della ricerca volto alla simulazione e investigazione delle interpretazioni del passato. PAROLE CHIAVE Cybertecnology, Teleimmersive archeology, 3D, Virtual Reality. AUTORE MAURIZIO FORTE MFORTE@UCMERCED.EDU UNIVERSITY OF CALIFORNIA, MERCED GREGORIJ KURILLO UNIVERSITY OF CALIFORNIA, BERKELEY