G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS Journal of Cultural Heritage xxx (2012) xxx–xxx Available online at www.sciencedirect.com Original article Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue Lucia Arbace a , Elisabetta Sonnino b , Marco Callieri c , Matteo Dellepiane c , Matteo Fabbri d , Antonio Iaccarino Idelson e , Roberto Scopigno c,∗ a Soprintendenza per i Beni Storici, Artistici ed Etnoantropologici dell’Aquila, Italy Impresa di restauro e conservazione opere d’arte, Italy c Istituto di Scienza e Tecnologie dell’Informazione (ISTI), Consiglio Nazionale delle Ricerche, V. Moruzzi 1, 56126, Pisa, Italy d TryeCo 2.0 s.r.l, Ferrara, Italy e Equilibrarte s.r.l., Roma, Italy b a r t i c l e i n f o Article history: Received 22 December 2011 Accepted 18 June 2012 Available online xxx Keywords: Restoration Fictile sculpture Fragmented artifact 3D scanning Geometric processing Rapid reproduction Virtual repainting a b s t r a c t This paper describes how some innovative methodologies have been designed and employed to support the restoration of the Madonna of Pietranico, a terracotta statue severely damaged in the 2009 earthquake. The statue, fragmented in many pieces, has undergone a complex restoration performed by a multidisciplinary working group. The contribution of digital technologies was planned from the very beginning, since the complexity of this restoration originated the design of innovative procedures for managing the reassembly and restoration process. The Madonna test bed was therefore an example of how technology innovation could be pushed by clear application needs. A first important contribution was the study of the recombination hypothesis of the fragments. This initial phase was performed on digitized 3D models of the statue fragments, with the aim of reducing fragments manipulation, preventing further damages and increasing the capabilities to rehearse and evaluate different reassembly options. The accuracy of the 3D scanned models and the new recombination procedure introduced in this paper allowed to manage this phase in the digital domain with successful results. The digital 3D models were also used to design and produce an innovative supporting structure, constructed with a rapid prototyping device. Another important contribution concerned the study and virtual restoration of the polychrome decoration of the statue; our aim was to reproduce and restore in the virtual 3D domain the very complex original polychrome decoration, on the base of the remaining traces. Consequently, new virtual painting functionalities have been designed on the MeshLab platform (an open-source tool for 3D models visualization and manipulation) for reproducing pictorial decorations over digital 3D models and have been assessed on this specific test bed. This allowed us also to investigate the complexity of the virtual repainting process and to identify further technology enhancements. Finally, computer graphics technologies have been also used to produce a video that tells the story of the restoration. © 2012 Elsevier Masson SAS. All rights reserved. 1. Research aims The main research aim of this paper is to show how a number of innovative methodologies have been designed, implemented and assessed to follow the specific needs of a real restoration project, the restoration of a terracotta statue fragmented in several pieces. We aim to demonstrate the level of contribution that innovative computer-based technologies could bring to Cultural Heritage (CH) restorers: in finding proper recombination hypothesis of the fragmented artwork; in physically reassembling the original artwork; in studying hypothesis of the original painted decoration and performing a virtual reconstruction; finally, in documenting the restoration process and presenting it to the public. 2. Introduction ∗ Corresponding author. E-mail addresses: lucia.arbace@beniculturali.it (L. Arbace), bettasonnino@gmail.com (E. Sonnino), m.callieri@isti.cnr.it (M. Callieri), m.dellepiane@isti.cnr.it (M. Dellepiane), info@tryeco.com (M. Fabbri), iaccarino.a@gmail.com (A. Iaccarino Idelson), r.scopigno@isti.cnr.it (R. Scopigno). This paper describes the design and application of innovative solutions and methodologies tailored to support and assist CH restorers, focusing on the issues concerning the reassembly of a specific fragmented artwork. The technical focus of this work is to show the potentialities of the digital technologies and 1296-2074/$ – see front matter © 2012 Elsevier Masson SAS. All rights reserved. http://dx.doi.org/10.1016/j.culher.2012.06.008 Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 2 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx high-quality 3D digital models in the framework of a complex restoration case. While general-purpose tools for 3D scanning and rendering have proved to be extremely useful in supporting interactive visualization and inspection, we do believe most of the true potential of digital techniques resides more in being able to address specific needs of CH practitioners (e.g. restorers, scholars, museum curators); consequently, a major effort should be dedicated to the design of the required digital instruments. These digital tools should combine the experience of professional restorers or scholars with the skill of experts in the processing of digital models. The concrete test bed selected for the design and assessment work presented in this paper was the restoration of an artwork severely damaged by the recent earthquake in Abruzzo (Italy) on 6th April 2009. The Madonna of Pietranico is an important devotional sculpture in terracotta due to an unknown Renaissance artist, originally located in the main church of the Pietranico village. This artwork was fragmented in many pieces, spanning quite different sizes from large, bulky fragments, small down to a multitude of very tiny pieces. The pre-restoration study and analysis revealed that the artwork was subject to previous important restorations, that already altered in a significant manner its shape and appearance. The chief restorer, Elisabetta Sonnino, proposed to tackle the study and restoration intervention on a multidisciplinary approach, with the aim of solving the complex task by means of different knowledge backgrounds and experiences. The initial project was approved by Luciano Marchetti (supervisor for CH issues in the post-earthquake recovery authority) and by Lucia Arbace (CH Superintendent for the Abruzzi region); she also supervised the overall activity. The resulting complex teamwork was coordinated by Elisabetta Sonnino during the entire duration of the different phases of work. Traditional materials and methodologies were at the base of the planned restoration (Section 4), but at the same time the management staff decided wisely that the work should have also been assisted by the latest digital technologies. The analysis of the artwork, the initial consolidation of the fragments and the restoration of the surfaces were based on state of the art restoration practices and were performed by Sonnino and her collaborators. A local sculptor, Marco Appicciafuoco, specialist in terracotta and expert in antique techniques was invited to recreate the parts that had been lost and to modify some missing parts added in previous restorations. Most of these actions were supported by the use of innovative digital technologies, based on 3D scanning, interactive 3D graphics and geometry processing, designed by CNR-ISTI. The three most important actions carried out were as follows: 2.1. Define a reassembly hypothesis Here the goal was to find and assess an ideal reassembly hypothesis by working mostly in the virtual domain. 3D scanning technologies were used to create digital replicas of all fragments (Section 5). We designed a new methodology for managing the reconstruction of reassembly hypothesis, which should keep the restorer inside the loop and make proficient use of his knowledge and experience. Therefore, we have designed: a user-driven approach to perform and validate the fragment matches in the digital domain, that allows easily to gather a digital encoding of the pairs of matching fragments detected by restorers and allows further discovery of new matches; a geometrical validation of the intermediate results; and the capability to rehearsal and discuss the final placement of the fragments. The goal of this process was twofold: to visually present and to geometrically validate a global recombination hypothesis. The proposed virtual reassembly methodology is the focus of Section 6. 2.2. Physical reassembly Another crucial task was the design of the supporting structure to reassemble the fragments in the real physical space, which was accomplished again by the synergic composition of consolidated restoration methodologies with geometric shape design, visualization and rapid prototyping techniques. This activity was coordinated by Antonio Iaccarino Idelson and developed in strict cooperation with the restorers and the technical partners (CNR-ISTI and Tryeco). Instead of using consolidated solutions, we designed a new approach that uses the high-resolution 3D model of the reassembled fragments. Starting from this data it was possible to design an innovative supporting structure, which precisely fills the void space inside the body of the artwork (transformed into a physical object by rapid reproduction) to produce a supporting structure for the real fragments. Physical reassembly is the focus of Section 7. 2.3. Virtual restoration of painted decorations The usual analysis of the terracotta surface, the analysis of original pigments and the stratigraphy of the painted decorations were also paired by an attempt to restore the original appearance over the digital 3D model, using digital painting and visualization technologies. We have implemented and used a 3D surface painting functionality that has been recently added to the MeshLab opensource mesh-processing tool. The Madonna test bed gave us a very complex assessment case, which allowed us to evaluate the current limitation of out tool and to gather new ideas for further extensions. Virtual restoration is the focus of Section 8. At the same time, by using this tool, it was possible to show to the restorers and the curators a possible reconstruction of the original color of the artifact with a reasonable effort. 2.4. Documentation Finally, one side goal of the project was to preserve the knowledge gathered on the artwork, including its history and devotional value, and on its restoration; a professional film director, Michele Bevilacqua, assisted with the filming and editing of a documentary that presents all the story in a pleasant and easy to follow manner (Section 9). 3. State of the art The goal of this section is to present some background information on the previous work done in the domain of digital 3D techniques applied to the restoration of solid artifacts. Computeraided technologies have been used in many restoration projects and we might say that no serious restoration process can nowadays be planned and executed without the help of some computerbased tool. In most previous experiences [1], technology is used: to document the status of the artwork (documentation of current conditions is mostly produced with digital photography or 3D scanning; moreover, many physical investigation technologies are digital and produce results that can be mapped on the digital model); and to produce digital documents (from the usual digital docs to more advanced GIS systems, data bases or web-based presentation solutions). The focus of this paper is the adoption of 3D graphics technologies to help the restorers in a specific task: reassembling and restoring a fragmented artwork. Therefore, we briefly review the previous work in this specific domain. 3.1. 3D digitization technologies Semi-automatic digital acquisition of the shape and color of artifacts can be achieved through a variety of recording Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx instruments and acquisition methods. The acquisition of a 3D digital model is now common practice in many CH applications [2]. These instruments have been developed through intensive research, both academic and industrial. It should be noted that to date there is no single method or device suitable for every possible type of artwork or every possible use of the acquired data. The great variety of shapes, sizes and materials that characterize the CH domain have a significant impact on the choice of the most suitable technology for a given case study. This choice is greatly influenced by the physical characteristics of the object to be scanned, including the size, the complexity of its outer surface, the light-reflecting properties of the surface of the object and even the possible constraints on access/manipulation. Use of digitization technology in the framework of CH applications has been pioneered by a number of projects [3–5]. 3.2. Using digital 3D models in Cultural Heritage restoration Several experiences have reported the use of digital 3D models in the framework of conservation or restoration activities; we cannot give here a comprehensive and detailed overview of all those works. We mention here just a few representative experiences: the execution of digital investigations and the support of digital documentation [1]; the virtual reconstruction of dismantled artwork [6]; the design of a protection structure needed to ensure a safe travel for a delicate bronze artifact [7]; the use of digitized models as support for planning restoration by calculating centre for gravity and weight for heavy objects [1]; the design of sophisticated solutions for gathering an accurate digitization of the reflectance of artwork’s surface [8–10] which produce different types of recording (PTM, BRDF, BTF); the production of copies with digital manufacturing technologies; or, finally, the methodologies that allow to simulate and predict ageing effects over materials [11]. 3.3. Reassembly of fragmented artifacts with the support of digital technologies Another possible role for visual technologies on CH tasks is the digital reunification of disassembled or fragmented artworks. Physical reassembly is a process usually done manually by archeologists or restorers. The adoption of a computer-aided approach is justified by the fragility of the artifact or by the difficulty of rehearsing complex arrangement in 3D that usually cannot be done physically without damage or major efforts. Moreover, the possibility of performing an accurate and easy comparative evaluation of different reconstruction hypothesis is a big advantage of visual 3D technologies. Early methods have been proposed for special cases, such as the reassembly of shreds of ancient pottery, where some hypothesis on regularity and symmetry of shape can simplify the reassembly task [12]. In some cases, the recombination of fragmented frescos was reduced to a two-dimensional problem, solved by working just on the 2D representation and matching the fragments borders [13]. Moving back to solutions coping with the 3D spatial arrangement, the generic process can be solved in a robust manner by taking into account also the non-precise and eroded fractures of archeological remains [14]. Archeological artifacts are often largely incomplete; thus, solutions are needed to recombine fragments in a plausible way even when we miss proper adjacency between fragments [15]. Finally, the issues concerning the efficient acquisition of a large number of shreds and their accurate alignment have been studied in [16]. The joint improvement of 3D scanning and automatic reassembly methods can open new insights in very complex reconstruction problems. 3 4. Objectives of the restoration project The overall goal of the Madonna of Pietranico restoration was to reunify the fragmented artwork. This was done bearing in mind not only all the relevant information available and the technical aspects to be considered for the restoration, but also the piece’s religious and historical value. The latter was explored by analyzing its conservation throughout its history and by research into the area it originates from. The restoration of a piece is always an important opportunity for the recovery of the context to which the work belongs, understanding its historical memory and knowledge links interconnecting the work of art to its context and local community. In this sense, this project represents part of the reconstruction process after the terrible damages caused by the earthquake. In particular, the specific artwork was created for religious purposes. When the aim of conservation is to return a shared work of art to its original community, this “additional value” plays an important role while deciding on the techniques and methodology to be used throughout the restoration. In the case of the reconstruction and restoration of the Madonna of Pietranico, this principle guided various choices in methodology, taking into account what should be returned at the end of the project and how far the restoration should go in terms of reproduction of the lost components. The statue should, therefore, be reassembled from the pieces, with the aim to be recognizable as a devotional object; moreover, the history of its conservation and previous restorations, along with the stories told about it, should also be “reassembled”. For this reason, making also a film documentary about the project was proposed (Section 9) to help disseminating all the knowledge gathered within the restoration and to document the decisions and actions performed. 4.1. Initial conditions of the artwork, nature and effect of the damages It was only possible to recover nineteen larger and five smaller terracotta fragments from the ruins of the Museo Nazionale d’Abruzzo, where the work had been kept. Among these were the two largest pieces, which were part of the seat and the legs, eight smaller fragments of the bust and many splinters of very small dimensions (Fig. 1). Considerable fictile material had been lost, e.g. the left hand of the Madonna. Missing too were the sleeves of the gown around the wrists, the nose, part of the lips, the back and left side of the head, the veil framing the face, an area of the back and right shoulder and an area below the bust. Some of the folds of the cloak, which protruded further, were damaged or missing too. On all the pieces, the remaining polychromy was dirty, incomplete and had been retouched in previous historical restorations. These subsequent applications of painted decoration had even been applied directly onto the terracotta, after it had lost its preparatory layer. The more recent historical restoration repainted the face using tempera of a color that was close to the color of the terracotta itself, which had clearly been seen as a quick and easy solution to conceal the poor state of preservation (Fig. 1). This also concealed the few remaining, precious traces of the original polychrome decoration. The area near the Madonna’s left leg was seriously deteriorated with a total loss of preparation and polychromies; the terracotta was also noticeably friable with evident flaking on contact. The fragments of the bust from the lower part of the robes had been glued together in previous restorations by using a caseinbased adhesive, which had also been applied to damaged areas of the legs; some of these had been incorporated into the plaster used to repair the statue, inserting pieces of iron wire as reinforcement (again, this addition was done in an previous restoration intervention). Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 4 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx Fig. 1. The initial status of the Madonna after the earthquake, from top to bottom: medium size pieces from the bust; the large pieces of the throne (left image) and the legs (right image); the head fragments. 4.2. Study and first intervention The initial phase of intervention involved finding initial hypothesis for the attachment positions for the various fragments. This was a lengthy task largely due to the difficulty involved in repositioning some of the smallest terracotta slivers from the internal faces of the fragments and in handling the many minute flakes of polychromies. During this operation, each piece was protectively consolidated, so as to prevent any risk of further damage during the manipulation of the pieces1 . Later on, some of the smaller adjoining fragments were glued together, leaving aside pieces that would require some kind of support due to their size and weight. This phase is extremely important and should be based strongly on the experience and knowledge 1 One advantage of adopting a virtual reassembly procedure (Section 6) was also to minimize the manipulation of the original fragments, to reduce the risk of producing further damages. Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx 5 Fig. 2. Historical images of the Madonna, documenting the status of the statue at different times (the leftmost is presumably from end of 19th century or early the 20th century). of the restorer. In some cases, ICT literature proposes to resolve this action with completely automatic solutions based on geometric processing algorithms. Even if those solutions are able to produce good results, on our opinion the skill and experience of the restorer should not be abandoned or under-estimated, since it can be instrumental in discovering much more than current technologies (based only on geometry processing kernels) could obtain. We also retrieved and studied a few archival images of the sculpture (Fig. 2b), which made it possible to confirm that the work had already undergone an extensive restoration in 1934. At that time, most of the seat was remade, along with the left part of the legs, the left hand, the nose and the entire lap, where the Holy Child had originally been cradled. Sadly, there is no photographic evidence or memory of the Holy Child, which had already been removed before 1934 as revealed by the old photograph, taken when the work was still in the Pietranico Church (Fig. 2a). Heavy repainting can also be seen in the photograph in Fig. 2a, as well as some repair of the artwork, especially on the base and on the area around the left knee. More details on the initial analysis are described in [17]. 5. Reconstruction of the digital models via 3D scanning 5.1. Producing the geometric models of the fragments 3D digitization was started just after the initial study and consolidation of the fragments. Several 3D digitization technologies are available nowadays; the selection of the specific instrument is influenced by the characteristics of the 3D model required, in terms of accuracy and resolution, which depend on the intended use for the digital model. In the case of the Madonna of Pietranico we considered: the small size of many fragments; the type of material (partially painted terracotta and plaster with good surface reflection properties); and the highresolution and precision required. We used a Konica Minolta Vivid 910 laser scanner, a system based on triangulation. The individual 3D scanning of the all fragments (19 major and five minor) was carried out by a team of two people in 2 working days, under the supervision of the restoration team. The size of the fragments varied greatly, ranging from large fragments of the main base (30 × 60 × 50 cm) to pieces of average size, such as the shoulder (27 × 17 × 5 cm) and head (12 × 22 × 17 cm), down to smaller pieces, such as a portion of the armrest of the seat (7 × 12 × 3 cm). It was not considered useful to scan the very small fragments (those of size less than one cubic centimeter) because of the difficulty in acquiring and processing the relevant data, but also because they would not have had a significant impact on the virtual reassembly. The scanning resolution (i.e., the distance between each pair of consecutive points recorded) was kept at less than half a millimeter, permitting highly-detailed digital models to be produced. The number of raw single scans for each fragment varied, depending on the fragment’s size and geometric complexity; we acquired from a minimum of 15 to a maximum of 60 scans for each fragment. A total of 580 scans were shot and 6.5 gigabytes of raw data were collected. Complete digital models of the fragments were produced from the raw sampled data collected [2]. This processing typically takes longer than the acquisition phase and requires specific software and some user interaction. Post-processing of the sampled data necessarily includes the alignment of individual views, the elimination of redundant areas (the base on which the fragments were placed, for example, which was also partially sampled by the scanner), the reconstruction of a single unified model integrating the individual scans, any cleaning or necessary completion of the resulting model and, finally the optimization of the digital model. The result are digital surfaces consisting of an enormous number of extremely small triangles - in our case, from three to six triangles define each square millimeter of the object surface. The output of this phase is the so-called “master model”, which is highly accurate, but also very complex. Automatic simplification operations can reduce the number of triangles without losing important geometric detail. The final models of the fragments were encoded with a number of triangles ranging from 1 to 5 million, depending on the size and complexity of the objects. All post-processing of the sampled 3D data was performed with the MeshLab2 tool [18]. 2 MeshLab is an open-source tool, available on http://meshlab.sourceforge.net/. Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 6 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx blending approach, the information in each surface point is synthesized from multiple images, exploiting the redundancy between different viewpoints. This means that the photographic dataset contains much more information in terms of pixels usable for standard 2D documentation purposes. Considering also that the photos are geo-referenced to the 3D models, the photographic dataset is by itself a quite detailed source of documentation. 6. Reassembling the artwork with the aid of computer-aided technologies The main and more innovative technological contribution was in the reassembly phase. Restorers had to solve a complex problem: first, solving the puzzle given the many fragments and designing the ideal reassembly pattern; second, recombining all the pieces and remounting the statue. Fig. 3. Two synthetic images produced from the digital 3D model of the head fragment; on the left, the pure geometry model; on right, the geometry after mapping the photographic color. 5.2. Sampling surface color to complete the digital 3D models The reason for pairing the geometry with the color of the fragments was twofold: first, the documentation of the current status of the fragments should include a global sampling of the surface color; second, the restoration staff was also interested to evaluate the possibility to study the residual polychromy traces and to experiment a possible digital restoration (Section 8). Therefore, a detailed series of photographs were also taken for each fragment, in order to produce 3D models with high-resolution color information. We used a Nikon D80, obtaining 10 Mpixel images. Each piece was photographed from different angles, covering its entire surface. A Gretag MacBeth color calibration table was used to obtain a coherent color reconstruction; lights were positioned in order to obtain a good lighting on all sides and a light tent was used to produce an evenly diffuse illumination (the light tent is a “box” of white cloth in which the object to be photographed is placed and which is lit from the outside to give optimal light diffusion and reduce specular highlights). All this made it possible to accurately record surface color with the support of a total of more than 500 digital photographs. Color data were then integrated and mapped over the 3D digital models of the fragments using the CNR two-phase software process (recently also incorporated into MeshLab). In this process, first, the pictures are aligned with the three-dimensional model (2D to 3D registration). This corresponds to estimating the viewpoint and shooting direction of each photograph, including focal length and possible radial distortion of the camera [19]. This process requires user intervention to select the “point-to-point” relationship between a few points on the surface of the model and the corresponding pixels in the relevant photograph. Once all the images have been aligned, then the color data encoded in the photographs is projected and blended onto the surface [20]. The end result of the scanning phase was a set of 3D color-enhanced digital models faithfully representing the selected fragments (Fig. 3). The resolution and number of images acquired to sample each fragment allowed us to produce a 3D color mapping with a resolution in the order of 1.4 pixels per square millimeter for the smaller fragments, and around 0.5 to 1 pixel per square millimeter on the two larger fragments of the base (pixel coverage does change over the surface, according to local surface accessibility). It has to be considered that, if color is mapped on the 3D surface using a weighted 6.1. Definition of a proper reassembly hypothesis via a digital process One of the goals of the project was to complete and validate the virtual assembly of the fragments prior to its physical reconstruction. This would, in the first place, permit the validation of different hypotheses about the fragments configuration and, later, assist in the design and implementation of the required physical support structure. As briefly reviewed in the state of the art section, automatic reassembly algorithms exist in literature. Since our case was not complex as other CH problems (such as for examples fragmented frescos with hundreds of pieces), we designed and experimented in this project a more simple and easy to deploy approach. The approach proposed is also more robust w.r.t. the case of fragments with eroded matching surfaces. The more irregular and incomplete the fracture surface are, the more difficult is to apply a simple surface matching approach (which, in most cases, assumes a quite perfect matching of the common surface); in our case, the extreme irregularities of the fractures made the surface matching quite unreliable, and forced us to find a more controllable approach. A main pre-requisite and policy was also to leave the expert in the loop and not to demand the solution of the puzzle problem just to the computer. In the case of artifact decomposed in a few tens of pieces, usually the restorer can easily find a large number of matching pairs, based on visual analysis and personal experience. We devised a new approach to gather a digital recording of the matching pairs detected by the restorers, using our 3D scanning device. We asked the restorer to hold each of those pairs in adjoining position (Fig. 4) and shot a couple of range maps with our 3D scanner, trying to capture the join from different positions. These range maps represent a 3D “bridge”, which encodes precise data on the relative positions of these two adjoining fragments, i.e. they contain sufficient geometric data to align the two pieces in a common space in a manner compliant with the restorers’ specification. Using the standard 3D-model-to-3D-model alignment feature of MeshLab (the same used to reconstruct the geometry of the fragments from their scans), it was then possible to compute the spatial relationship between the various fragments. Loading the 3D models of two adjacent fragments and aligning them with the “bridging” range scans aforementioned, it is possible to obtain their relative position. The same procedure is applied to the entire set of fragments and of “bridges”: for each possible fragment coupling, the bridge scans ensure their relative position; when every fragment is aligned to its adjoining element by the bridge scans, the fragments are in their correct position. Considering all the fragments and the bridges at the same time prevents accumulation of error (large joins Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx Fig. 4. Two fragments (C3 and C2) are positioned in the correct adjoining position by hand, then a single range map is acquired with the 3D scanner to gather an accurate description of the adjoining assembly. pulling weak ones), and produces a globally valid solution. At the end, the “bridge” scans are removed, and the respective positioning of the fragment is the needed output. In this way, the experience and analytical capacity of the restoration team was the starting point for the subsequent elaboration, making the restorer knowledge a key element of the virtual reassembly process. The manipulation required was considered by restorers’ sufficient safe for the sake of the fragments conservation (requiring just the rehearsal of a few single pair-wise matches). Starting from these base matching pairs, it was possible to bring the various fragments into alignment in the digital space, using the known contact points. During this digital assembly, two new points of attachment between pairs of fragments were found, which the restorers had not previously identified. It was possible to find these two new contact surfaces because the digital representation allowed to assemble and test more fragments then it could be safely done on the physical objects. These new contact surfaces were then confirmed on the real fragments under the supervision of the restoration staff and added to the “bridges” dataset (repeating the global alignment, thus obtaining an improved global positioning result). In this way, a complete reassembly of the statue was tested in the virtual domain and assessed satisfactorily. The discovery of new points of contact confirms the accuracy of the procedure and clearly demonstrates the usefulness of a 3D digital representation during the restoration of the Madonna. For both the intermediate working theories and the final virtual reassembly, it was possible to explore the geometry of the reconstructed 3D model, and to make a validation in an extremely easily and quickly manner. Furthermore, it was possible to take measurements and to perform visual analysis of the proposed assembly directly on the virtual counterpart of the statue. We also tested the proposed reassembly hypothesis with the historical images available, to find evidence of the correctness of the reunion. The original idea was to use photogrammetry (or a more simple camera match) to obtain precise spatial relationship between the different parts of the statue. However, the available historical photo dataset was some medium-low quality scans of old prints, cropped and processed; additionally, the available view points were quite close, making impossible a precise camera estimation and registration. For this reason, we just again used our technology to roughly align those images on the assembled 3D model; in this way, it was possible to visually check if the current reassembled object was coherent enough with the original state of the statue. 7 Fig. 5. Virtual assembly of the various fragments of the statue: a different color is assigned to each fragment. It’s important to note that these operations are usually very difficult to carry out in the real world, since they require the physical manipulation of the fragments and construction of the necessary support structures to hold the pieces in the defined position. Using only digital clones, the restorers were able to analyze the intermediate results and the possible placement of other fragments and to clearly understand if there was obstruction and the overall quality of the proposed reassembly hypothesis (Fig. 5). The fragment reassembly problem described so far was mostly a problem encompassing the pieces forming the bust and the head. The following problem was how to adjoin these two large components (bust and head) with the seat and leg sections. During the course of the present restoration work, we understood that the overall shape of the statue could be improved with regard to the alterations made in the restoration done on 1934. Our hypothesis was that the elimination of part of the extensive plaster-based repairs introduced in the previous restoration could make the assembly of the major pieces much closer to the original in terms of style and, especially, positioning. This was accomplished by studying and comparing the historical images together with the different dispositions of the digital 3D models, working on repositioning the formerly macro-pieces more precisely, according to the true attachment areas of the original terracotta surface. The only original armrest of the seat, for example, had been reattached with a thick layer of mortar and strong glue separating it, therefore modifying the shape of the seat by widening and raising it. The areas that would be in contact at the join were cleaned to set the armrest directly onto the rest of the seat block. It was even possible to use this part as a point of reference to which to match other restored parts and those that had been lost and were to be completely reproduced. These hypothesis were verified through accurate visual and numerical examination of the various possible virtual reconstructions in 3D, checking each separate modification or improvement of the parts repaired in 1934, so as to better arrange the whole artwork (Fig. 6). The work done on the virtual reassembly allowed us to create evidence of the errors introduced in the previous restoration: even if small, those changes had modified in a visible manner the global shape of the statue (e.g. the bust was rotated towards the legs). After a rehearsal of different options and the necessary validation in virtual space, we were able to take decisions concerning the optimal points of contact between fragments and removed some of the plaster added in the previous restoration. Without the use of a digital virtual approach, we would have needed to perform Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 8 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx Fig. 6. Some frames from the video depicting the virtual assembly of the Madonna statue. demolitions before having a clear plan of the final disposition of the fragments! 7. Physical reassembly - which type of supporting structure? 6.2. Modeling the missing pieces The following phase concerned the study of the structural issues in the physical reassembly, i.e. how to produce a support to hold in place the fragments of the bust and the head. It was necessary to fulfill the following constraints: The potential use of digital 3D models is not simply related to visualization and reassembly. Measurements can be taken very easily using many software tools (again, MeshLab is a good example of these instruments). Moreover, we can produce different types of drawings from the digital 3D model, such as for example: orthographic views, plotted at the reproduction scale selected by the user; or cut-through sections. Those digital drawings may supplement or replace the technical drawings made using traditional methods. As a practical exploitation of these technologies, 3D digital models of the fragments were used as a source of measurements for the manual creation by a sculptor of some replacements in terracotta. This was done for the left hand of the statue (based on the morphology of the right hand) and for the left armrest of the seat (again, based on the right one). In both cases, the corresponding models were digitally mirrored (to produce a left hand from a right one) and then orthographic images and cut-through sections were taken from different angles along the three axes (Fig. 7), giving the terracotta artist a clear reference for the shape of the original part in 1:1 scale. The sculptor then made these parts in terracotta, managing to judge from his experience to what degree the terracotta would shrink during firing. • we should consider the peculiar shapes of the fragments and the points where they would be attached, limiting their weightbearing capacity; • the attachment of the very heavy, forward-leaning head to the neck has an excessively small contact area to support it (due to the incompleteness of the remains in the neck area); • it would be necessary to avoid noticeable external signs of any support, of the damaged areas, or of the gaps between the various fragments (to reduce, as much as possible, any visual disturbance); • finally, we had to take into account the requirements introduced by the need to transport the statue (a loan to a USA museum was planned just after the completion of the restoration, as well as other temporary expositions in Italy). It was proposed that the sculpture would be subdivided into three separate blocks, as it had originally been. These would be the seat, the legs and the bust including the head, leaving, Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx 9 Fig. 7. Cross-sections of the right hand fragment, used as reference in the reconstruction of the missing left hand. however, the hands separate thanks to a reversible assembly using magnets. The initial plan at the early stage of the restoration was to design a metallic structure (based on a central vertical bar and a number of branches grasping each fragment) and to install it in the internal cavity of the statue. After having inspected the virtual assembly of the fragments (Fig. 5) and especially the back of the recomposed statue, it was possible to confirm that the space available within the internal cavities of the sculpture was smaller than expected. Moreover, some contact surfaces between the blocks were ample and substantially intact, since they had been caused by cracks that had not undergone any further alteration. These two aspects convinced that the installation of the mechanical supports was not a good solution, because it was not possible to identify a sufficient number of places to attach the fragments and because it seemed that there was not enough space for the joints and their articulation on the central bar. A solid mechanical attachment would have required at least three or four suitable points around the perimeter of each fragment, to support the weight and avoid rotation. Finally, since the artifact would have to travel, then the fragments should have to be fixed with a very rigid assembly to avoid any damage being caused by vibration. All this contributed to the design of a completely different and innovative solution. The Pietranico Madonna gave us a further demonstration that restoration planning should only be done as a work in progress, since the innovative solution adopted came only after observing the virtual reassembly and several discussions among the interdisciplinary team. The empty space within the sculpture, between the adjoining fragments, was precisely revealed by the digital reassembly. This visualization allowed us to gather the idea that the support should be created by exploiting the cavities of the reassembled statue, “printing in solid” the shape of the internal cavity and to use this to provide a rigid support to the fragments. This appeared as a highly promising technique to be tested. For these purpose we had to design and implement a digital methodology for designing the proper 3D models and for reproducing them. The scanning error (1 to 2 tenth of a millimeter) and the one expected from the rapid reproduction are sufficiently small, making it possible to obtain a solid reproduction of the internal space to a tolerance of less than a millimeter which should be sufficiently accurate for supporting the reassembly. 7.1. Digital design of the filling elements The cavity in the back of the torso of the statue (Fig. 8) was modeled in the digital domain by starting from the surfaces of the fragments oriented towards the center of the bust. Those surface sections were isolated, merged and paired with some surface chunks that should help in modeling the external posterior torso surface, now missing in the original statue. We started with those disconnected surface portions, using Poisson reconstruction (one of the reconstruction methods available in MeshLab, characterized by the fact that it produces in output a closed, hole free surface also when the surface sampling is incomplete). The surface mesh produced was then edited by hand, using digital surface editing features of the MeshLab tool (3D sculpting), to improve the design and to remove some protrusions in the shape, to facilitate the assembly of the fragments over the created support element. A similar approach was adopted to create the head support element. Again, we modeled it by suing the digital surface delimiting the internal cavity of the head. The resulting digital model was then cut in two pieces by subtracting a slab in the middle (to make some room for the metallic bar described in the following, Figs. 8 and 9). 7.2. Producing the filling elements with Rapid Prototyping To make the transition from the digital models to their solid physical reproduction, a proper rapid prototyping technology was chosen after an analysis based on required precision, cost, and structural characteristics. We considered two alternative solutions, numerically controlled milling and 3D printing in chalk/clay powder. Using a subtractive approach was not the ideal solution for our purposes, since milling a high-density plastic block of the requested stiffness would have produced a model much heavier in weight, less handy to be retouched in shape by the restorers and also more costly to be produced. Therefore, we chose to use a high-precision additive Rapid Prototyping system (a Zcorp 650 3D printer). This system is analogous to a computer printer, but instead of drawing ink on paper, it sprays glue on powder, with the object created by deposition of successive layers of synthetic gypsum interspersed with a special fixative, making the material sufficiently hard and tough. This technology is the most popular and appreciated in CH applications since the material used has a pleasant aspect and allows direct intervention with all tools Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 10 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx Fig. 8. The supporting structure: the images show the design of the two support elements (head and bust) created as digital shapes and then constructed using rapid prototyping. and methodologies normally used in restoration. The accuracy of the finished model is approximately 0.2 mm. Since the size of the bust element (29.2 × 44.3 × 43.3 cm) was larger than the maximum working size of the specific 3D printer, the element was divided in four parts, recombined after printing (Fig. 9b). Each part was produced as a hollow block having the required exterior shape, with the external shell created with a thickness of 3 mm. This thickness is an optimal value for structural strength and allows subsequent manual processing such as making of holes, sanding or cutting, it also allows for a perfect wetting of the resin material used to consolidate and harden the reproduction. Some windows were opened in the vertical contact walls (designed to facilitate bonding or fastening with bolts and screws), to provide an opening for filling up the interior of the bust element. Fig. 9. The bust support element designed to hold the chest fragments in position, in digital mockup (a) and after rapid reproduction (b); the nucleus during reassembly tests (c). Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx 7.3. Customizing the supporting nucleus and mounting the fragments The first reassembly tests of the fragments over the support elements made it clear that it would be necessary to make some adjustments to the surfaces of the latter, by eliminating any areas of interference in the undercuts with the terracotta fragments. To simplify the removal of small sections and to preserve the structural performance of the support element, each of the four bust sub-elements was filled with epoxy resin lightened with vinylidene chloride, micro-sphere balls which would guarantee structural continuity and lightness. It was therefore possible to modify the support elements as necessary to permit all of the terracotta fragments to be positioned correctly, mechanically removing any undercuts that had not been identified on the virtual model at the design stage. The provisory positioning of the fragments over the internal support structure gave further demonstration that there was not enough space between the fragments to insert an adequate number of pins to hold them in place. Since we wanted to avoid gluing the terracotta directly to the support surface, a solution was suggested by the method we had developed to glue murals to a solid support [21,22]. A very thin, chemically-reversible layer was created which acts as a barrier to prevent any adhesive from penetrating in the constituent material. This reversible layer, made with a very low polarity resin (Paraloid B67), can be removed by dissolving it with a solvent. Mounting the fragments onto the internal structure was therefore achieved using drops of this epoxy resin containing vinyl chloride micro-spheres and a small percentage of micronized silica as a thixotropic. Care was taken to avoid the adhesive entering the junctures between fragments, so as to provide the highest possible number of access points for future application of the solvent that would be necessary to chemically reverse adhesion. It was decided that a mechanical approach would be used to fix the head in position, which would better support the weight and help to position the fragment of the neck correctly while gluing it to the internal structure. Therefore, the two half-pieces of the hollow of the head were mounted and held in slight expansion using two reverse threaded screws. This expansion system guarantees a correct and firm contact with the cavities and their support elements. These pieces were screwed onto a stainless steel plate, connected to the body by a steel bar, which had been designed so as to give the head the correct inclination (Fig. 10). 7.4. Final considerations on the assembly methodology proposed The support system created for this piece was, therefore, the best compromise that could be found to satisfy the needs to provide a solid structure to support the terracotta fragments and to respect the inter-fragment spaces. From an esthetic perspective this has “restored” the image of the statue to its worshippers, while maintaining reversibility. It may be useful to bear in mind some considerations on the various options for the assembly of this piece: • had a purely metallic (and elastic) mechanical system been used, the weight of the fragments would have necessitated the dismantling of the work for every journey to avoid even the slightest movement, which would have been dangerous at the junctures. Another important consideration is connected to its esthetic appearance–dismantling the work for travel would have made it necessary for the final plastering to be done again on site, removing and replacing it every single time the statues would have travelled; • the internal support elements could have been produced, alternatively, using consolidated technologies, i.e. making models of the individual pieces in silicone rubber, then assembling their 11 respective copies in chalk-plaster, verifying the shape against the fragments in terracotta and then producing a mold of the internal space again in rubber and from this a physical model of the internal space in resin. This process would have been much more complicated, less accurate and more time-consuming than the proposed digital counterpart; moreover, no harm on the artwork was introduced (i.e. no silicone plaster). The innovative method used to make the internal support revealed itself to be highly efficient, although we had initially forecasted an easier reassembly of the fragments over the support structure. The very rough surface of the internal void region of the terracotta made the design of the surface of the supporting structure not easy. We understood that a more sophisticated design approach is required, that should take into account not only the shape of the pieces to hold, but also the possible self-intersections which can be created at the physical reassembly time. We think this could be an interesting algorithmic problem to investigate as future research. Any protrusion or shape feature that could enter in collision with the fragment during the physical reassembly should be automatically removed from the support model at design time, rather than be eliminated after fabrication. Moreover, the design could also be more sophisticated, by taking into account the need of supporting each fragment according to the gravity law; this can be obtained by creating ad-hoc small features to support the fragments in rest position (some sort of small pins or frames fitting the inter-fragments gaps). 8. Restoration of the residual polychrome decoration The cleaning of the original surfaces and of the residual polychrome decoration was very complex, mostly because the various layers of over-painting did not overlay each other uniformly. Previous restorations had, in fact, never involved uniform cleaning, but had left residual dirt, older paint and signs of retouching. Moreover, retouching substances had also been applied directly onto the degraded terracotta without its preparatory layer and thus they had penetrated into the rough, porous surface, making their removal significantly more complicated. A detailed description of the procedure followed to clean and restore the surface is presented in [17]. We just note here that the cleaning and restoration was based on the results of preliminary analysis of the surface (based on visual analysis and the acquisition of macro-photographs and some stratigraphic analyses performed on a few samples removed from the artworks). Those analyses gave us an understanding of the impact on the status of the artwork of the previous restoration interventions. This cleaning phase permitted an extensive analysis of the original polychromy and its decorative finishing, even if the small remaining decoration fragments are seriously damaged in terms of their state of conservation. It was also possible to identify traces of metal leaf: silver leaf on which red lacquer had been added for the robes and gold leaf on azurite for the Madonna’s mantle. The original polycromy and the following multiple restoration interventions created in some regions a multitude of overlapping layers (the stratigraphy sampled from the blue mantle reveals the presence of up to 16 layers). However, in summary, at least three separate, historically-distinct moments can be identified when the surface of the Madonna’s mantle had been repainted, clearly always using a blue or sky blue color. Sadly, the scarcity of residual material does not permit the pattern of decoration of these painted layers to be completely understood. It is therefore not possible to establish the pattern of the gold leaf decoration on the mantle, nor it is possible to establish whether or not the silver leaf and red lacquer were spread uniformly. Such elements, however, Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 12 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx Fig. 10. The mechanical system designed to hold the head fragments in position (a); the mechanical system installed at working location (b). certainly document how finely the original work must have been decorated and the chromatic variety it would have displayed. There are many traces that demonstrate the presence of a very sophisticated decoration, but unfortunately those traces are few and only remain in the undercut and most protected areas of the remaining fragment of the seat. They are not enough for allowing us to draw a complete and solid global hypothesis of the original polychromy. One of the initial goals of the project was to pair the analysis and restoration of the painted surface with the virtual restoration of the surface decoration, according to the data available about the original pictorial decoration of the artwork. The purpose of this virtual restoration was to provide both a visual documentation and a visual presentation of the supposed original decoration to the experts and to the general public. To fulfill this purpose we finalized the design and implementation of a painting plugin contributed to MeshLab (Fig. 11). This painting feature allows using a virtual brush (size and shape of the brush are configurable parameters of the plugin) and a virtual palette of tints and textures to overlay the tint over the digital surface of a 3D model. The reproduction of the polychromy over the Madonna’s digital model was an extremely complex task, due to the incomplete information over the statue surface and the complexity of the decoration. During the analysis phase we discovered that a sophisticated painting methodology was used, based on a preparatory background layer and several different tint layers (including even gold- or silver-gilding layers). We found also evidence that the surface was scraped in some regions in a previous restoration, with a loss of surface finish and of the thick preparation layer. Therefore, even if interactive retouching of the color over the digital 3D model is possible and has been experimented, the restorers’ evaluations was that the available knowledge was not sufficient to obtain a final result on the entire artwork that should qualify itself as scientifically solid (due to the lack of data on many surface regions) and sufficiently high-quality in terms of final appearance of the painted surface. Therefore, we produced just some results on the head region, using the digital painting features of MeshLab, which proved to be handy and could be very useful for didactical purposes (Fig. 11). However, those results are just a first approximation of what the Fig. 11. The image on the left show a snapshot of the new MeshLab plugin for painting 3D digital surfaces, while repainting the head section; an image of the final repainted head is on the right. Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx restorers would like to produce. The Madonna experience was very important to clarify some limitations of the current 3D painting tools and to give us insight on more sophisticated functionalities required: • by the deposition of a thick layer of preparation material, artist usually produce a basic surface that is much smoother than the raw terracotta surface (while just the latter is what remains on many regions of the statue in its current conservation conditions); this means that we need a digital tool to simulate the deposition of this thick layer on the digital 3D model, by changing the surface geometry (i.e. to apply a material that has both tint and thickness, possibly in a progressive manner). The same feature could be also useful for virtually restoring paintings on canvas or wood, since also in that case we have a preparatory layer; • polychromy is usually implemented by the superposition of different layers, in such a way that the final appearance is produced by the way the light interfere with these physical layers. If the goal is to produce an accurate virtual reproduction/restoration, than the existence of those layers should be explicitly encoded (possibly, storing also the reflection/transmission properties of each of those materials) and not just supporting the progressive averaging of different RGB values into a single final color assigned to each surface parcel. This means that the painting tool should be extended by giving the possibility to organize the paint action into a number of subsequent layers, each one characterized by a base color and a specific reflectance property (to allow also to simulate the use of gold leaves); • finally, the rendering stage should also take into account the complexity of this layer-based description of the surface reflection properties. This means that we need more sophisticated rendering algorithms, able to simulate how the light is absorbed, transmitted and reflected by the combination of all these layers. This is even more complicated when we think that rendering should be produced hopefully at interactive speed, to allow the artist to have a proper visual feedback while he/she is painting the virtual surface. In summary, the experience done with the Madonna was precious, since it made clear the very high requirements of some specific restorations projects. As it usually happens with CH applications, the high-profile requirements of this specific domain allowed us to understand some limitations of the current technology that can be taken into account in future work. Moreover, another concern of our users is how to encode different hypothesis on the same model (e.g. to present different reconstruction hypothesis) or how to present to the user the degree of reliability of a given reconstruction. Reconstruction affecting the polychrome decoration as well as directly the shape of an artwork can show a different degree of reliability in different areas of the artwork (since residual knowledge is not uniformly distributed); how do we convey this information to the person who is presented the virtual digital model? Finally, CH experts are usually very scared by the evocative power of digital and visual technologies. Since what you see is usually what you trust (and memorize), they are very reluctant to disseminate visual reproductions produced by virtual restoration studies, since those hypothesis can easily become part of the visual memory of ordinary people, transforming a virtual hypothesis into the visual representation of the original artwork. A good image can become easily a mental model for ordinary people, and thus our work may have a huge impact in education (if good or bad will depend on many factors and many different evaluation criteria). 13 9. Using the digital models to document the restoration 3D digital models, generated from a set of measurements of the object, are widely accepted as an ideal medium to record the conservation conditions of an artwork. In the particular case of the Madonna, both the shape and the color of the various fragments underwent significant changes during restoration. The digital models produced in this project are, therefore, a highly-detailed record of the condition of the statue at the beginning of the restoration work. Moreover, the Pietranico Madonna could be an ideal test case for the documentation of a complex restoration action, which could be implemented by attaching metadata to the 3D models and showing the derivation rules from the models of the initial fragments up to the 3D representation of the final restored artwork. The approach proposed in the 3DCOFORM project [23] could be an ideal way to encode all those data and obtain semantically-enriched models. The goal of this specific restoration project was less ambitious in terms of documentation and was specifically committed to the communication to the general public rather than to the experts. The production of a video documentary was therefore considered an ideal solution for granting a wide dissemination of the knowledge gathered and to illustrate the technical procedures used for the restoration. The use of digital technology was planned also in this action from the very beginning. Photo-realistic or interactive visualization techniques permit to produce images and computeranimated video clips, which are very handy to illustrate some specific phases of the process and the decision taken. A few video clips were created using computer animation, in order to complement the video footage taken during the recovery and restoration of the statue. Those sequences show the 3D models of the colored fragments and details can be clearly seen while they are being combined to re-form the complete statue. The potential of digital animation permits various parts to be shown, even while being assembled, without resorting to complex video filming or to the creation of ad-hoc support structures. The documentary was produced as a 20-minutes high-quality video and a shorter duration excerpt (5 minutes duration). The latter video version is available (with English translation) at: http://www.youtube.com/watch?v=2dquqRwkpqk. 10. Conclusions This paper describes the various phases of a complex restoration project and how digitization technologies and 3D graphic modeling were designed and used to contribute to solve specific restoration problems. Our approach was to go beyond pure cloning of the artwork in a digital 3D model and visualizing it. We made a proficient use of the digital 3D replica and of advanced digital modeling technologies, following two main objectives: firstly, to provide active support for the restoration phase and, secondly, to document the operation itself. The production of 3D models of the main fragments of the statue represented an accurate record of their condition, featuring detailed information on their shape and color. Interactive software tools also provided the opportunity to extrapolate data to obtain measurements, elevations and sections. However, the more innovative contribution was in designing new digital solutions to support the restoration team. In particular, we designed a methodology and the instruments to allow the restorers to formulate and assess reassembly hypotheses in the digital space, and to prepare plans for physical reconstruction using the recorded information. This was done with the aims of reducing the operational complexity and avoiding any risk of further deterioration of the artwork, while, at the same time, augmenting the capacity for analysis and insight. The virtual reconstruction of the statue was also Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008 G Model CULHER-2654; No. of Pages 14 14 ARTICLE IN PRESS L. Arbace et al. / Journal of Cultural Heritage xxx (2012) xxx–xxx useful during reproduction of missing parts. An innovative approach for the reassembly of the artwork has been also proposed and assesses, based on the digital design and production of supporting elements by means of geometric processing and rapid prototyping technologies. This permitted a number of practical issues of the physical reassembly of the fragments to be overcome much more easily. Finally, we have focused on the virtual restoration of the polychrome decoration, implementing and evaluating a new painting plugin contributed to MeshLab; virtual decoration restoration resulted a very complex task on the Madonna test bed and allowed us to gain considerable experience on the limitations of the current digital instruments and some suggestions for future work. The restoration of the Pietranico Madonna is, on our opinion, a good example of how a modern restoration project should be planned and implemented. Consolidated solutions and procedures have been paired with the search of new approaches to solve specific problems; therefore, specific restoration requirements were the stimulus for designing new methodologies and to extend existing digital tools, resulting into an extensive use of the digital media (3D scanning, geometric processing, virtual reassembly, visual analysis, rapid reproduction, and finally computer animation used for documentation purposes). The work was very practical, driven by specific application needs and allowed us to produce innovative solutions and to demonstrate their effectiveness on the field. This restoration - union of consolidated restoration practice and new visual computing technologies - can therefore be considered a success story for the new frontier of computer-assisted restoration. Acknowledgements The restoration project was financed by the Italian American Museum (New York, USA). The research on digital tools leading to these results has also received funding from the European Community (FP7/2007-2013) under grant agreement no. 231809 (IST IP “3DCOFORM” project). References [1] M. Callieri, P. Cignoni, F. Ganovelli, G. Impoco, C. Montani, P. Pingi, et al., Visualization and 3D data processing in David’s restoration. IEEE Comp. Soc, IEEE Comput. Graph. Appl. 24 (2) (2004) 16–21. [2] M. Callieri, M. Dellepiane, P. Cignoni, R. Scopigno, Processing sampled 3D data: reconstruction and visualization technologies, in: F. Stanco, S. Battiato, G. Gallo (Eds.), Digital imaging for cultural heritage preservation: analysis, restoration, reconstruction of ancient artworks, Taylor and Francis, London (UK), 2011, pp. 103–132. [3] M. Levoy, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller, L. Pereira, et al., The digital Michelangelo project: 3D scanning of large statues, in: SIGGRAPH 2000, Computer Graphics Proceedings (July 24–28 2000), Addison Wesley, New York, NY, USA, 2000, pp. 131–44. [4] F. Bernardini, I. Martin, J. Mittleman, H. Rushmeier, G. Taubin, Building a digital model of Michelangelo’s Florentine Pietà, IEEE Comput. Graph. Appl. 22 (1) (2002) 59–67. [5] R. Fontana, M. Greco, M. Materazzi, E. Pampaloni, L. Pezzati, C. Rocchini, et al., Three-dimensional modelling of statues: the Minerva of Arezzo, J. Cult. Herit. 3 (4) (2002) 325–331. [6] A. Gruen, F. Remondino, L. Zhang, Image-based automated reconstruction of the great buddha of Bamiyan, Afghanistan, Comput. Vis. Pattern Recognit. Workshop 1 (2003) 13. [7] G. Accardo, et al. Trasporto in sicurezza delle Opere d’Arte, http://iscr. beniculturali.it/index.php?option=com content&task=view&id=125&Itemid =16. [8] G. Earl, K. Martinez, T. Malzbender, Archaeological applications of polynomial texture mapping: analysis, conservation and representation, J. Archaeological Sci. 37 (8) (2010) 2040–2050. [9] P. Hendrik, A. Lensch, J. Kautz, M. Goesele, W. Heidrich, H.P. Seidel, Image-based reconstruction of spatial appearance and geometric detail, ACM Trans. Graph. 22 (2) (2003) 234–257. [10] G. Müller, J. Meseth, M. Sattler, R. Sarlette, R. Klein, Acquisition, synthesis, and rendering of bidirectional texture functions, Comput. Graph. Forum 24 (1) (2005) 83–109. [11] H. Rushmeier, Computer graphics techniques for capturing and rendering the appearance of aging materials, in: J.W. Martin, R.A. Ryntz, J. Chin, R. Dickie (Eds.), Service life prediction of polymeric materials, Springer, New York, 2009, pp. 283–292. [12] M. Kampel, R. Sablatnig, “Virtual reconstruction of broken and unbroken pottery, in: Int Conf. on 3D Digital Imaging and Modeling (3DIM), IEEE Computer Society, Los Alamitos, CA, USA, 2003, pp. 318–326. [13] M. Andaloro, G. Basile, F. Cristoferi, Guide to the recovery, recomposition and restoration of shattered wall paintings: experience gained at the Basilica of St Francis in Assisi, Istituto Centrale per il Restauro, Roma, 2001, pp. 120. [14] Q.X. Huang, S. Flry, N. Gelfand, M. Hofer, H. Pottmann, Reassembling fractured objects by geometric matching, ACM Trans. Graph. 25 (3) (2006) 569–578. [15] A. Adan Olivier. Reconstruction of the Aeneas group through a hybrid human-computer approach, III Congreso Internacional de Arqueología e Informática Gráfica, Patrimonio e Innovación ARQUEOLÓGICA 2.0, Seville, Spain, 2011. [16] C. Toler-Franklin, B. Brown, T. Weyrich, T. Funkhouser, S. Rusinkiewicz, Multi feature matching of fresco fragments, ACM Trans. Graph. 6 (2010) 185–197 (Proc. SIGGRAPH Asia) 29. [17] E. Sonnino, Reconstruction and restoration, in: L. Arbace, E. Sonnino (Eds.), La madonna di Pietranico - Storia, restauro e ricostruzione di un’opera in terracotta (text in Italian and English), Pescara, Edizioni ZIP, Italy, 2011. [18] P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli, G. Ranzuglia. MeshLab: an open-source mesh-processing tool, sixth eurographics Italian Chapter Conference 2008,pp. 129–136. [19] T. Franken, P. Cignoni, M. Dellepiane, F. Ganovelli, C. Montani, R. Scopigno, Minimizing user intervention in registering 2D images to 3D models, The Visual Computer, 21, Springer, Heidelberg, 2005, pp. 619–628, 21 (8–10). [20] M. Callieri, P. Cignoni, M. Corsini, R. Scopigno, Masked photo blending: mapping dense photographic dataset on dense 3D models, Computer & Graphics, 32, Elsevier Science, Amsterdam, 2008, pp. 464–473, 32 (4). [21] E. Huber, A. Iaccarino Idelson, C. Serino, The mechanical assembly of a renaissance terracotta relief - restoring a tin-glazed work of art from conference proceedings holding it all together - ancient & modern approaches to joining, repair and consolidation organized by Janet Ambers, Catherine Higgitt, Lynne Harrison and David Saunders, British Museum, London, 2008. [22] A. Borzomati, A. Iaccarino Idelson. Innovative materials and methods to mount a damaged mural on a new support, Fifth Congress of the Italian Section of the International Institute for Conservation (2007). [23] S. Pena Serna, R. Scopigno, M. Doerr, M. Theodoridou, C. Georgis, F. Ponchio, A. Stork. 3D-centered media linking and semantic enrichment through integrated searching, browsing, viewing and annotating, Proc. of EG VAST: international symposium on virtual reality, archaeology and intelligent cultural heritage (2011) pp. 89–96. Please cite this article in press as: L. Arbace, et al., Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue, Journal of Cultural Heritage (2012), http://dx.doi.org/10.1016/j.culher.2012.06.008