research-article

Solving archaeological puzzles

Authors:

Ilan ShimshoniAuthors Info & Claims

Volume 119, Issue C

https://doi.org/10.1016/j.patcog.2021.108065

Published: 01 November 2021 Publication History

Highlights

•

In the paper we propose a novel approach for solving puzzles whose pieces are of general shape, which need not necessarily match precisely due to erosion, and which have degraded unevenly over time.

•

We define the notion of valid transformations and propose a method of sampling them. This method may find uses in other domains.

•

We define a novel dissimilarity score that takes into account the special characteristics of the domain.

•

We introduce the notion of confidence in the dissimilarity scores, which depends not only on the value of the dissimilarity, but also on those of the competing matches. Confidence may be used not only in re-assembly, but also in other applications that are based on dissimilarity.

Abstract

This paper focuses on the re-assembly of an archaeological artifact, given images of its fragments. This problem can be considered as a special challenging case of puzzle solving. The restricted case of re-assembly of a natural image from square pieces has been investigated extensively and was shown to be a difficult problem in its own right. Likewise, the case of matching “clean” 2D polygons/splines based solely on their geometric properties has been studied. But what if these ideal conditions do not hold? This is the problem addressed in the paper. Three unique characteristics of archaeological fragments make puzzle solving extremely difficult: (1) The fragments are of general shape; (2) They are abraded, especially at the boundaries (where the strongest cues for matching should exist); and (3) The domain of valid transformations between the pieces is continuous. The key contribution of this paper is a fully-automatic and general algorithm that addresses puzzle solving in this intriguing domain. We show that our approach manages to correctly reassemble dozens of broken artifacts and frescoes.

References

[1]

W. Marande, G. Burger, Mitochondrial dna as a genomic jigsaw puzzle, Science 318 (5849) (2007).

[2]

A. Deever, A. Gallagher, Semi-automatic assembly of real cross-cut shredded documents, ICIP, 2012, pp. 233–236.

[3]

T. Cho, S. Avidan, W. Freeman, The patch transform, PAMI 32 (8) (2010) 1489–1501.

[4]

B.J. Brown, C. Toler-Franklin, D. Nehab, M. Burns, D. Dobkin, A. Vlachopoulos, C. Doumas, S. Rusinkiewicz, T. Weyrich, A system for high-volume acquisition and matching of fresco fragments: reassembling theran wall paintings, ACM Trans Graph 27 (3) (2008) 1–9.

Digital Library

[5]

D. Koller, M. Levoy, Computer-aided reconstruction and new matches in the forma urbis romae, Bullettino Della Commissione Archeologica Comunale di Roma 15 (2006) 103–125.

[6]

E. Sizikova, T. Funkhouser, Wall painting reconstruction using a genetic algorithm, J. Comput. Cult. Herit. 11 (1) (2017) 3:1–3:17.

[7]

E. Demaine, M. Demaine, Jigsaw puzzles, edge matching,and polyomino packing: connections and complexity, Graphs and Combinatorics 23 (2007) 195–208.

[8]

H. Freeman, L. Garder, Apictorial jigsaw puzzles: the computer solution of a problem in pattern recognition., IEEE Trans. Electronic Computers 13 (2) (1964) 118–127.

[9]

T.S. Cho, S. Avidan, W.T. Freeman, A probabilistic image jigsaw puzzle solver, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 2010, pp. 183–190.

[10]

A.C. Gallagher, Jigsaw puzzles with pieces of unknown orientation, 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2012, pp. 382–389.

[11]

S. Gur, O. Ben-Shahar, From square pieces to brick walls: The next challenge in solving jigsaw puzzles., ICCV, 2017, pp. 4049–4057.

[12]

K. Son, J. Hays, D.B. Cooper, Solving square jigsaw puzzles with loop constraints, European Conference on Computer Vision, Springer, 2014, pp. 32–46.

[13]

G. Paikin, A. Tal, Solving multiple square jigsaw puzzles with missing pieces., CVPR, 2015, pp. 4832–4839.

[14]

D. Pomeranz, M. Shemesh, O. Ben-Shahar, A fully automated greedy square jigsaw puzzle solver, CVPR 2011, IEEE, 2011, pp. 9–16.

[15]

D. Sholomon, O. David, N.S. Netanyahu, A genetic algorithm-based solver for very large jigsaw puzzles, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1767–1774.

[16]

E. Tsamoura, I. Pitas, Automatic color based reassembly of fragmented images and paintings., IEEE Trans. Image Processing 19 (3) (2010) 680–690.

[17]

X. Yang, N. Adluru, L.J. Latecki, Particle filter with state permutations for solving image jigsaw puzzles, CVPR 2011, IEEE, 2011, pp. 2873–2880.

[18]

D. Goldberg, C. Malon, M. Bern, A global approach to automatic solution of jigsaw puzzles, Proceedings of the Eighteenth Annual Symposium on Computational Geometry, 2002, pp. 82–87.

[19]

W. Kong, B.B. Kimia, On solving 2D and 3D puzzles using curve matching, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, IEEE, 2001, pp. 583–590.

[20]

K. Son, D. Moreno, J. Hays, D.B. Cooper, Solving small-piece jigsaw puzzles by growing consensus., CVPR, 2016, pp. 1193–1201.

[21]

D. Goldberg, C. Malon, M.W. Bern, A global approach to automatic solution of jigsaw puzzles., Comput. Geom. 28 (2–3) (2004) 165–174.

[22]

R. Lotus, J. Varghese, S. Saudia, An approach to automatic reconstruction of apictorial hand torn paper document., Int. Arab J. Inf. Technol. 13 (4) (2016) 457–461.

[23]

K. Lalitha, S. Das, A. Menon, K. Varghese, Graph-based clustering for apictorial jigsaw puzzles of hand shredded content-less pages., IHCI, Lecture Notes in Computer Science, volume 10127, Springer, 2016, pp. 135–147.

[24]

L. Zhu, Z. Zhou, D. Hu, Globally consistent reconstruction of ripped-up documents., IEEE Trans. Pattern Anal. Mach. Intell. 30 (1) (2008) 1–13.

[25]

M.G. Chung, M.M. Fleck, D.A. Forsyth, Jigsaw puzzle solver using shape and color, ICSP’98. 1998 Fourth International Conference on Signal Processing (Cat. No. 98TH8344), volume 2, IEEE, 1998, pp. 877–880.

[26]

D.A. Kosiba, P.M. Devaux, S. Balasubramanian, T.L. Gandhi, K. Kasturi, An automatic jigsaw puzzle solver, Proceedings of 12th International Conference on Pattern Recognition, volume 1, IEEE, 1994, pp. 616–618.

[27]

M. Makridis, N. Papamarkos, A new technique for solving a jigsaw puzzle, 2006 International Conference on Image Processing, IEEE, 2006, pp. 2001–2004.

[28]

T.R. Nielsen, P. Drewsen, K. Hansen, Solving jigsaw puzzles using image features, Pattern Recognit Lett 29 (14) (2008) 1924–1933.

[29]

M.S. Sagiroglu, A. Erçil, A texture based matching approach for automated assembly of puzzles, 18th International Conference on Pattern Recognition (ICPR’06), volume 3, IEEE, 2006, pp. 1036–1041.

[30]

F.-H. Yao, G.-F. Shao, A shape and image merging technique to solve jigsaw puzzles, Pattern Recognit Lett 24 (12) (2003) 1819–1835.

[31]

K. Zhang, X. Li, A graph-based optimization algorithm for fragmented image reassembly., Graph Models 76 (5) (2014) 484–495.

[32]

J. Zhang, K. Kang, D. Liu, Y. Yuan, E. Yanli, Vis4heritage: visual analytics approach on grotto wall painting degradations, IEEE Trans. Vis. Comput. Graph. 19 (12) (2013) 1982–1991.

[33]

S. Houde, S. Bonde, D.H. Laidlaw, An evaluation of three methods for visualizing uncertainty in architecture and archaeology, 2015 IEEE Scientific Visualization Conference, SciVis 2015, Chicago, IL, USA, October 25–30, 2015, 2015, pp. 149–150.

[34]

H. Rushmeier, Eternal egypt: Experiences and research directions, International Workshop on Recording, Modeling and Visualization of Cultural Heritage, Taylor & Francis, Taylor & Francis, 2006, pp. 22–27.

[35]

A.R. Willis, D.B. Cooper, Computational reconstruction of ancient artifacts, IEEE Signal Process Mag 25 (4) (2008) 65–83.

[36]

T.A. Funkhouser, H. Shin, C. Toler-Franklin, A.G. Castañeda, B.J. Brown, D.P. Dobkin, S. Rusinkiewicz, T. Weyrich, Learning how to match fresco fragments, JOCCH 4 (2) (2011) 1–13.

[37]

C. Toler-Franklin, B. Brown, T. Weyrich, T. Funkhouser, S. Rusinkiewicz, Multi-feature matching of fresco fragments, ACM Transactions on Graphics (TOG) 29 (6) (2010) 1–12.

[38]

H. Wolfson, E. Schonberg, A. Kalvin, Y. Lamdan, Solving jigsaw puzzles by computer, Ann Oper Res 12 (1) (1988) 51–64.

[39]

F. Kleber, R. Sablatnig, A Survey of Techniques for Document and Archaeology Artefact Reconstruction, ICDAR, IEEE Computer Society, 2009, pp. 1061–1065.

[40]

H. Leitao, J. Stolfi, A multiscale method for the reassembly of fragmented objects, Proceedings of the British Machine Vision Conference, 2000, pp. 71.1–71.10.

[41]

G. Papaioannou, T. Schreck, A. Andreadis, P. Mavridis, R. Gregor, I. Sipiran, K. Vardis, From reassembly to object completion: a complete systems pipeline, Journal on Computing and Cultural Heritage (JOCCH) 10 (2) (2017) 1–22.

[42]

G. Papaioannou, E.-A. Karabassi, T. Theoharis, Reconstruction of three-dimensional objects through matching of their parts, IEEE Trans Pattern Anal Mach Intell 24 (1) (2002) 114–124.

[43]

Q.-X. Huang, S. Flöry, N. Gelfand, M. Hofer, H. Pottmann, Reassembling fractured objects by geometric matching, ACM Trans. Graph. 25 (3) (2006) 569–578.

[44]

E. Vendrell-Vidal, C. Sánchez-Belenguer, A discrete approach for pairwise matching of archaeological fragments, J. Comput. Cult. Herit. 7 (3) (2014) 15:1–15:19.

[45]

C. Ostertag, M. Beurton-Aimar, Matching ostraca fragments using a siamese neural network, Pattern Recognit Lett 131 (2020) 336–340.

[46]

A.G. Castañeda, B. Brown, S. Rusinkiewicz, T. Funkhouser, T. Weyrich, Global consistency in the automatic assembly of fragmented artefacts, Proceedings of the 12th International Conference on Virtual Reality, Archaeology and Cultural Heritage, VAST’11, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 2011, pp. 73–80.

[47]

C. Sánchez-Belenguer, E. Vendrell-Vidal, An efficient technique to recompose archaeological artifacts from fragments, International Conference on Virtual Systems & Multimedia (VSMM), IEEE, 2014, pp. 337–344.

[48]

J.C. McBride, B.B. Kimia, Archaeological fragment reconstruction using curve-matching, 2003 Conference on Computer Vision and Pattern Recognition Workshop, volume 1, IEEE, 2003.

[49]

H. Huang, K. Yin, M. Gong, D. Lischinski, D. Cohen-Or, U. Ascher, B. Chen, ”Mind the gap”: tele-registration for structure-driven image completion, ACM Trans Graph 32 (6) (2013) 174:1–174:10.

[50]

M.-M. Paumard, D. Picard, H. Tabia, Image reassembly combining deep learning and shortest path problem, ECCV, 2018, pp. 153–167.

[51]

A. Aides, T. Avraham, Y. Schechner, Multiscale ultrawide foveated video extrapolation, ICCP, 2011, pp. 1–8.

[52]

J.B. Huang, S.B. Kang, N. Ahuja, J. Kopf, Image completion using planar structure guidance, ACM Trans Graph 33 (4) (2014) 1–10.

[53]

J.-J. Huang, P.L. Dragotti, Photo realistic image completion via dense correspondence., IEEE Trans. Image Processing 27 (11) (2018) 5234–5247.

[54]

S. Iizuka, E. Simo-Serra, H. Ishikawa, Globally and locally consistent image completion., ACM Trans Graph 36 (4) (2017) 107:1–107:14.

[55]

Y. Poleg, S. Peleg, Alignment and mosaicing of non-overlapping images, ICCP, IEEE, 2012, pp. 1–8.

[56]

S. Darabi, E. Shechtman, C. Barnes, D.B. Goldman, P. Sen, Image melding: combining inconsistent images using patch-based synthesis, ACM Transactions on Graphics (TOG) 31 (4) (2012) 82:1–82:10.

[57]

J.-C. Latombe, Robot motion planning, Kluwer Academic Publishers, Norwell, MA, USA, 1991.

[58]

N. Derech, A. Tal, I. Shimshoni, Placement visualization, 2019. https://www.youtube.com/watch?v=2-pjbiiJ1GQ&feature=youtu.be.

[59]

R. Litman, S. Korman, A.M. Bronstein, S. Avidan, Inverting RANSAC: global model detection via inlier rate estimation, CVPR, 2015, pp. 5243–5251.

[60]

H. Chang, O. Fried, Y. Liu, S. DiVerdi, A. Finkelstein, Palette-based photo recoloring., ACM Trans Graph 34 (4) (2015) 139.

[61]

S. Ghafurian, I. Hacihaliloglu, D.N. Metaxas, V. Tan, K. Li, A computationally efficient 3D/2D registration method based on image gradient direction probability density function, Neurocomput. 229 (C) (2017) 100–108.

[62]

R. Gregor, D. Bauer, I. Sipiran, P. Perakis, T. Schreck, Automatic 3D Object Fracturing for Evaluation of Partial Retrieval and Object Restoration Tasks - Benchmark and Application to 3D Cultural Heritage Data, Eurographics Workshop on 3D Object Retrieval, The Eurographics Association, 2015, pp. 7–14.

[63]

N. Derech, A. Tal, I. Shimshoni, Web site: Solving Archaeological Puzzles, 2019, (https://cgm.technion.ac.il/Computer-Graphics-Multimedia/Software/Solving/).

[64]

G. Liu, F.A. Reda, K.J. Shih, T.-C. Wang, A. Tao, B. Catanzaro, Image inpainting for irregular holes using partial convolutions, Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 85–100.

Cited By

Zhang YFang ZYang XZhang SHe BDou HYan JZhang YWu FEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Reconnecting the Broken Civilization: Patchwork Integration of Fragments from Ancient ManuscriptsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613804(1157-1166)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613804
Yılmaz SNabiyev V(2023)Comprehensive survey of the solving puzzle problemsComputer Science Review10.1016/j.cosrev.2023.10058650:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.cosrev.2023.100586
Niu XWang QLiu BZhang J(2022)An Automatic Chinaware Fragments Reassembly Method Framework Based on Linear Feature of Fracture Surface ContourJournal on Computing and Cultural Heritage 10.1145/356909116:1(1-22)Online publication date: 25-Oct-2022
https://dl.acm.org/doi/10.1145/3569091
Show More Cited By

Index Terms

Solving archaeological puzzles

Index terms have been assigned to the content through auto-classification.

Recommendations

Solving jigsaw puzzles using image features

In this article, we describe a method for automatic solving of the jigsaw puzzle problem based on using image features instead of the shape of the pieces. The image features are used for obtaining an accurate measure for edge similarity to be used in a ...
No easy puzzles

We model jigsaw puzzle solving and study the number of edge matchings required.Realistic jigsaw puzzles require ( n 2 ) comparisons in the worst case.Realistic jigsaw puzzles require ( n 2 ) comparisons on average.Generalised puzzles require (tightly) O ...
Solving Sudoku puzzles: nifty course assignments

The puzzle game Sudoku has recently gained considerable popularity. The puzzle involves completing a grid of cells by assigning a single number to each cell (see description below). This type of puzzle is a good assignment candidate for use when ...

Comments

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 119, Issue C

Nov 2021

516 pages

ISSN:0031-3203

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 November 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YFang ZYang XZhang SHe BDou HYan JZhang YWu FEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Reconnecting the Broken Civilization: Patchwork Integration of Fragments from Ancient ManuscriptsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3613804(1157-1166)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3613804
Yılmaz SNabiyev V(2023)Comprehensive survey of the solving puzzle problemsComputer Science Review10.1016/j.cosrev.2023.10058650:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.cosrev.2023.100586
Niu XWang QLiu BZhang J(2022)An Automatic Chinaware Fragments Reassembly Method Framework Based on Linear Feature of Fracture Surface ContourJournal on Computing and Cultural Heritage 10.1145/356909116:1(1-22)Online publication date: 25-Oct-2022
https://dl.acm.org/doi/10.1145/3569091
Cascone LDondi PLombardi LNarducci F(2022)Automatic Classification of Fresco Fragments: A Machine and Deep Learning StudyImage Analysis and Processing – ICIAP 202210.1007/978-3-031-06427-2_58(701-712)Online publication date: 23-May-2022
https://dl.acm.org/doi/10.1007/978-3-031-06427-2_58

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents