Abstract
The graph matching optimization problem is an essential component for many tasks in computer vision, such as bringing two deformable objects in correspondence. Naturally, a wide range of applicable algorithms have been proposed in the last decades. Since a common standard benchmark has not been developed, their performance claims are often hard to verify as evaluation on differing problem instances and criteria make the results incomparable. To address these shortcomings, we present a comparative study of graph matching algorithms. We create a uniform benchmark where we collect and categorize a large set of existing and publicly available computer vision graph matching problems in a common format. At the same time we collect and categorize the most popular open-source implementations of graph matching algorithms. Their performance is evaluated in a way that is in line with the best practices for comparing optimization algorithms. The study is designed to be reproducible and extensible to serve as a valuable resource in the future.
Our study provides three notable insights: (i) popular problem instances are exactly solvable in substantially less than 1 s, and, therefore, are insufficient for future empirical evaluations; (ii) the most popular baseline methods are highly inferior to the best available methods; (iii) despite the NP-hardness of the problem, instances coming from vision applications are often solvable in a few seconds even for graphs with more than 500 vertices.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
For sets A and B the notation \(x\in A^B\) denotes a vector x whose coordinates take on values from the set A and are indexed by elements of B, i.e., each element of B corresponds to a value from A.
- 2.
The web site for the benchmark is available at https://vislearn.github.io/gmbench/.
- 3.
- 4.
Non-negative in original maximization formulations.
- 5.
Reparametrized costs are also known as reduced costs, e.g., in the simplex tableau.
- 6.
Most of the considered graphical models are dense in terms of [61].
References
Adams, W.P., Johnson, T.A.: Improved linear programming-based lower bounds for the quadratic assignment problem. Discrete Math. Theor. Comput. Sci. 16, 43–77 (1994)
Ahuja, R.K., Magnanti, T.L., Orlin, J.B.: Network Flows: Theory. Prentice Hall, Algorithms and Applications (1993)
Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: GraphFlow - 6D large displacement scene flow via graph matching. In: Proceedings of the DAGM German Conference on Pattern Recognition (2015)
Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: Graph Matching Problems for GraphFlow - 6D Large Displacement Scene Flow Problem Instances (2018), https://research-explorer.app.ist.ac.at/record/5573
Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: Project GraphFlow - 6D Large Displacement Scene Flow Images (2018). https://hci.iwr.uni-heidelberg.de/vislearn/research/image-matching/graphflow/
Beiranvand, V., Hare, W., Lucet, Y.: Best practices for comparing optimization algorithms. Optimization and Engineering (2017)
Bernard, F., Thunberg, J., Gemmar, P., Hertel, F., Husch, A., Goncalves, J.: A solution for multi-alignment by transformation synchronisation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)
Bertsekas, D.P.: A distributed algorithm for the assignment problem. Lab. for Information and Decision Systems Working Paper, MIT (1979)
Bertsekas, D.P.: Nonlinear programming, second edition. Athena scientific (1999)
Burkard, R., Dell’Amico, M., Martello, S.: Assignment Problems. SIAM (2009)
Burkard, R., Karisch, S., Rendl, F.: QAPLIB - a quadratic assignment problem library. J. Global Optim. 10, 391–403 (1997)
Drezner, Z.: The quadratic assignment problem. In: Laporte, G., Nickel, S., da Gama, F.S. (eds.) Location Science, pp. 345–363. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-13111-5_13
Caetano, T.: Data for Learning Graph Matching (2011). https://www.tiberiocaetano.com/data/
Caetano, T.S., McAuley, J.J., Cheng, L., Le, Q.V., Smola, A.J.: Learning Graph Matching. IEEE Trans. Pattern Anal. Mach. Intell. 31, 1048–1058 (2009)
Cela, E.: The Quadratic Assignment Problem: Theory and Algorithms, vol. 1. Springer Science & Business Media (2013)
Cho, M., Jungmin, L., Kyoung, M.L.: Reweighted Random Walks for Graph Matching: Project Page (2010). https://cv.snu.ac.kr/research/~RRWM/
Cho, M., Lee, J., Lee, K.M.: Reweighted random walks for graph matching. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 492–505. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15555-0_36
Cho, M., Sun, J., Duchenne, O., Ponce, J.: Finding matches in a haystack: a max-pooling strategy for graph matching in the presence of outliers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)
Cho, M., Sun, J., Duchenne, O., Ponce, J.: Finding matches in a haystack source code (2014). https://www.di.ens.fr/willow/research/maxpoolingmatching/
Cour, T.: Graph Matching Toolbox in MATLAB (2010). http://www.timotheecour.com/software/graph_matching/graph_matching.html
Cour, T., Srinivasan, P., Shi, J.: Balanced graph matching. In: Advances in Neural Information Processing Systems (2007)
Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Mathematical Programming (2002)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2007 results (2007). http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html
Foggia, P., Percannella, G., Vento, M.: Graph matching and learning in pattern recognition in the last 10 years. Int. J. Pattern Recogn. Artif. Intelli. 28, 1450001 (2014)
Frank, M., Wolfe, P., et al.: An Algorithm for Quadratic Programming. Naval Research Logistics Quarterly (1956)
Globerson, A., Jaakkola, T.S.: Fixing max-product: convergent message passing algorithms for MAP LP-relaxations. In: Advances in Neural Information Processing Systems (2008)
Gold, S., Rangarajan, A.: A graduated assignment algorithm for graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1450001 (1996)
Guignard, M., Kim, S.: Lagrangean decomposition: a model yielding stronger lagrangean bounds. Mathematical Programming (1987)
Heimann, T., Meinzer, H.P.: Statistical shape models for 3D medical image segmentation: a review. Med. Image Anal. 13, 543–563 (2009)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press (2012)
Hutschenreiter, L., Haller, S., Feineis, L., Rother, C., Kainmüller, D., Savchynskyy, B.: Fusion moves for graph matching. In: Proceedings of the IEEE International Conference on Computer Vision (2021)
Hutschenreiter, L., Haller, S., Feineis, L., Rother, C., Kainmüller, D., Savchynskyy, B.: Fusion moves for graph matching website (2021). https://vislearn.github.io/libmpopt/iccv2021/
Jiang, B., Tang, J., Ding, C., Luo, B.: A local sparse model for matching problem. In: Proceedings of the AAAI Conference on Artificial Intelligence (2015)
Kainmueller, D., Jug, F., Rother, C., Myers, G.: Active graph matching for automatic joint segmentation and annotation of C. elegans. In: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (2014)
Kainmueller, D., Jug, F., Rother, C., Myers, G.: Graph matching problems for annotating C. elegans (2017). https://doi.org/10.15479/AT:ISTA:57
Kappes, J.H., et al.: A comparative study of modern inference techniques for structured discrete energy minimization problems. Int. J. Comput. Vis. 115, 155–184 (2015)
Kappes, J.H., et al.: OpenGM Benchmark (2015). http://hciweb2.iwr.uni-heidelberg.de/opengm/index.php?l0=benchmark
Kolmogorov, V.: Feature Correspondence via Graph Matching Source Code (2015). https://pub.ist.ac.at/~vnk/software.html#GRAPH-MATCHING
Komodakis, N., Paragios, N.: Beyond Loose LP-Relaxations: Optimizing MRFs by Repairing Cycles. In: Proceedings of the European Conference on Computer Vision (2008)
Kosowsky, J., Yuille, A.: The invisible hand algorithm: solving the assignment problem with statistical physics. Neural Networks (1994)
Kuhn, H.W.: The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly (1955)
Lawler, E.L.: The quadratic assignment problem. Manage. Sci. 9, 586–599 (1963)
Leordeanu, M., Hebert, M.: A Spectral Technique for Correspondence Problems Using Pairwise Constraints. In: Proceedings of the IEEE International Conference on Computer Vision (2005)
Leordeanu, M.: Efficient methods for graph matching and MAP inference (2013). https://sites.google.com/site/graphmatchingmethods/
Leordeanu, M., Hebert, M.: Cars and Motor Models. https://datasets.d2.mpi-inf.mpg.de/discrete_cv_problems/car_motor_graph_matching.zip
Leordeanu, M., Hebert, M., Sukthankar, R.: An Integer projected fixed point method for graph matching and MAP inference. In: Advances in Neural Information Processing Systems (2009)
Leordeanu, M., Sukthankar, R., Hebert, M.: Unsupervised Learning for Graph Matching. International Journal of Computer Vision (2012)
Loiola, E.M., Maia de Abreu, N.M., Boaventura-Netto, P.O., Hahn, P., Querido, T.: An analytical survey for the quadratic assignment problem. Eur. J. Oper. Res. (2007)
Ma, J., Jiang, X., Fan, A., Jiang, J., Yan, J.: Image matching from handcrafted to deep features: a survey. Int. J. Comput. Vis. 129, 23–79 (2021)
Pardalos, P.M., Rendl, F., Wolkowicz, H.: The Quadratic Assignment Problem - A Survey and Recent Developments. Quadratic Assignment and Related Problems (1993)
Rolínek, M., Swoboda, P., Zietlow, D., Paulus, A., Musil, V., Martius, G.: Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers. In: Proceedings of the European Conference on Computer Vision (2020)
Rother, C., Kolmogorov, V., Lempitsky, V.S., Szummer, M.: Optimizing binary MRFs via extended roof duality. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007)
Sahni, S.: Computationally related problems. SIAM J. Comput. 3, 262–279 (1974)
Savchynskyy, B.: Discrete graphical models - an optimization perspective. In: Foundations and Trends in Computer Graphics and Vision (2019)
Sun, H., Zhou, W., Fei, M.: A survey on graph matching in computer vision. In: International Congress on Image and Signal Processing. BioMedical Engineering and Informatics (2020)
Swoboda, P.: LPMP Source Code (2021). https://github.com/LPMP/LPMP
Swoboda, P., Rother, C., Abu Alhaija, H., Kainmuller, D., Savchynskyy, B.: A study of lagrangean decompositions and dual ascent solvers for graph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Torresani, L., Kolmogorov, V., Rother, C.: Hotel and House-sparse Models. https://datasets.d2.mpi-inf.mpg.de/discrete_cv_problems/graph_matching_hotel_house.zip
Torresani, L., Kolmogorov, V., Rother, C.: A dual decomposition approach to feature correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 35, 259–271 (2013)
Tourani, S., Shekhovtsov, A., Rother, C., Savchynskyy, B.: MPLP++: fast, parallel dual block-coordinate ascent for dense graphical models. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 264–281. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_16
Tourani, S., Shekhovtsov, A., Rother, C., Savchynskyy, B.: Taxonomy of dual block-coordinate ascent methods for discrete energy minimization. In: Proceedings of the Conference on Artifical Intelligence and Statistics (2020)
Vogelstein, J.T., et al.: Fast approximate quadratic programming for graph matching. PLOS ONE (2015)
Yan, J., Yin, X.C., Lin, W., Deng, C., Zha, H., Yang, X.: A short survey of recent advances in graph matching. In: Proceedings of the ACM International Conference on Multimedia Retrieval (2016)
Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. 38, 13-es (2006)
Zass, R., Shashua, A.: Probabilistic graph and hypergraph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)
Zhang, Z.: HungarianBP: pairwise matching through max-weight bipartite belief propagation source code (2016). https://github.com/zzhang1987/HungarianBP
Zhang, Z., Shi, Q., McAuley, J., Wei, W., Zhang, Y., van den Hengel, A.: Pairwise matching through max-weight bipartite belief propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
Zhou, F.: Implementation of Factorized Graph Matching (2018). https://github.com/zhfe99/fgm
Zhou, F., la Torre, F.D.: Factorized graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 38, 1774–1789 (2016)
Acknowledgements
This work was supported by the DFG grant SA 2640/2-1 and the Helmholtz Information & Data Science School for Health. We thank the ZIH at TU Dresden for providing high performance computing resources.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Haller, S. et al. (2022). A Comparative Study of Graph Matching Algorithms in Computer Vision. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-20050-2_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20049-6
Online ISBN: 978-3-031-20050-2
eBook Packages: Computer ScienceComputer Science (R0)