Abstract
A novel region-based approach is proposed to find a thin plate spline map between a pair of deformable 3D objects represented by triangular surface meshes. The proposed method works without landmark extraction and feature correspondences. The aligning transformation is simply found by solving a system of integral equations. Each equation is generated by integrating a non-linear function over the object domains. We derive recursive formulas for the efficient computation of these integrals for open and closed surface meshes. Based on a series of comparative tests on a large synthetic dataset, our triangular mesh-based algorithm outperforms state of the art methods both in terms of computing time and accuracy. The applicability of the proposed approach has been demonstrated on the registration of 3D lung CT volumes, brain surfaces and 3D human faces.



















Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Alliez, P., Saboret, L., & Guennebaud, G. (2000). Surface reconstruction from point sets. In CGAL user and reference manual (4th edn.). CGAL Editorial Board. http://doc.cgal.org/4.4/Manual/packages.html#PkgSurfaceReconstructionFromPointSetsSummary. Accessed 12 Apr 2018.
Aubry, M., Schlickewei, U., & Cremers D. (2011). The wave kernel signature: A quantum mechanical approach to shape analysis. In Proceedings of international conference on computer vision (pp. 1626–1633). https://doi.org/10.1109/ICCVW.2011.6130444.
Audette, M. A., Ferrie, F. P., & Peters, T. M. (2000). An algorithmic overview of surface registration techniques for medical imaging. Medical Image Analysis, 4(3), 201–217. https://doi.org/10.1016/S1361-8415(00)00014-1.
Blais, F., Picard, M., & Godin, G. (2004). Accurate 3D acquisition of freely moving objects. In Proceedings of 2nd international symposium on 3D data processing, visualization and transmission (pp. 422–429). https://doi.org/10.1109/TDPVT.2004.1335269.
Bookstein, F. L. (1989). Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machince Intelligence, 11(6), 567–585. https://doi.org/10.1109/34.24792.
Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006). Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching. Proceedings of the National Academy of Sciences of the United States of America, 103(5), 1168–1172. https://doi.org/10.1073/pnas.0508601103.
Bronstein, M. M., & Bronstein, A. M. (2011). Shape recognition with spectral distances. IEEE Transactions on Pattern Analysis and Machince Intelligence, 33(5), 1065–1071. https://doi.org/10.1109/TPAMI.2010.210.
Bryant, A. S., & Cerfolio, R. J. (2006). The maximum standardized uptake values on integrated FDG-PET/CT is useful in differentiating benign from malignant pulmonary nodules. The Annals of Thoracic Surgery, 82, 1016–1020. https://doi.org/10.1016/j.athoracsur.2006.03.095.
Camion, V., & Younes, L. (2001). Geodesic interpolating splines. In M. Figueiredo, J. Zerubia, & A. Jain (Eds.), Energy minimization methods in computer vision and pattern recognition, Lecture notes in computer science (Vol. 2134, pp. 513–527). Berlin: Springer. https://doi.org/10.1007/3-540-44745-8_34.
Cheung, W., & Hamarneh, G. (2009). \(n\)-sift: \(n\)-dimensional scale invariant feature transform. IEEE Transactions on Image Processing, 18(9), 2012–2021. https://doi.org/10.1109/TIP.2009.2024578.
Christensen, G. (1999). Consistent linear-elastic transformations for image matching. In Proceedings of information processing in medical imaging (pp. 224–237). Springer. https://doi.org/10.1007/3-540-48714-X_17.
Chui, H., & Rangarajan, A. (2003). A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding, 89(2–3), 114–141. https://doi.org/10.1016/S1077-3142(03)00009-2.
Corneanu, C. A., Simn, M. O., Cohn, J. F., & Guerrero, S. E. (2016). Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machince Intelligence, 38(8), 1548–1568. https://doi.org/10.1109/TPAMI.2016.2515606.
Creusot, C., Pears, N., & Austin, J. (2013). A machine-learning approach to keypoint detection and landmarking on 3D meshes. International Journal on Computer Vision, 102(1–3), 146–179. https://doi.org/10.1007/s11263-012-0605-9.
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 1, 886–893. https://doi.org/10.1109/CVPR.2005.177.
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society Series B (Methodological), 39(1), 1–38.
Domokos, C., Nemeth, J., & Kato, Z. (2012). Nonlinear shape registration without correspondences. IEEE Transactions on Pattern Analysis and Machince Intelligence, 34(5), 943–958. https://doi.org/10.1109/TPAMI.2011.200.
Duchenne, O., Bach, F., Kweon, I. S., & Ponce, J. (2011). A tensor-based algorithm for high-order graph matching. IEEE Transactions on Pattern Analysis and Machince Intelligence, 33(12), 2383–2395. https://doi.org/10.1109/TPAMI.2011.110.
Forsyth, D., Torr, P., & Zisserman, A. (Eds.) (2008). Feature correspondence via graph matching: Models and global optimization. Lecture notes in computer science (Vol. 5303). Berlin: Springer. https://doi.org/10.1007/978-3-540-88688-4_44.
Geman, S., & Geman, D. (1984). Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machince Intelligence, 6(6), 721–741. https://doi.org/10.1109/TPAMI.1984.4767596.
Glocker, B., Komodakis, N., Tziritas, G., Navab, N., & Paragios, N. (2008). Dense image registration through MRFs and efficient linear programming. Medical Image Analysis, 12(6), 731–741. https://doi.org/10.1016/j.media.2008.03.006. (Special issue on information processing in medical imaging 2007).
Glocker, B., Sotiras, A., Komodakis, N., & Paragios, N. (2011). Deformable medical image registration: Setting the state of the art with discrete methods. Annual Review of Biomedical Engineering, 13(1), 219–244. https://doi.org/10.1146/annurev-bioeng-071910-124649.
Guo, H., Rangarajan, A., & Joshi, S. (2006). Diffeomorphic point matching. In N. Paragios, Y. Chen, & O. Faugeras (Eds.), Handbook of mathematical models in computer vision (pp. 205–219). Springer. https://doi.org/10.1007/0-387-28831-7_13
Heike, C. L., Cunningham, M. L., Hing, A. V., Stuhaug, E., & Starr, J. R. (2009). Picture perfect?: Reliability of craniofacial anthropometry using three-dimensional digital stereophotogrammetry. Plastic and Reconstructive Surgery, 124(4), 1261–1272. https://doi.org/10.1097/PRS.0b013e3181b454bd.
Holden, M. (2008). A review of geometric transformations for nonrigid body registration. IEEE Transactions on Medical Imaging, 27(1), 111–128. https://doi.org/10.1109/TMI.2007.904691.
Jian, B., & Vemuri, B. (2011). Robust point set registration using gaussian mixture models. IEEE Transactions on Pattern Analysis and Machince Intelligence, 33(8), 1633–1645. https://doi.org/10.1109/TPAMI.2010.223.
Jin, G., Lee, S. J., Hahn, J. K., Bielamowicz, S., Mittal, R., & Walsh, R. (2006). 3D surface reconstruction and registration for image guided medialization laryngoplasty. In Proceedings of the international conference on advances in visual computing (pp. 761–770). Berlin: Springer. https://doi.org/10.1007/11919476_76.
Kazhdan, M., Bolitho, M., & Hoppe, H. (2006) Poisson surface reconstruction. In Proceedings of the fourth eurographics symposium on geometry processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SGP ’06 (pp. 61–70). https://doi.org/10.2312/SGP/SGP06/061-070.
Kim, V. G., Lipman, Y., & Funkhouser, T. (2011). Blended intrinsic maps. ACM Transactions on Graphics, 30(4), 79:1–79:12. https://doi.org/10.1145/2010324.1964974.
Koehl, P. (2012). Fast recursive computation of 3D geometric moments from surface meshes. IEEE Transactions on Pattern Analysis and Machince Intelligence, 34(11), 2158–2163. https://doi.org/10.1109/TPAMI.2012.23.
Li, S. Z., & Singh, S. (2009). Markov random field modeling in image analysis (Vol. 26). Berlin: Springer. https://doi.org/10.1007/978-1-84800-279-1.
Lin, H., Gao, J., Zhou, Y., Lu, G., Ye, M., Zhang, C., et al. (2013). Semantic decomposition and reconstruction of residential scenes from lidar data. ACM Transactions on Graphics, 32(4), 66:1–66:10. https://doi.org/10.1145/2461912.2461969.
Lipman, Y., & Daubechies, I. (2011). Conformal wasserstein distances: Comparing surfaces in polynomial time. Advances in Mathematics, 227(3), 1047–1077. https://doi.org/10.1016/j.aim.2011.01.020.
Lombaert, H., Grady, L., Pennec, X., Ayache, N., & Cheriet, F. (2014). Spectral log-demons: Diffeomorphic image registration with very large deformations. International Journal on Computer Vision, 107(3), 254–271.
Lourakis, M. (2004). Levmar: Levenberg–Marquardt nonlinear least squares algorithms in C/C++. http://www.ics.forth.gr/~lourakis/levmar/. Accessed 12 Apr 2018.
Ma, J., Zhao, J., Tian, J., Tu, Z., & Yuille, A. (2013). Robust estimation of nonrigid transformation for point set registration. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 2147–2154). https://doi.org/10.1109/CVPR.2013.279.
Mian, A., Bennamoun, M., & Owens, R. (2007). An efficient multimodal 2D–3D hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machince Intelligence, 29(11), 1927–1943. https://doi.org/10.1109/TPAMI.2007.1105.
Mitra, J., Kato, Z., Marti, R., Oliver, A., Llado, X., Ghose, S., Vilanova, J. C., & Meriaudeau, F. (2011) A non-linear diffeomorphic framework for prostate multimodal registration. In Proceedings of international conference on digital image computing: Techniques and applications (pp. 31–36). Noosa: IEEE.
Mitra, J., Kato, Z., Ghose, S., Sidibe, D., Mart, R., Llado, X., Oliver, A., Vilanova, J. C., & Meriaudeau, F. (2012a) Spectral clustering to model deformations for fast multimodal prostate registration. In Proceedings of international conference on pattern recognition. IAPR, IEEE, Tsukuba Science City, Japan (pp. 2622–2625).
Mitra, J., Kato, Z., Marti, R., Oliver, A., Llado, X., Sidibe, D., et al. (2012b). A spline-based non-linear diffeomorphism for multimodal prostate registration. Medical Image Analysis, 16(6), 1259–1279. https://doi.org/10.1016/j.media.2012.04.006.
Myronenko, A., & Song, X. (2010). Point set registration: Coherent point drift. IEEE Transactions on Pattern Analysis and Machince Intelligence, 32(12), 2262–2275. https://doi.org/10.1109/TPAMI.2010.46.
Nair, P., & Cavallaro, A. (2009). 3D face detection, landmark localization, and registration using a point distribution model. IEEE Transactions on Multimedia, 11(4), 611–623. https://doi.org/10.1109/tmm.2009.2017629.
Newcombe, R. A., Fox, D., & Seitz, S. M. (2015). Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 343–352). https://doi.org/10.1109/CVPR.2015.7298631.
Nooruddin, F. S., & Turk, G. (2003). Simplification and repair of polygonal models using volumetric techniques. IEEE Transactions on Visualization and Computer Graphics, 9(2), 191–205. https://doi.org/10.1109/TVCG.2003.1196006.
Papazov, C., & Burschka, D. (2011). Deformable 3D shape registration based on local similarity transforms. Computer Graphics Forum, 30(5), 1493–1502.
Pennec, X., Cachier, P., & Ayache, N. (1999). Understanding the ”demons algorithm”: 3D non-rigid registration by gradient descent. In C. Taylor, & A. Colchester (Eds.), Medical image computing and computer-assisted intervention, Lecture notes in computer science (Vol. 1679, pp. 597–605). Berlin: Springer. https://doi.org/10.1007/10704282_64
Peyré, G., & Cohen, L. (2009). Geodesic methods for shape and surface processing. In J. M. R. S. Tavares & R. M. N. Jorge (Eds.), Advances in computational vision and medical image processing, computational methods in applied sciences (pp. 29–56). Netherlands: Springer. https://doi.org/10.1007/978-1-4020-9086-8_2.
Postelnicu, G., Zollei, L., Desikan, R., & Fischl, B. (2007). Geometry driven volumetric registration. In N. Karssemeijer & B. Lelieveldt (Eds.), Information processing in medical imaging, Lecture notes in computer science (pp. 675–686). Berlin: Springer. https://doi.org/10.1007/978-3-540-73273-0_56.
Pozo, J. M., Villa-Uriol, M. C., & Frangi, A. F. (2011). Efficient 3D geometric and zernike moments computation from unstructured surface meshes. IEEE Transactions on Pattern Analysis and Machince Intelligence, 33(3), 471–484. https://doi.org/10.1109/TPAMI.2010.139.
Rineau, L., & Yvinec, M. (2007). A generic software design for delaunay refinement meshing. Computational Geometry, 38(12), 100–110. https://doi.org/10.1016/j.comgeo.2006.11.008.
Rineau, L., & Yvinec, M. (2012). 3D surface mesh generation. In CGAL user and reference manual (4.0 edn.). CGAL Editorial Board.
Sagawa, R., Akasaka, K., Yagi, Y., Hamer, H., & Van Gool, L. (2009). Elastic convolved ICP for the registration of deformable objects. In Proceedings of international conference on computer vision (pp. 1558–1565). Kyoto: IEEE. https://doi.org/10.1109/ICCVW.2009.5457428.
Sánta, Z., & Kato, Z. (2012a). A unifying framework for non-linear registration of 3D objects. In Proceedings of IEEE international conference on cognitive infocommunications (pp. 547–552). Kosice: IEEE.
Sánta, Z., & Kato, Z. (2012b). Elastic registration of 3D deformable objects. In Proceedings of international conference on digital image computing: Techniques and applications. Fremantle: IEEE.
Sánta, Z., & Kato, Z. (2013). Correspondence-less non-rigid registration of triangular surface meshes. In Proceedings of IEEE conference on computer vision and pattern recognition. Portland: IEEE.
Sánta, Z., & Kato, Z. (2016). 3D face alignment without correspondences. In Proceedings of European conference on computer vision workshops (pp. 521–535). https://doi.org/10.1007/978-3-319-48881-3_36.
Savran, A., Alyüz, N., Dibekliolu, H., Celiktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008). Bosphorus database for 3D face analysis. In Biometrics and identity management (pp. 47–56). Springer. https://doi.org/10.1007/978-3-540-89991-4_6.
Scott, D. W. (2009). The l2E method. Wiley Interdisciplinary Reviews: Computational Statistics, 1(1), 45–51. https://doi.org/10.1002/wics.4.
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 519–528). IEEE Computer Society. https://doi.org/10.1109/CVPR.2006.19.
Sharifi, A., Jones, R., Ayoub, A., Moos, K., Walker, F., Khambay, B., et al. (2008). How accurate is model planning for orthognathic surgery? International Journal of Oral and Maxillofacial Surgery, 37(12), 1089–1093. https://doi.org/10.1016/j.ijom.2008.06.011.
Shewchuk, J. R. (2002). Delaunay refinement algorithms for triangular mesh generation. Computational Geometry. 16th ACM Symposium on Computational Geometry, 22(13), 21–74. https://doi.org/10.1016/S0925-7721(01)00047-5.
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) Sun RGB-D: A RGB-D scene understanding benchmark suite. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 567–576). https://doi.org/10.1109/CVPR.2015.7298655.
Sotiras, A., Davatzikos, C., & Paragios, N. (2013). Deformable medical image registration: A survey. IEEE Transactions on Medical Imaging, 32(7), 1153–1190. https://doi.org/10.1109/TMI.2013.2265603.
Sukno, F. M., Waddington, J. L., & Whelan, P. F. (2015). 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics, 45(9), 1717–1730. https://doi.org/10.1109/TCYB.2014.2359056.
Sun, J., Ovsjanikov, M., & Guibas, L. (2009). A concise and provably informative multi-scale signature based on heat diffusion. Computer Graphics Forum, 28(5), 1383–1392. https://doi.org/10.1111/j.1467-8659.2009.01515.x.
Tam, G. K. L., Cheng, Z. Q., Lai, Y. K., Langbein, F. C., Liu, Y., Marshall, D., et al. (2013). Registration of 3D point clouds and meshes: A survey from rigid to nonrigid. IEEE Transactions on Visualization and Computer Graphics, 19(7), 1199–1217. https://doi.org/10.1109/TVCG.2012.310.
Tamas, L., Frohlich, R., & Kato, Z. (2014). Relative pose estimation and fusion of omnidirectional and lidar cameras. In Proceedings of the ECCV workshop on computer vision for road scene understanding and autonomous driving. Lecture notes in computer science. Berlin: Springer.
Tanács, A., Lindblad, J., Sladoje, N., & Kato, Z. (2015). Estimation of linear deformations of 2D and 3D fuzzy objects. Pattern Recognition, 48(4), 1391–1403. https://doi.org/10.1016/j.patcog.2014.10.006.
Thirion, J. P. (1998). Image matching as a diffusion process: An analogy with maxwell’s demons. Medical Image Analysis, 2(3), 243–260. https://doi.org/10.1016/S1361-8415(98)80022-4.
Vercauteren, T., Pennec, C., Perchant, A., & Ayache, N. (2007). Non-parametric diffeomorphic image registration with the demons algorithm. In N. Ayache, S. Ourselin, & A. Maeder (Eds.), Medical image computing and computer-assisted intervention. Lecture notes in computer science (pp. 319–326). Berlin: Springer. https://doi.org/10.1007/978-3-540-75759-7_39.
Wahba, G. (1990). Spline models for observational data. Society for Industrial and Applied Mathematics,. https://doi.org/10.1137/1.9781611970128.
Weise, T., Leibe, B., Van Gool, L. (2007). Fast 3D scanning with automatic motion compensation. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 1–8). https://doi.org/10.1109/CVPR.2007.383291.
Xu, W., Salzmann, M., Wang, Y., & Liu, Y. (2015). Deformable 3D fusion: From partial dynamic 3D observations to complete 4D models. In Proceedings of international conference on computer vision (pp. 2183–2191). https://doi.org/10.1109/ICCV.2015.252.
Yamazaki, S., Kagami, S., & Mochimaru, M. (2013). Non-rigid shape registration using similarity-invariant differential coordinates. In Proceedings of international conference on 3D vision (pp. 191–198) https://doi.org/10.1109/3DV.2013.33.
Zagorchev, L., & Goshtasby, A. (2006). A comparative study of transformation functions for nonrigid image registration. IEEE Transactions on Image Processing, 15(3), 529–538. https://doi.org/10.1109/TIP.2005.863114.
Zaharescu, A., Boyer, E., Varanasi, K., & Horaud, R. (2009). Surface feature detection and description with applications to mesh matching. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 373–380). https://doi.org/10.1109/CVPR.2009.5206748.
Zeng, Y., Wang, C., Gu, X., Samaras, D., & Paragios, N. (2013). A generic deformation model for dense non-rigid surface registration: A higher-order mrf-based approach. In Proceedings of international conference on computer vision (pp. 3360–3367). https://doi.org/10.1109/ICCV.2013.417.
Zeng, Y., Wang, C., Wang, Y., Gu, X., Samaras, D., & Paragios, N. (2010). Dense non-rigid surface registration using high-order graph matching. In Proceedings of IEEE conference on computer vision and pattern recognition (pp. 382–389). https://doi.org/10.1109/CVPR.2010.5540189.
Zimmermann, H. J. (2001). Fuzzy set theory and its applications. Berlin: Springer.
Zitová, B., & Flusser, J. (2003). Image registration methods: A survey. Image and Vision Computing, 21(11), 977–1000. https://doi.org/10.1016/S0262-8856(03)00137-9.
Acknowledgements
This research was partially supported by the NKFI-6 fund through Project K120366; the Research and Development Operational Programme for the project "Modernization and Improvement of Technical Infrastructure for Research and Development of J. Selye University in the Fields of Nanotechnology and Intelligent Space", ITMS 26210120042, co-funded by the European Regional Development Fund; the Agence Universitaire de la Francophonie (AUF) and the Romanian Institute for Atomic Physics (IFA), through the AUF-RO project NETASSIST. Lung images provided by Mediso Ltd., Budapest, Hungary. The MR brain data sets and their manual segmentations were provided by the Center for Morphometric Analysis at Massachusetts General Hospital and the face scans have been obtained from the Bosphorus Dataset (Savran et al. 2008).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Sven J. Dickinson.
Rights and permissions
About this article
Cite this article
Sánta, Z., Kato, Z. Elastic Alignment of Triangular Surface Meshes. Int J Comput Vis 126, 1220–1244 (2018). https://doi.org/10.1007/s11263-018-1084-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-018-1084-4