Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

Published: 26 July 2023 Publication History

Abstract

This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the isosurface of a scalar field, an increasingly common paradigm in applications including photogrammetry, generative modeling, and inverse physics. Existing implementations adapt classic isosurface extraction algorithms like Marching Cubes or Dual Contouring; these techniques were designed to extract meshes from fixed, known fields, and in the optimization setting they lack the degrees of freedom to represent high-quality feature-preserving meshes, or suffer from numerical instabilities. We introduce FlexiCubes, an isosurface representation specifically designed for optimizing an unknown mesh with respect to geometric, visual, or even physical objectives. Our main insight is to introduce additional carefully-chosen parameters into the representation, which allow local flexible adjustments to the extracted mesh geometry and connectivity. These parameters are updated along with the underlying scalar field via automatic differentiation when optimizing for a downstream task. We base our extraction scheme on Dual Marching Cubes for improved topological properties, and present extensions to optionally generate tetrahedral and hierarchically-adaptive meshes. Extensive experiments validate FlexiCubes on both synthetic benchmarks and real-world applications, showing that it offers significant improvements in mesh quality and geometric fidelity.

Supplementary Material

ZIP File (papers_662-supplemental.zip)
supplemental material
MP4 File (papers_662_VOD.mp4)
presentation

References

[1]
Samir Akkouche and Eric Galin. Adaptive implicit surface polygonization using marching triangles. In Computer Graphics Forum, volume 20:2, pages 67--80, 2001.
[2]
Sergei Azernikov and Anath Fischer. Anisotropic meshing of implicit surfaces. In International Conference on Shape Modeling and Applications 2005 (SMI'05), pages 94--103. IEEE, 2005.
[3]
Jules Bloomenthal. Polygonization of implicit surfaces. Computer Aided Geometric Design, 5(4):341--355, 1988. Jules Bloomenthal. An implicit surface polygonizer. Graphics gems, 4:324--350, 1994.
[4]
Jules Bloomenthal, Chandrajit Bajaj, Jim Blinn, Marie-Paule Cani, Brian Wyvill, Alyn Rockwood, and Geoff Wyvill. Introduction to implicit surfaces. Morgan Kaufmann, 1997.
[5]
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural Reflectance Decomposition from Image Collections. In IEEE International Conference on Computer Vision (ICCV), 2021.
[6]
Andrea Bottino, Wim Nuij, and Kees Van Overveld. How to shrinkwrap through a critical point: an algorithm for the adaptive triangulation of iso-surfaces with arbitrary topology. In Proc. Implicit Surfaces, volume 96, pages 53--72, 1996.
[7]
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient Geometry-aware 3D Generative Adversarial Networks. In CVPR, 2022.
[8]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[9]
Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer. In Advances In Neural Information Processing Systems, 2019.
[10]
Zhiqin Chen and Hao Zhang. Neural Marching Cubes. ACM Trans. Graph., 40(6), 2021.
[11]
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[12]
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.00277, 2022a.
[13]
Zhiqin Chen, Andrea Tagliasacchi, Thomas Funkhouser, and Hao Zhang. Neural Dual Contouring. ACM Trans. Graph., 41(4), 2022b.
[14]
Evgeni Chernyaev. Marching Cubes 33: Construction of topologically correct isosurfaces. Technical report, Institute for High Energy Physics, Moscow, 1995.
[15]
Bruno Rodrigues De Araújo, Daniel S Lopes, Pauline Jepp, Joaquim A Jorge, and Brian Wyvill. A survey on implicit surface polygonization. ACM Computing Surveys (CSUR), 47(4):1--39, 2015.
[16]
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. Cvxnets: Learnable convex decomposition. arXiv preprint arXiv:1909.05736, 2019.
[17]
Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE Transactions on Information and Systems, 74(1): 214--224, 1991.
[18]
Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning Deformable Tetrahedral Meshes for 3D Reconstruction. In Advances In Neural Information Processing Systems, 2020.
[19]
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. In Advances In Neural Information Processing Systems, 2022.
[20]
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh R-CNN. ICCV 2019, 2019.
[21]
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[22]
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022.
[23]
Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. Point2Mesh: A Self-Prior for Deformable Meshes. ACM Trans. Graph., 39(4), 2020.
[24]
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, and Samuli Laine. Appearance-Driven Automatic 3D Model Simplification. In Eurographics Symposium on Rendering, 2021.
[25]
Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26]
Hans-Christian Hege, Martin Seebass, Detlev Stalling, and Malte Zöckler. A Generalized Marching Cubes Algorithm Based On Non-Binary Classifications. ZIB Preprint SC-97-05, 1997.
[27]
Adrian Hilton, Andrew J Stoddart, John Illingworth, and Terry Windeatt. Marching triangles: range image fusion for complex object modelling. In Proceedings of 3rd IEEE international conference on image processing, volume 2, pages 381--384. IEEE, 1996.
[28]
Adrian Hilton, John Illingworth, et al. Marching triangles: Delaunay implicit surface triangulation. University of Surrey, 1997.
[29]
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradSim: Differentiable simulation for system identification and visuomotor control. International Conference on Learning Representations (ICLR), 2021.
[30]
Tao Ju, Frank Losasso, Scott Schaefer, and Joe Warren. Dual Contouring of Hermite Data. ACM Trans. Graph., 21(3):339--346, 2002.
[31]
Tasso Karkanis and A James Stewart. Curvature-dependent triangulation of implicit surfaces. IEEE Computer Graphics and Applications, 21(2):60--69, 2001.
[32]
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401--4410, 2019.
[33]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
[34]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3907--3916, 2018.
[35]
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph., 36(4):78:1--78:13, 2017.
[36]
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular Primitives for High-Performance Differentiable Rendering. ACM Transactions on Graphics, 39(6), 2020.
[37]
Gerald Lasser. LDraw.org, 2022. URL https://www.ldraw.org/.
[38]
Thomas Lewiner, Hélio Lopes, Antônio Wilson Vieira, and Geovan Tavares. Efficient implementation of marching cubes' cases with topological guarantees. Journal of graphics tools, 8(2):1--15, 2003.
[39]
Xinghua Liang and Yongjie Zhang. An octree-based dual contouring method for triangular and tetrahedral mesh generation with guaranteed angle range. Engineering with Computers, 30(2):211--222, 2014.
[40]
Yiyi Liao, Simon Donné, and Andreas Geiger. Deep Marching Cubes: Learning Explicit Surface Representations. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2916--2925, 2018.
[41]
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440, 2022.
[42]
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. In International Conference on Computer Vision (ICCV), pages 7707--7716, 2019.
[43]
William E. Lorensen and Harvey E. Cline. Marching Cubes: A High Resolution 3D Surface Construction Algorithm. SIGGRAPH Comput. Graph., 21(4):163--169, 1987.
[44]
Neil H McCormick and Robert B Fisher. Edge-constrained marching triangles. In Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission, pages 348--351. IEEE, 2002.
[45]
Ishit Mehta, Manmohan Chandraker, and Ravi Ramamoorthi. A Level Set Theory for Neural Implicit Evolution Under Explicit Flows. In ECCV 2022, Proceedings, Part II, page 711--729, 2022.
[46]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
[47]
Claudio Montani, Riccardo Scateni, and Roberto Scopigno. Discretized marching cubes. In Proceedings Visualization'94, pages 281--287. IEEE, 1994.
[48]
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Mueller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8270--8280, 2022.
[49]
Ashish Myles, Nico Pietroni, and Denis Zorin. Robust field-aligned global parametrization. ACM Transactions on Graphics (TOG), 33(4):1--14, 2014.
[50]
Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML, 2020.
[51]
Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. Large Steps in Inverse Rendering of Geometry. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), 40(6), 2021.
[52]
Gregory M. Nielson. On Marching Cubes. IEEE Transactions on visualization and computer graphics, 9(3):283--297, 2003.
[53]
Gregory M Nielson. Dual Marching Cubes. In IEEE visualization 2004, pages 489--496. IEEE, 2004.
[54]
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021.
[55]
Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[56]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011.13961, 2020.
[57]
Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoît Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. Advances in Neural Information Processing Systems, 33:22468--22478, 2020.
[58]
RenderPeople. Renderpeople, 2020. https://renderpeople.com/3d-people/.
[59]
Scott Schaefer, Tao Ju, and Joe Warren. Manifold dual contouring. IEEE Transactions on Visualization and Computer Graphics, 13(3):610--619, 2007.
[60]
Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. ARXIV, 2022.
[61]
Roberto Scopigno. A modified look-up table for implicit disambiguation of marching cubes. The visual computer, 10:353--355, 1994.
[62]
Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[63]
Barton T Stander and John C Hart. Interactive re-polygonization of blobby implicit curves. In Proc. Western Computer Graphics Symposium, volume 5, 1995.
[64]
Oded Stein, Eitan Grinspun, and Keenan Crane. Developability of triangle meshes. ACM Trans. Graph., 37(4), 2018.
[65]
Subodh C Subedi, Chaman Singh Verma, and Krishnan Suresh. A review of methods for the geometric post-processing of topology optimized models. Journal of Computing and Information Science in Engineering, 20(6), 2020.
[66]
Minhyuk Sung, Hao Su, Vladimir G Kim, Siddhartha Chaudhuri, and Leonidas Guibas. Complementme: Weakly-supervised component suggestions for 3d modeling. ACM Transactions on Graphics (TOG), 36(6):1--12, 2017.
[67]
Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Computer Vision and Pattern Regognition (CVPR), 2017.
[68]
Kees Van Overveld and Brian Wyvill. Shrinkwrap: An efficient adaptive algorithm for triangulating an iso-surface. The visual computer, 20(6):362--379, 2004.
[69]
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52--67, 2018.
[70]
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), pages 27171--27183, 2021.
[71]
Kangxue Yin, Zhiqin Chen, Siddhartha Chaudhuri, Matthew Fisher, Vladimir Kim, and Hao Zhang. Coalesce: Component assembly by learning to synthesize connections. In Proc. of 3DV, 2020.
[72]
Jonathan Young. xatlas, 2021. https://github.com/jpcy/xatlas.
[73]
Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv preprint arXiv:2110.09788, 2021.
[74]
Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. SCORES: Shape composition with recursive substructure priors. ACM Transactions on Graphics, 37(6):Article 211, 2018.

Cited By

View all
  • (2024)Ray Tracing Harmonic FunctionsACM Transactions on Graphics10.1145/365820143:4(1-18)Online publication date: 19-Jul-2024
  • (2024)Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View SynthesisACM Transactions on Graphics10.1145/365813043:4(1-14)Online publication date: 19-Jul-2024
  • (2024)Reach for the Arcs: Reconstructing Surfaces from SDFs via Tangent PointsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657419(1-12)Online publication date: 13-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 42, Issue 4
August 2023
1912 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3609020
Issue’s Table of Contents
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2023
Published in TOG Volume 42, Issue 4

Check for updates

Author Tags

  1. isosurface extraction
  2. gradient-based mesh optimization
  3. photogrammetry
  4. generative models

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,646
  • Downloads (Last 6 weeks)116
Reflects downloads up to 10 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Ray Tracing Harmonic FunctionsACM Transactions on Graphics10.1145/365820143:4(1-18)Online publication date: 19-Jul-2024
  • (2024)Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based View SynthesisACM Transactions on Graphics10.1145/365813043:4(1-14)Online publication date: 19-Jul-2024
  • (2024)Reach for the Arcs: Reconstructing Surfaces from SDFs via Tangent PointsACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657419(1-12)Online publication date: 13-Jul-2024
  • (2024)MVD^2: Efficient Multiview 3D Reconstruction for Multiview DiffusionACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657403(1-11)Online publication date: 13-Jul-2024
  • (2024)LTM: Lightweight Textured Mesh Extraction and Refinement of Large Unbounded Scenes for Efficient Storage and Real-Time Rendering2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00483(5053-5063)Online publication date: 16-Jun-2024
  • (2024)Self-Supervised Dual Contouring2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00448(4681-4691)Online publication date: 16-Jun-2024
  • (2024)QUADify: Extracting Meshes with Pixel-Level Details and Materials from Images2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00446(4661-4670)Online publication date: 16-Jun-2024
  • (2024)PoNQ: A Neural QEM-Based Mesh Representation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00350(3647-3657)Online publication date: 16-Jun-2024
  • (2024)TeCH: Text-Guided Reconstruction of Lifelike Clothed Humans2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00152(1531-1542)Online publication date: 18-Mar-2024
  • (2024)McGrids: Monte Carlo-Driven Adaptive Grids for Iso-Surface ExtractionComputer Vision – ECCV 202410.1007/978-3-031-72998-0_8(127-144)Online publication date: 30-Sep-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media