Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15095))

Included in the following conference series:

  • 130 Accesses

Abstract

Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at: https://qingqing-zhao.github.io/PhysAvatar.

Y. Zheng and Q. Zhao—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 74.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdal, R., et al.: Gaussian shell maps for efficient 3d human generation. arXiv preprint arXiv:2311.17857 (2023)

  2. Abdrashitov, R., Raichstat, K., Monsen, J., Hill, D.: Robust skin weights transfer via weight inpainting. In: SIGGRAPH Asia 2023 Technical Communications, pp. 1–4 (2023)

    Google Scholar 

  3. Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: Scape: shape completion and animation of people. In: ACM SIGGRAPH 2005 Papers, pp. 408–416 (2005)

    Google Scholar 

  4. Bagautdinov, T., et al.: Driving-signal aware full-body avatars. ACM Trans. Graph. (TOG) 40(4), 1–17 (2021)

    Article  Google Scholar 

  5. Baraff, D., Witkin, A.: Large steps in cloth simulation. In: Seminal Graphics Papers: Pushing the Boundaries, vol. 2, pp. 767–778 (2023)

    Google Scholar 

  6. Bashirov, R., et al.: Morf: mobile realistic fullbody avatars from a monocular video. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3545–3555 (2024)

    Google Scholar 

  7. Bergman, A., Kellnhofer, P., Yifan, W., Chan, E., Lindell, D., Wetzstein, G.: Generative neural articulated radiance fields. Adv. Neural. Inf. Process. Syst. 35, 19900–19916 (2022)

    Google Scholar 

  8. Bertiche, H., Madadi, M., Escalera, S.: CLOTH3D: clothed 3D humans. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 344–359. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_21

    Chapter  Google Scholar 

  9. Bertiche, H., Madadi, M., Escalera, S.: Pbns: physically based neural simulation for unsupervised garment pose space deformation. ACM Trans. Graph. (TOG) 40(6), 1–14 (2021)

    Article  Google Scholar 

  10. Bertiche, H., Madadi, M., Escalera, S.: Neural cloth simulation. ACM Trans. Graph. (TOG) 41(6), 1–14 (2022)

    Article  Google Scholar 

  11. Bertiche, H., Madadi, M., Tylson, E., Escalera, S.: Deepsd: automatic deep skinning and pose space deformation for 3d garment animation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5471–5480 (2021)

    Google Scholar 

  12. Bhat, K.S., Twigg, C.D., Hodgins, J.K., Khosla, P.K., Popović, Z., Seitz, S.M.: Estimating cloth simulation parameters from video. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 37–51 (2003)

    Google Scholar 

  13. Blender Online Coummunity: Blender, blender Foundation. https://www.blender.org/

  14. Chen, B., Shen, Y., Shuai, Q., Zhou, X., Zhou, K., Zheng, Y.: Anidress: animatable loose-dressed avatar from sparse views using garment rigging model. arXiv preprint arXiv:2401.15348 (2024)

  15. Chen, H.Y., Sastry, A., van Rees, W.M., Vouga, E.: Physical simulation of environmentally induced thin shell deformation. ACM Trans. Graph. (TOG) 37(4), 1–13 (2018)

    Google Scholar 

  16. Chen, X., et al.: gdna: towards generative detailed neural avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20427–20437 (2022)

    Google Scholar 

  17. Chen, X., Zheng, Y., Black, M.J., Hilliges, O., Geiger, A.: Snarf: differentiable forward skinning for animating non-rigid neural implicit shapes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11594–11604 (2021)

    Google Scholar 

  18. Chen, Y., et al.: Uv volumes for real-time rendering of editable free-view human performance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16621–16631 (2023)

    Google Scholar 

  19. Chentanez, N., Macklin, M., Müller, M., Jeschke, S., Kim, T.Y.: Cloth and skin deformation with a triangle mesh based convolutional neural network. In: Computer Graphics Forum, vol. 39, pp. 123–134. Wiley Online Library (2020)

    Google Scholar 

  20. Clyde, D., Teran, J., Tamstorf, R.: Modeling and data-driven parameter estimation for woven fabrics. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 1–11 (2017)

    Google Scholar 

  21. Contributors, M.: Openmmlab pose estimation toolbox and benchmark (2020). https://github.com/open-mmlab/mmpose

  22. De Luigi, L., Li, R., Guillard, B., Salzmann, M., Fua, P.: Drapenet: garment generation and self-supervised draping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1451–1460 (2023)

    Google Scholar 

  23. Dong, Z., Chen, X., Yang, J., Black, M.J., Hilliges, O., Geiger, A.: Ag3d: learning to generate 3d avatars from 2d image collections. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14916–14927 (2023)

    Google Scholar 

  24. Feng, Y., Yang, J., Pollefeys, M., Black, M.J., Bolkart, T.: Capturing and animation of body and clothing from monocular video. In: SIGGRAPH Asia 2022 Conference Papers, pp. 1–9 (2022)

    Google Scholar 

  25. Fridovich-Keil, S., Meanti, G., Warburg, F.R., Recht, B., Kanazawa, A.: K-planes: explicit radiance fields in space, time, and appearance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479–12488 (2023)

    Google Scholar 

  26. Genay, A., Lécuyer, A., Hachet, M.: Being an avatar “for real’’: a survey on virtual embodiment in augmented reality. IEEE Trans. Visualizat. Comput. Graph. 28(12), 5071–5090 (2021)

    Article  Google Scholar 

  27. Geng, C., Peng, S., Xu, Z., Bao, H., Zhou, X.: Learning neural volumetric representations of dynamic humans in minutes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8759–8770 (2023)

    Google Scholar 

  28. Grigorev, A., Black, M.J., Hilliges, O.: Hood: hierarchical graphs for generalized modelling of clothing dynamics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16965–16974 (2023)

    Google Scholar 

  29. Grinspun, E., Hirani, A.N., Desbrun, M., Schröder, P.: Discrete shells. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 62–67. Citeseer (2003)

    Google Scholar 

  30. Gundogdu, E., Constantin, V., Seifoddini, A., Dang, M., Salzmann, M., Fua, P.: Garnet: a two-stream network for fast and accurate 3d cloth draping. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8739–8748 (2019)

    Google Scholar 

  31. Guo, J., Li, J., Narain, R., Park, H.S.: Inverse simulation: reconstructing dynamic geometry of clothed humans via optimal control. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14698–14707 (2021)

    Google Scholar 

  32. Habermann, M., Liu, L., Xu, W., Pons-Moll, G., Zollhoefer, M., Theobalt, C.: Hdhumans: a hybrid approach for high-fidelity digital humans. In: Proceedings of the ACM on Computer Graphics and Interactive Techniques, vol. 6, no. 3, pp. 1–23 (2023)

    Google Scholar 

  33. Habermann, M., Liu, L., Xu, W., Zollhoefer, M., Pons-Moll, G., Theobalt, C.: Real-time deep dynamic characters. ACM Trans. Graph. (ToG) 40(4), 1–16 (2021)

    Article  Google Scholar 

  34. Hu, L., et al.: Gaussianavatar: towards realistic human avatar modeling from a single video via animatable 3d gaussians. arXiv preprint arXiv:2312.02134 (2023)

  35. Işık, M., et al.: Humanrf: high-fidelity neural radiance fields for humans in motion. ACM Trans. Graph. (TOG) 42(4), 1–12 (2023). https://doi.org/10.1145/3592415

    Article  Google Scholar 

  36. Jambon, C., Kerbl, B., Kopanas, G., Diolatzis, S., Leimkühler, T., Drettakis, G.: Nerfshop: interactive editing of neural radiance fields. Proc. ACM Comput. Graph. Interact. Tech. 6, 1:1–1:21 (2023).https://doi.org/10.1145/3585499

  37. Jiang, Y., Wang, R., Liu, Z.: A survey of cloth simulation and applications. In: 2008 9th International Conference on Computer-Aided Industrial Design and Conceptual Design, pp. 765–769. IEEE (2008)

    Google Scholar 

  38. Jiang, Y., et al.: Hifi4g: high-fidelity human performance rendering via compact gaussian splatting. arXiv preprint arXiv:2312.03461 (2023)

  39. Kato, H., Ushiku, Y., Harada, T.: Neural 3d mesh renderer. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3907–3916 (2017). https://doi.org/10.1109/CVPR.2018.00411

  40. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023)

    Google Scholar 

  41. Keselman, L., Hebert, M.: Approximate differentiable rendering with algebraic surfaces. In: European Conference on Computer Vision, pp. 596–614. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19824-3_35

  42. Komatsu, K.: Human skin model capable of natural shape variation. Vis. Comput. 3, 265–271 (1988)

    Article  Google Scholar 

  43. Korban, M., Li, X.: A survey on applications of digital human avatars toward virtual co-presence. arXiv preprint arXiv:2201.04168 (2022)

  44. Kwon, Y., Kim, D., Ceylan, D., Fuchs, H.: Neural human performer: learning generalizable radiance fields for human performance rendering. Adv. Neural. Inf. Process. Syst. 34, 24741–24752 (2021)

    Google Scholar 

  45. Kwon, Y., Kim, D., Ceylan, D., Fuchs, H.: Neural image-based avatars: generalizable radiance fields for human avatar modeling. In: The Eleventh International Conference on Learning Representations (2023)

    Google Scholar 

  46. Kwon, Y., Liu, L., Fuchs, H., Habermann, M., Theobalt, C.: Deliffas: deformable light fields for fast avatar synthesis. Adv. Neural Inf. Process. Syst. 36 (2023)

    Google Scholar 

  47. Larionov, E., Eckert, M.L., Wolff, K., Stuyck, T.: Estimating cloth elasticity parameters using position-based simulation of compliant constrained dynamics. arXiv preprint arXiv:2212.08790 (2022)

  48. Lassner, C., Zollhofer, M.: Pulsar: efficient sphere-based neural rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1440–1449 (2021)

    Google Scholar 

  49. Lee, D., Kang, H., Lee, I.K.: Clothcombo: modeling inter-cloth interaction for draping multi-layered clothes. ACM Trans. Graph. (TOG) 42(6), 1–13 (2023)

    Article  Google Scholar 

  50. LeVeque, R.J.: Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM (2007)

    Google Scholar 

  51. Li, M., et al.: Incremental potential contact: intersection-and inversion-free, large-deformation dynamics. ACM Trans. Graph. 39(4), 49 (2020)

    Article  Google Scholar 

  52. Li, M., Kaufman, D.M., Jiang, C.: Codimensional incremental potential contact. ACM Trans. Graph. (TOG) 40(4), 1–24 (2021)

    Article  Google Scholar 

  53. Li, R., et al.: Tava: template-free animatable volumetric actors (2022)

    Google Scholar 

  54. Li, X., Li, X.R., Li, Y., Feng, W.: Review of cloth modeling and simulation for virtual fitting. Text. Res. J. 93(7–8), 1699–1711 (2023)

    Article  Google Scholar 

  55. Li, Y., Chen, H.Y., Larionov, E., Sarafianos, N., Matusik, W., Stuyck, T.: Diffavatar: simulation-ready garment optimization with differentiable simulation. arXiv preprint arXiv:2311.12194 (2023)

  56. Li, Y., Du, T., Wu, K., Xu, J., Matusik, W.: Diffcloth: differentiable cloth simulation with dry frictional contact. ACM Trans. Graph. (TOG) 42(1), 1–20 (2022)

    Article  Google Scholar 

  57. Li, Z., Zheng, Z., Liu, Y., Zhou, B., Liu, Y.: Posevocab: learning joint-structured pose embeddings for human avatar modeling. arXiv preprint arXiv:2304.13006 (2023)

  58. Li, Z., Zheng, Z., Wang, L., Liu, Y.: Animatable gaussians: learning pose-dependent gaussian maps for high-fidelity human avatar modeling. arXiv preprint arXiv:2311.16096 (2023)

  59. Liu, L., Habermann, M., Rudnev, V., Sarkar, K., Gu, J., Theobalt, C.: Neural actor: neural free-view synthesis of human actors with pose control. ACM Trans. Graph. (TOG) 40(6), 1–16 (2021)

    Google Scholar 

  60. Liu, L., et al.: Neural rendering and reenactment of human actor videos. ACM Trans. Graph. (TOG) 38(5), 1–14 (2019)

    Article  Google Scholar 

  61. Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3d reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7708–7717 (2019)

    Google Scholar 

  62. Liu, T., Bouaziz, S., Kavan, L.: Quasi-newton methods for real-time simulation of hyperelastic materials. ACM Trans. Graph. (TOG) 36(3), 1–16 (2017)

    Article  Google Scholar 

  63. Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 (2019)

  64. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: a skinned multi-person linear model. ACM Trans. Graph. (TOG) 34(6) (2015)

    Google Scholar 

  65. Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3d gaussians: tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023)

  66. Ma, Q., Saito, S., Yang, J., Tang, S., Black, M.J.: Scale: modeling clothed humans with a surface codec of articulated local elements. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16082–16093 (2021)

    Google Scholar 

  67. Ma, Q., Yang, J., Black, M.J., Tang, S.: Neural point-based shape modeling of humans in challenging clothing. In: 2022 International Conference on 3D Vision (3DV), pp. 679–689. IEEE (2022)

    Google Scholar 

  68. Ma, Q., et al.: Learning to dress 3d people in generative clothing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6469–6478 (2020)

    Google Scholar 

  69. Ma, Q., Yang, J., Tang, S., Black, M.J.: The power of points for modeling humans in clothing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10974–10984 (2021)

    Google Scholar 

  70. Macklin, M., Müller, M., Chentanez, N.: Xpbd: position-based simulation of compliant constrained dynamics. In: Proceedings of the 9th International Conference on Motion in Games, MIG 2016, p. 49–54. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2994258.2994272

  71. Magnenat, T., Laperrière, R., Thalmann, D.: Joint-dependent local deformations for hand animation and object grasping. Technical report, Canadian Inf. Process. Soc (1988)

    Google Scholar 

  72. Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: archive of motion capture as surface shapes. In: ICCV (2019)

    Google Scholar 

  73. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460–4470 (2019)

    Google Scholar 

  74. Miguel, E., et al.: Data-driven estimation of cloth simulation models. In: Computer Graphics Forum, vol. 31, pp. 519–528. Wiley Online Library (2012)

    Google Scholar 

  75. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)

    Article  Google Scholar 

  76. Mixamo. www.mixamo.com

  77. Newcombe, R.A., et al.: Kinectfusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136. IEEE (2011)

    Google Scholar 

  78. Nimier-David, M., Vicini, D., Zeltner, T., Jakob, W.: Mitsuba 2: a retargetable forward and inverse renderer. ACM Trans. Graph. (TOG) 38(6), 1–17 (2019)

    Article  Google Scholar 

  79. Noguchi, A., Sun, X., Lin, S., Harada, T.: Neural articulated radiance field. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5762–5772 (2021)

    Google Scholar 

  80. Noguchi, A., Sun, X., Lin, S., Harada, T.: Unsupervised learning of efficient geometry-aware neural articulated representations. In: European Conference on Computer Vision, pp. 597–614. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19790-1_36

  81. Pan, X., et al.: Predicting loose-fitting garment deformations using bone-driven motion networks. In: ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10 (2022)

    Google Scholar 

  82. Pang, H., Zhu, H., Kortylewski, A., Theobalt, C., Habermann, M.: Ash: animatable gaussian splats for efficient and photoreal human rendering. arXiv preprint arXiv:2312.05941 (2023)

  83. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

    Google Scholar 

  84. Park, K., et al.: Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. (TOG) 40(6), 1–12 (2021)

    Article  Google Scholar 

  85. Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985 (2019)

    Google Scholar 

  86. Peng, S., et al.: Animatable neural radiance fields for modeling dynamic human bodies. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14314–14323 (2021)

    Google Scholar 

  87. Peng, S., et al.: Neural body: implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9054–9063 (2021)

    Google Scholar 

  88. Peng, T., et al.: Pgn-cloth: physics-based graph network model for 3d cloth animation. Displays 80, 102534 (2023)

    Article  Google Scholar 

  89. Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: neural radiance fields for dynamic scenes. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10313–10322 (2020). https://doi.org/10.1109/CVPR46437.2021.01018

  90. Saito, S., Yang, J., Ma, Q., Black, M.J.: Scanimate: weakly supervised learning of skinned clothed avatar networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2886–2897 (2021)

    Google Scholar 

  91. Santesteban, I., Otaduy, M.A., Casas, D.: Learning-based animation of clothing for virtual try-on. In: Computer Graphics Forum, vol. 38, pp. 355–366. Wiley Online Library (2019)

    Google Scholar 

  92. Santesteban, I., Otaduy, M.A., Casas, D.: Snug: self-supervised neural dynamic garments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8140–8150 (2022)

    Google Scholar 

  93. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)

    Google Scholar 

  94. Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., Zollhöfer, M.: Deepvoxels: learning persistent 3d feature embeddings. In: Proceedings of Computer Vision and Pattern Recognition (CVPR). IEEE (2019)

    Google Scholar 

  95. Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3d-structure-aware neural scene representations. Adv. Neural Inf. Process. Syst. (2019)

    Google Scholar 

  96. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3d. In: ACM Siggraph 2006 Papers, pp. 835–846 (2006)

    Google Scholar 

  97. Stoll, C., Gall, J., De Aguiar, E., Thrun, S., Theobalt, C.: Video-based reconstruction of animatable human characters. ACM Trans. Graph. (TOG) 29(6), 1–10 (2010)

    Article  Google Scholar 

  98. Stuyck, T., Chen, H.y.: Diffxpbd: differentiable position-based simulation of compliant constraint dynamics. Proc. ACM Comput. Graph. Interact. Tech. 6(3), 1–14 (2023)

    Google Scholar 

  99. Su, S.Y., Yu, F., Zollhöfer, M., Rhodin, H.: A-nerf: articulated neural radiance fields for learning human shape, appearance, and pose. Adv. Neural. Inf. Process. Syst. 34, 12278–12291 (2021)

    Google Scholar 

  100. Su, Z., et al.: Caphy: capturing physical properties for animatable human avatars. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14150–14160 (2023)

    Google Scholar 

  101. Sun, M., et al.: Human 3d avatar modeling with implicit neural representation: a brief survey. In: 2022 14th International Conference on Signal Processing Systems (ICSPS), pp. 818–827. IEEE (2022)

    Google Scholar 

  102. Tamstorf, R., Grinspun, E.: Discrete bending forces and their jacobians. Graph. Models 75(6), 362–370 (2013)

    Article  Google Scholar 

  103. Tang, M., Tong, R., Narain, R., Meng, C., Manocha, D.: A gpu-based streaming algorithm for high-resolution cloth simulation. In: Computer Graphics Forum, vol. 32, pp. 21–30. Wiley Online Library (2013)

    Google Scholar 

  104. Tatarchenko, M., Richter, S.R., Ranftl, R., Li, Z., Koltun, V., Brox, T.: What do single-view 3d reconstruction networks learn? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3405–3414 (2019)

    Google Scholar 

  105. Tewari, A., et al.: State of the art on neural rendering. In: Computer Graphics Forum, vol. 39 (2020). https://doi.org/10.1111/cgf.14022

  106. Tewari, A., et al.: Advances in neural rendering. In: Computer Graphics Forum, vol. 41, pp. 703–735. Wiley Online Library (2022)

    Google Scholar 

  107. Tiwari, G., Sarafianos, N., Tung, T., Pons-Moll, G.: Neural-gif: neural generalized implicit functions for animating people in clothing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11708–11718 (2021)

    Google Scholar 

  108. Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., Theobalt, C.: Non-rigid neural radiance fields: reconstruction and novel view synthesis of a dynamic scene from monocular video. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12939–12950 (2020). https://doi.org/10.1109/ICCV48922.2021.01272

  109. Vassilev, T., Spanlang, B., Chrysanthou, Y.: Fast cloth animation on walking avatars. In: Computer Graphics Forum, vol. 20, pp. 260–267. Wiley Online Library (2001)

    Google Scholar 

  110. Vidaurre, R., Santesteban, I., Garces, E., Casas, D.: Fully convolutional graph neural networks for parametric virtual try-on. In: Computer Graphics Forum, vol. 39, pp. 145–156. Wiley Online Library (2020)

    Google Scholar 

  111. Wang, H.: Gpu-based simulation of cloth wrinkles at submillimeter levels. ACM Trans. Graph. (TOG) 40(4), 1–14 (2021)

    Google Scholar 

  112. Wang, H., O’Brien, J.F., Ramamoorthi, R.: Data-driven elastic models for cloth: modeling and measurement. ACM Trans. Graph. (TOG) 30(4), 1–12 (2011)

    Google Scholar 

  113. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: learning neural implicit surfaces by volume rendering for multi-view reconstruction. Adv. Neural. Inf. Process. Syst. 34, 27171–27183 (2021)

    Google Scholar 

  114. Wang, S., Schwarz, K., Geiger, A., Tang, S.: Arah: animatable volume rendering of articulated human sdfs. In: European Conference on Computer Vision, pp. 1–19. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19824-3_1

  115. Wang, Tet al.: Rodin: a generative model for sculpting 3d digital avatars using diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4563–4573 (2023)

    Google Scholar 

  116. Wu, G., et al.: 4d gaussian splatting for real-time dynamic scene rendering. ArXiv arxiv:2310.08528 (2023)

  117. Xiang, D., et al.: Dressing avatars: deep photorealistic appearance for physically simulated clothing. ACM Trans. Graph. (TOG) 41(6), 1–15 (2022)

    Article  Google Scholar 

  118. Xu, Y., Yifan, W., Bergman, A.W., Chai, M., Zhou, B., Wetzstein, G.: Efficient 3d articulated human generation with layered surface volumes. arXiv preprint arXiv:2307.05462 (2023)

  119. Xu, Y., et al.: Gaussian head avatar: ultra high-fidelity head avatar via dynamic gaussians. arXiv preprint arXiv:2312.03029 (2023)

  120. Yang, Z., Zeng, A., Yuan, C., Li, Y.: Effective whole-body pose estimation with two-stages distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210–4220 (2023)

    Google Scholar 

  121. Yariv, L., Gu, J., Kasten, Y., Lipman, Y.: Volume rendering of neural implicit surfaces. Adv. Neural. Inf. Process. Syst. 34, 4805–4815 (2021)

    Google Scholar 

  122. Yee, K.S., Chen, J.S.: The finite-difference time-domain (fdtd) and the finite-volume time-domain (fvtd) methods in solving maxwell’s equations. IEEE Trans. Antennas Propag. 45(3), 354–363 (1997)

    Article  Google Scholar 

  123. Yifan, W., Serena, F., Wu, S., Öztireli, C., Sorkine-Hornung, O.: Differentiable surface splatting for point-based geometry processing. ACM Trans. Graph. (TOG) 38(6), 1–14 (2019)

    Article  Google Scholar 

  124. Yu, T., et al.: Simulcap: single-view human performance capture with cloth simulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5504–5514 (2019)

    Google Scholar 

  125. Yuan, Y.J., Sun, Y.T., Lai, Y.K., Ma, Y., Jia, R., Gao, L.: Nerf-editing: geometry editing of neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18353–18364 (2022)

    Google Scholar 

  126. Zeller, C.: Cloth simulation on the gpu. In: ACM SIGGRAPH 2005 Sketches, pp. 39–es (2005)

    Google Scholar 

  127. Zhang, J.X., Lin, G.W.C., Bode, L., Chen, H.Y., Stuyck, T., Larionov, E.: Estimating cloth elasticity parameters from homogenized yarn-level models. arXiv preprint arXiv:2401.15169 (2024)

  128. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

  129. Zhao, F., et al.: Human performance modeling and rendering via neural animated mesh. ACM Trans. Graph. (TOG) 41(6), 1–17 (2022)

    Article  Google Scholar 

  130. Zheng, Z., Huang, H., Yu, T., Zhang, H., Guo, Y., Liu, Y.: Structured local radiance fields for human avatar modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15893–15903 (2022)

    Google Scholar 

  131. Zielonka, W., Bagautdinov, T., Saito, S., Zollhöfer, M., Thies, J., Romero, J.: Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.08581 (2023)

Download references

Acknowledgement

We would like to thank Jiayi Eris Zhang for the discussions. This material is based on work that is partially funded by an unrestricted gift from Google, Samsung, an SNF Postdoc Mobility fellowship, ARL grant W911NF-21-2-0104, and a Vannevar Bush Faculty Fellowship.

Author information

Authors and Affiliations

  1. Stanford University, Stanford, USA

    Yang Zheng, Qingqing Zhao, Guandao Yang, Wang Yifan, Leonidas Guibas & Gordon Wetzstein

  2. Carnegie Mellon University, Pittsburgh, USA

    Donglai Xiang

  3. Google, Mountain View, USA

    Florian Dubost, Dmitry Lagun, Thabo Beeler & Federico Tombari

  4. Technical University of Munich, Munich, Germany

    Federico Tombari

Authors
  1. Yang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingqing Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guandao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wang Yifan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Donglai Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Florian Dubost
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dmitry Lagun
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Thabo Beeler
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Federico Tombari
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Leonidas Guibas
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Gordon Wetzstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Zheng .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 952 KB)

Supplementary material 2 (mp4 94068 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zheng, Y. et al. (2025). PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15095. Springer, Cham. https://doi.org/10.1007/978-3-031-72913-3_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72913-3_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72912-6

  • Online ISBN: 978-3-031-72913-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation