Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

3DEgo: 3D Editing on the Go!

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15088))

Included in the following conference series:

  • 92 Accesses

Abstract

We introduce 3DEgo to address a novel problem of directly synthesizing photorealistic 3D scenes from monocular videos guided by textual prompts. Conventional methods construct a text-conditioned 3D scene through a three-stage process, involving pose estimation using Structure-from-Motion (SfM) libraries like COLMAP, initializing the 3D model with unedited images, and iteratively updating the dataset with edited images to achieve a 3D scene with text fidelity. Our framework streamlines the conventional multi-stage 3D editing process into a single-stage workflow by overcoming the reliance on COLMAP and eliminating the cost of model initialization. We apply a diffusion model to edit video frames prior to 3D scene creation by incorporating our designed noise blender module for enhancing multi-view editing consistency, a step that does not require additional training or fine-tuning of T2I diffusion models. 3DEgo utilizes 3D Gaussian Splatting to create 3D scenes from the multi-view consistent edited frames, capitalizing on the inherent temporal continuity and explicit point cloud data. 3DEgo demonstrates remarkable editing precision, speed, and adaptability across a variety of video sources, as validated by extensive evaluations on six datasets, including our own prepared GS25 dataset. Project Page: https://3dego.github.io/.

U. Khalid and H. Iqbal–Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bao, C., et al.: SINE: semantic-driven image-based nerf editing with prior-guided editing field. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20919–20929 (2023)

    Google Scholar 

  2. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470–5479 (2022)

    Google Scholar 

  3. Bian, W., Wang, Z., Li, K., Bian, J.W., Prisacariu, V.A.: Nope-NeRF: optimising neural radiance field with no pose prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4160–4169 (2023)

    Google Scholar 

  4. Brooks, T., Holynski, A., Efros, A.A.: InstructPix2Pix: learning to follow image editing instructions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392–18402 (2023)

    Google Scholar 

  5. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)

    Google Scholar 

  6. Chiang, P.Z., Tsai, M.S., Tseng, H.Y., Lai, W.S., Chiu, W.C.: Stylizing 3D scene via implicit representation and hypernetwork. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1475–1484 (2022)

    Google Scholar 

  7. Dong, J., Wang, Y.X.: ViCA-NeRF: view-consistency-aware 3D editing of neural radiance fields. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  8. Fu, Y., Liu, S., Kulkarni, A., Kautz, J., Efros, A.A., Wang, X.: Colmap-free 3D gaussian splatting (2023). https://arxiv.org/abs/2312.07504

  9. Gal, R., Patashnik, O., Maron, H., Bermano, A.H., Chechik, G., Cohen-Or, D.: StyleGAN-NADA: CLIP-guided domain adaptation of image generators. ACM Trans. Graph. 41(4), 1–13 (2022). https://doi.org/10.1145/3528223.3530164

    Article  Google Scholar 

  10. Gao, W., Aigerman, N., Groueix, T., Kim, V.G., Hanocka, R.: TextDeformer: geometry manipulation using text guidance. arXiv preprint arXiv:2304.13348 (2023)

  11. Haque, A., Tancik, M., Efros, A.A., Holynski, A., Kanazawa, A.: Instruct-NeRF2NeRF: editing 3D scenes with instructions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19740–19750 (2023)

    Google Scholar 

  12. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)

  13. Hong, F., Zhang, M., Pan, L., Cai, Z., Yang, L., Liu, Z.: AvatarCLIP: zero-shot text-driven generation and animation of 3D avatars. ACM Trans. Graph. (TOG) 41(4), 1–19 (2022)

    Article  Google Scholar 

  14. Huang, Y.H., He, Y., Yuan, Y.J., Lai, Y.K., Gao, L.: StylizedNeRF: consistent 3D scene stylization as stylized nerf via 2D-3D mutual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18342–18352 (2022)

    Google Scholar 

  15. Jeong, Y., Ahn, S., Choy, C., Anandkumar, A., Cho, M., Park, J.: Self-calibrating neural radiance fields. In: ICCV (2021)

    Google Scholar 

  16. Karim, N., Khalid, U., Iqbal, H., Hua, J., Chen, C.: Free-editor: zero-shot text-driven 3D scene editing. arXiv preprint arXiv:2312.13663 (2023)

  17. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. (ToG) 42(4), 1–14 (2023)

    Article  Google Scholar 

  18. Khalid, U., Iqbal, H., Karim, N., Hua, J., Chen, C.: LatentEditor: text driven local editing of 3D scenes. arXiv preprint arXiv:2312.09313 (2023)

  19. Kim, S., Lee, K., Choi, J.S., Jeong, J., Sohn, K., Shin, J.: Collaborative score distillation for consistent visual editing. In: Thirty-seventh Conference on Neural Information Processing Systems (2023). https://openreview.net/forum?id=0tEjORCGFD

  20. Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)

  21. Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and Temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (2017)

    Google Scholar 

  22. Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing nerf for editing via feature field distillation. arXiv preprint arXiv:2205.15585 (2022)

  23. Li, Y., Lin, Z.H., Forsyth, D., Huang, J.B., Wang, S.: ClimateNeRF: physically-based neural rendering for extreme climate synthesis. arXiv e-prints pp. arXiv–2211 (2022)

    Google Scholar 

  24. Li, Y., et al.: FocalDreamer: text-driven 3D editing via focal-fusion assembly. arXiv preprint arXiv:2308.10608 (2023)

  25. Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: BARF: bundle-adjusting neural radiance fields. In: ICCV (2021)

    Google Scholar 

  26. Liu, H.K., Shen, I., Chen, B.Y., et al.: NeRF-in: free-form nerf inpainting with RGB-D priors. arXiv preprint arXiv:2206.04901 (2022)

  27. Long, X., et al.: Wonder3D: single image to 3D using cross-domain diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9970–9980 (2024)

    Google Scholar 

  28. Michel, O., Bar-On, R., Liu, R., et al.: Text2Mesh: text-driven neural stylization for meshes. In: CVPR 2022, pp. 13492–13502 (2022)

    Google Scholar 

  29. Nguyen-Phuoc, T., Liu, F., Xiao, L.: SNERF: stylized neural implicit representations for 3D scenes. arXiv preprint arXiv:2207.02363 (2022)

  30. Nichol, A., et al.: GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)

  31. Noguchi, A., Sun, X., Lin, S., Harada, T.: Neural articulated radiance field. In: ICCV 2021, pp. 5762–5772 (2021)

    Google Scholar 

  32. Park, H.S., Jun, C.H.: A simple and fast algorithm for K-medoids clustering. Expert Syst. Appl. 36(2), 3336–3341 (2009)

    Article  Google Scholar 

  33. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  34. Rajič, F., Ke, L., Tai, Y.W., Tang, C.K., Danelljan, M., Yu, F.: Segment anything meets point tracking. arXiv preprint arXiv:2307.01197 (2023)

  35. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 (2022)

  36. Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., Novotny, D.: Common objects in 3D: large-scale learning and evaluation of real-life 3D category reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901–10911 (2021)

    Google Scholar 

  37. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR 2022, pp. 10684–10695 (2022)

    Google Scholar 

  38. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500–22510 (2023)

    Google Scholar 

  39. Saharia, C., Chan, W., Saxena, S.E.A.: Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS 2022 35, 36479–36494 (2022)

    Google Scholar 

  40. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)

    Google Scholar 

  41. Suvorov, R., et al.: Resolution-robust large mask inpainting with fourier convolutions. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2149–2159 (2022)

    Google Scholar 

  42. Tancik, M., et al.: Nerfstudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–12 (2023)

    Google Scholar 

  43. Tschernezki, V., Laina, I., Larlus, D., Vedaldi, A.: Neural feature fusion fields: 3D distillation of self-supervised 2D image representations. In: 2022 International Conference on 3D Vision (3DV), pp. 443–453. IEEE (2022)

    Google Scholar 

  44. Wang, C., Chai, M., He, M., et al.: CLIP-NeRF: text-and-image driven manipulation of neural radiance fields. In: CVPR 2022, pp. 3835–3844 (2022)

    Google Scholar 

  45. Wang, C., Jiang, R., Chai, M., He, M., Chen, D., Liao, J.: NeRF-Art: text-driven neural radiance fields stylization. IEEE Trans. Vis. Comput. Graph. (2023)

    Google Scholar 

  46. Weng, H., et al.: Consistent123: improve consistency for one image to 3D object synthesis. arXiv preprint arXiv:2310.08092 (2023)

  47. Wu, Q., Tan, J., Xu, K.: PaletteNeRF: palette-based color editing for NeRFs. arXiv preprint arXiv:2212.12871 (2022)

  48. Xu, T., Harada, T.: Deforming radiance fields with cages. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pp. 159–175. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_10

    Chapter  Google Scholar 

  49. Yang, B., et al.: NeuMesh: learning disentangled neural mesh-based implicit field for geometry and texture editing. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI, pp. 597–614. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19787-1_34

    Chapter  Google Scholar 

  50. Ye, M., Danelljan, M., Yu, F., Ke, L.: Gaussian Grouping: segment and edit anything in 3D scenes. arXiv preprint arXiv:2312.00732 (2023)

  51. Zhang, K., et al.: ARF: artistic radiance fields. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXI, pp. 717–733. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19821-2_41

    Chapter  Google Scholar 

  52. Zhuang, J., Wang, C., Lin, L., Liu, L., Li, G.: DreamEditor: text-driven 3D scene editing with neural fields. In: SIGGRAPH Asia 2023 Conference Papers, pp. 1–10 (2023)

    Google Scholar 

Download references

Acknowledgement

This work was partially supported by the NSF under Grant Numbers OAC-1910469 and OAC-2311245.

Author information

Authors and Affiliations

  1. University of Central Florida, Orlando, FL, USA

    Umar Khalid & Chen Chen

  2. Department of Computer Science, Wayne State University, Detroit, MI, USA

    Hasan Iqbal & Jing Hua

  3. Department of Computer Science and Software Engineering, Miami University, Oxford, OH, USA

    Azib Farooq

Authors
  1. Umar Khalid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hasan Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Azib Farooq
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Umar Khalid .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 405 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Khalid, U., Iqbal, H., Farooq, A., Hua, J., Chen, C. (2025). 3DEgo: 3D Editing on the Go!. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15088. Springer, Cham. https://doi.org/10.1007/978-3-031-73404-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73404-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73403-8

  • Online ISBN: 978-3-031-73404-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation