ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images
Authors:
Xiaoshuai Zhang,
Zhicheng Wang,
Howard Zhou,
Soham Ghosh,
Danushen Gnanapragasam,
Varun Jampani,
Hao Su,
Leonidas Guibas
Abstract:
To advance the state of the art in the creation of 3D foundation models, this paper introduces the ConDense framework for 3D pre-training utilizing existing pre-trained 2D networks and large-scale multi-view datasets. We propose a novel 2D-3D joint training scheme to extract co-embedded 2D and 3D features in an end-to-end pipeline, where 2D-3D feature consistency is enforced through a volume rende…
▽ More
To advance the state of the art in the creation of 3D foundation models, this paper introduces the ConDense framework for 3D pre-training utilizing existing pre-trained 2D networks and large-scale multi-view datasets. We propose a novel 2D-3D joint training scheme to extract co-embedded 2D and 3D features in an end-to-end pipeline, where 2D-3D feature consistency is enforced through a volume rendering NeRF-like ray marching process. Using dense per pixel features we are able to 1) directly distill the learned priors from 2D models to 3D models and create useful 3D backbones, 2) extract more consistent and less noisy 2D features, 3) formulate a consistent embedding space where 2D, 3D, and other modalities of data (e.g., natural language prompts) can be jointly queried. Furthermore, besides dense features, ConDense can be trained to extract sparse features (e.g., key points), also with 2D-3D consistency -- condensing 3D NeRF representations into compact sets of decorated key points. We demonstrate that our pre-trained model provides good initialization for various 3D tasks including 3D classification and segmentation, outperforming other 3D pre-training methods by a significant margin. It also enables, by exploiting our sparse features, additional useful downstream tasks, such as matching 2D images to 3D scenes, detecting duplicate 3D scenes, and querying a repository of 3D scenes through natural language -- all quite efficiently and without any per-scene fine-tuning.
△ Less
Submitted 30 August, 2024;
originally announced August 2024.
Kubric: A scalable dataset generator
Authors:
Klaus Greff,
Francois Belletti,
Lucas Beyer,
Carl Doersch,
Yilun Du,
Daniel Duckworth,
David J. Fleet,
Dan Gnanapragasam,
Florian Golemo,
Charles Herrmann,
Thomas Kipf,
Abhijit Kundu,
Dmitry Lagun,
Issam Laradji,
Hsueh-Ti,
Liu,
Henning Meyer,
Yishu Miao,
Derek Nowrouzezahrai,
Cengiz Oztireli,
Etienne Pot,
Noha Radwan,
Daniel Rebain,
Sara Sabour,
Mehdi S. M. Sajjadi
, et al. (10 additional authors not shown)
Abstract:
Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential…
▽ More
Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification.
△ Less
Submitted 7 March, 2022;
originally announced March 2022.
