Jul 23, 2022 · Abstract:We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an ...

We apply our approach to two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language ...

We propose Semantic Abstraction (SemAbs), a framework for tackling visual-semantic reasoning in open-world 3D scene understanding tasks using 2D VLMs. We ...

This work proposes Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, ...

Sep 10, 2022 · We proposed Semantic Abstraction, a framework that equips 2D VLMs with 3D spatial capabilities for open-world 3D scene understanding tasks.

Our approach, Semantic Abstraction, unlocks 2D VLM's capabilities to 3D scene understanding. Trained with a limited synthetic dataset, our model generalizes ...

1 Appendix. 1.1 Relevancy as VLM's confidence. In our experiments, we have observed that directly using VLM relevancy maps works better than binarizing.

Jul 23, 2022 · This paper focuses on the task of semantic instance completion: from an incomplete, RGB-D scan of a scene, we aim to detect the individual ...

Current 3D scene understanding models are largely limited to low-level recognition tasks such as object detection or semantic segmentation, and do not ...