JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillation

As discussed above, the rendering distributions of the 3D representation should maintain 3D consistency across views $\tilde{\textbf{x}}=\{\textbf{x}^{1},\textbf{x}^{2},\ldots,\textbf{x}^{V}\}$ with respect to different poses $\tilde{\bf c}=\{c^{1},c^{2},\ldots,c^{V}\}$. However, for 2D pre-trained diffusion models, different views are generated independently. To ensure consistency, we propose modeling the joint image distribution of multiple views, denoted as $p_{0}(\tilde{\textbf{x}}|\tilde{\textbf{c}},y)$, within the diffusion model. Following the commonly adopted assumption in energy-based distribution modeling [44, 21, 45], we introduce an energy function that measures the inter-view coherence by $\mathcal{C}(\tilde{\textbf{x}},\tilde{\textbf{c}}):\mathbb{R}^{Vd}\to\mathbb{R}$ and define:

where a larger $\mathcal{C}(\tilde{\textbf{x}},\tilde{\textbf{c}})$ indicates greater coherence among the denoised view images. As a result, the joint image distribution is no longer view-independent. In practice, the joint energy function can be implemented via various view-aware models, as long as they can reflect the coherence across multiple views. In Sec. 4.2, we explore different choices in depth. The modeling of coherence across joint image distributions facilitates our JSD on multi-view, integrating inter-view coherence to ensure consistent 3D representation.

We extend the single-view KL-divergence in SDS to a multi-view version, based on the joint image distribution:

where the extra energy term $\mathcal{C}(\tilde{\bf x},\tilde{\bf c})$ in Eq. (3) accounts for the inter-view coherence. Without this constraint, e.g., $\mathcal{C}(\tilde{\bf x},\tilde{\bf c})\equiv 0$, different rendering views are optimized independently with the 2D diffusion model separately. In this sense, the original SDS can be seen as a special case of JSD.

To correspond to the gradient of the new rule of multi-view KL-divergence in Eq. (4), we derive our score distillation function that is jointly conducted on multiple views as follows:

where $\{\epsilon^{i}\}_{i=1}^{V}$ are noises during score matching for different views. The proof can be found in the Appendix. To intuitively compare SDS with JSD, we sample multi-view images from the forward pass of pre-trained diffusion model [38] as shown in Fig. 2(c). For each view $\textbf{x}_{t}$, we randomly select $\textbf{x}_{t}^{\prime}$ from a different view and adopt the binary classification model presented in Sec. 4.2 as the energy function to measure coherence across the two views. The results illustrate that JSD significantly enhances the correspondence with directional prompts and consistency across different views, particularly for the initially biased views such as side and back views. These theoretical and empirical results prove that the multi-face Janus issue in SDS is rooted in the view-agnostic and view-biased 2D image distribution, a challenge effectively mitigated by JSD.

The binary classification model $M_{\text{CLS}}$ is designed to classify the content consistency between two input images based on their relative camera pose. To ensure computational efficiency, we introduce a dedicated classification model, as shown in Fig. 4 (a). The training process takes two days with a single A800 GPU on Objaverse dataset [8]. Further details can be found in the appendix. The classification model $M_{\text{CLS}}$ processes pairs of images $\textbf{x}^{i}$ and $\textbf{x}^{j}$ captured from different viewpoints $c^{i}$ and $c^{j}$. It extracts image features using the DINO-ViT/s16 backbone [3]. These image features are conditioned on the camera feature obtained through MLP layers from the relative camera transformation matrix $\Delta(c^{j},c^{i})$. Finally, the classification head produces the binary score. To incorporate with JSD, we only consider neighboring views in $V$ as image pairs for coherence measurement, which is denoted as:

where the higher logit indicates the stronger geometric consistency.

The image-to-image translation model $M_{\text{I2I}}$ is tailored for novel view synthesis [26, 29, 40, 27]. We employ the most recent model, Wonder3D [29] as $M_{\text{I2I}}$, which is a viewpoint-conditioned image translation model and generates consistent content in the target viewpoint. When integrated with JSD, a random reference view $\textbf{x}^{\textbf{ref}}$ is selected from the set of 3D rendered images. Then we input the relative camera transformation $\Delta(c^{i},c^{\textbf{ref}})$ and rendered images to $M_{\text{I2I}}$. The measure of consistency is determined by calculating the reconstruction loss between the synthesized new image and its corresponding rendered image:

where a smaller reconstruction loss indicates stronger geometric consistency under the estimation of $M_{\text{I2I}}$.

The multi-view synthesis model is designed to generate multiple images conditioned on text prompts and camera poses, wherein we employ very recent work MVDream [41] as $M_{\textbf{MVS}}$. We compute the reconstruction loss for multiple generative views and rendered views:

The smaller reconstruction loss signifies better geometric consistency within $M_{\textbf{MVS}}$. These view-aware models measure the coherence across views according to their own 3D-aware insights. Incorporated with JSD, they provide distinct constraints, resulting in different 3D generations that nonetheless contribute to enhanced geometric consistency. We believe that JSD can be adapted to more universal view-aware models, which enables us to progressively redefine the benchmark of 3D generation with the advancement of multi-view tasks.

We aim to shift attention between geometric structure and texture details during optimization. Specifically, starting from iteration 5K, we reduce the learning rate of the density network of NeRF from $1e-2$ to $1e-6$ and set orientation loss to 0. Consequently, it benefits geometric convergence in the early phase of optimization, while allowing for decreased attention on geometry and increased attention on texture enhancement in the later stages.

CFG scheduling strategies have been employed in 2D domain [39, 15] to enhance quality. In this work, we propose to modify the CFG scale $s$ during the training for 3D generation. We are motivated by the observation that a large CFG scale can lead to accelerated geometric convergence but may result in under-optimized geometry and distorted texture. Unlike the annealed CFG approach in CLIP-Sculptor [39], our increasing strategy prioritizes texture while maintaining accurate geometry. Specifically, a smaller $s=30$ is employed in the early stages to preserve shape integrity, which allows for stronger coherence guidance from JSD. After 5K iterations, we increase the $s$ to 50, enhancing texture fidelity and overall quality.

We compare with several representative baselines on threestudio [11] project, which modularizes the text-to-3D framework allowing for fair comparisons by ablating individual components. The generative samples are shown in Fig. 5. (i) DreamFusion [37]: Compared to DreamFusion which utilizes traditional SDS optimization, JointDreamer significantly improves the Multi-Face Janus issues in the generations of DreamFusion by introducing an energy term to ensure inter-view coherence. (ii) Magic3D [24]: Magic3D introduces a two-stage generation pipeline, transferring NeRF to DMTet in the second stage to enhance generation quality. We also transfer NeRF to DMTet as JointDreamer-DMTet, showcasing consistent superiority in geometry and texture quality by utilizing JSD optimization instead of SDS. (iii) ProlificDreamer [42]: ProlificDreamer presents variational score distillation (VSD) as a variant of score distillation function. While VSD enhances photorealism in 3D renderings by introducing a LoRA model, the ill-posed association of pose and images during LoRA training deepens the geometric inconsistency of 3D representation, resulting in severe multi-view artifacts in generations. In contrast, JointDreamer with JSD achieves geometric consistency while maintaining high-fidelity texture quality. (iv) MVDream [41]: Compared to direct distillation from a finetuned model as MVDream, JointDreamer employs view-aware models as the coherence constraint in JSD, while still inherit the generalization capabilities of original diffusion models [38]. The results indicate that JointDreamer mitigates the overfitting issues of MVDream, which accurately responds to complex input text and enhances the 3D consistency of generations.

Following  [37, 18], we evaluate CLIP Score [14], CLIP R-Precision [36] and user preference on the object-centric caption subset of MS-COCO [25] with 153 prompts to measure the text congruence and generative quality. For computational efficiency, we generate each 3D asset using 5K iterations of 64$\times$64 rendered images and render 20 images per caption for evaluation. CLIP ViT-B/32 is adopted as the feature extractor for Clip Score and Clip R-Precision. As illustrated by the results in Table 5.1, JointDreamer can outperform all baselines on CLIP Score and CLIP R-Precision by large margins. Specifically, JointDreamer achieves an improvement of the R-Precision by 60.8% and 54.9% over DreamFusion and MVDream, demonstrating its superior corresponding to textual description. Notably, the severe Janus artifacts in ProlificDreamer compromise the quality of rendering with noisy background and semantic distortion, resulting in the lowest R-Precision. We also conduct a user study about shape preference on these prompts in Table 5.1.

Sec. 4.2 discusses the varying inter-view

coherency measurements provided by view-aware models trained. When incorporated with JSD, these models have distinct impacts on 3D generations, as shown in Fig. 6. The binary classifier effectively corrects inaccurate geometric structures in SDS. However, it cannot introduce additional imaginative elements as a discriminative model, resulting in oversaturated and monotonous textures. In contrast, as generative models, Wonder3D [29] and MVDream [41] employ reconstruction loss to estimate 3D consistency. Hence, they not only guide geometric structural modifications but also influence texture quality. we further conduct a quantitative comparison using 16 complex multi-Janus prompts, as outlined in Table 3. The experimental setup details can be found in the Appendix. The results indicate that our JSD consistently mitigates Janus artifacts across different view-aware models, with only a slight increase in computational requirements compared to SDS. We find that the energy function $\mathcal{C}_{\text{I2I}}$ derived from the image-to-image translation model exhibits poor performance, likely attributed to a mismatch in camera range and inaccurate translated results.

We conduct incremental ablations on our proposed techniques in JointDreamer, including the Geometry Fading scheme and CFG Switching strategy. As shown in Fig. 7, increasing the CFG value enhances texture detail compared to the original JSD, but over-optimizes the geometry, resulting in a bumpier shape. Compared to “+CFG Switching”, the Geometry fading effectively protects shape when larger CFG guidance. We also conduct a quantitative evaluation on a 30% MS-COCO subset in Table 5.1. JSD demonstrates superior texture quality compared to SDS, as evidenced by the Clip Score (CS) and FID metrics. The two proposed mechanisms further enhance the texture quality.

To make further comparisons with JSD

and SDS, we aggregate training losses from multiple prompts on two optimization functions and visualize the training loss curve as illustrated in Fig. 8. We observe that SDS experiences significant fluctuations due to the randomness of single-view optimization. In contrast, JSD can converge gradually and smoothly, demonstrating the introduction of multi-view optimization with inter-view coherence in JSD can reduce the randomness of optimization and contribute to better convergence for 3D representation.

To show the impact of seed and prompt, we modify seeds and key prompt components in Fig. 7, such as subject, object and verb. Results in Fig. 9 demonstrate the robustness of JointDreamer for different seed. Note that the default seed is 0.

The classification model surpasses MVDream in training speed by a factor of 48. Fig.4 (b) provides additional high-quality results obtained using JSD in conjunction with the binary classification model $M_{\text{CLS}}$. Furthermore, we conduct an ablation study by replacing $\frac{\partial\mathcal{C}(\tilde{\textbf{x}})}{\partial\textbf{x}_{t}^{i}}$ in Eq.5 with a randomly generated value between $[0,1]$. However, it yields shapeless results due to unrelated disturbances in the optimization process. These findings highlight the significance of the proposed classification model in achieving a balance between computational efficiency and generation quality.

View-aware models can be incorporated into 3D generation by combining them with JSD or using them directly with SDS. In this discussion, we use MVDream [41] as an example to demonstrate the superiority of JSD. MVDream can generate consistent shapes when calculating the SDS loss directly. However, it may miss components in the complete input text due to its fine-tuning on limited 3D data, as illustrated in Fig. 5. To address this limitation, a straightforward approach is to “SDS+MVDream”.

However, balancing the impact of SDS and MVDream is challenging. As shown in Fig. 10, setting $\lambda_{\textbf{SDS}}$ to 0 degrades to MVDream alone, while setting $\lambda_{\textbf{MVDream}}$ to 0 yields a combination similar to DreamFusion [37]. Achieving a balanced impact between SDS and MVDream through a simple combination is difficult. When $\lambda_{\textbf{MVDream}}$ is large, textual consistency remains constrained, whereas decreasing $\lambda_{\textbf{MVDream}}$ leads to the Janus problem. This is due to gradient misalignment across multiple views in the “SDS+MVDream” combination, as SDS lacks multi-view information and cannot derive our objective outlined in Eq. 5. In contrast, JSD based on the joint image distribution provides supervision for text consistency and high-fidelity texture. JSD also promotes inter-view consistency by introducing an energy term with the view-aware model.