X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap
  Between Text-to-2D and Text-to-3D Generation
In recent times, automatic text-to-3D content creation has made significant
progress, driven by the development of pretrained 2D diffusion models. Existing
text-to-3D methods typically optimize the 3D representation to ensure that the
rendered image aligns well with the given text, as evaluated by the pretrained
2D diffusion model. Nevertheless, a substantial domain gap exists between 2D
images and 3D assets, primarily attributed to variations in camera-related
attributes and the exclusive presence of foreground objects. Consequently,
employing 2D diffusion models directly for optimizing 3D representations may
lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a
novel approach for high-quality text-to-3D content creation that effectively
bridges the gap between text-to-2D and text-to-3D synthesis. The key components
of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation
(CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically
incorporates camera information into the pretrained diffusion models by
employing camera-dependent generation for trainable parameters. This
integration enhances the alignment between the generated 3D assets and the
camera's perspective. AMA loss guides the attention map of the pretrained
diffusion model using the binary mask of the 3D object, prioritizing the
creation of the foreground object. This module ensures that the model focuses
on generating accurate and detailed foreground objects. Extensive evaluations
demonstrate the effectiveness of our proposed method compared to existing
text-to-3D approaches. Our project webpage:
https://xmuxiaoma666.github.io/Projects/X-Dreamer .
656d3a397825b3101031e7da
Yiwei Ma
656d3a397825b3101031e7db
Yijun Fan
656d3a397825b3101031e7dc
Jiayi Ji
admin_assigned
2023-12-04T10:56:07.285Z
61c7cb3fc4157c1d283da869
/avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
Jiayi Ji
user
amy1254
656d3a397825b3101031e7dd
Haowei Wang
656d3a397825b3101031e7de
Xiaoshuai Sun
656d3a397825b3101031e7df
Guannan Jiang
656d3a397825b3101031e7e0
Annan Shu
656d3a397825b3101031e7e1
Rongrong Ji
X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap
  Between Text-to-2D and Text-to-3D Generation
In recent times, automatic text-to-3D content creation has made significant
progress, driven by the development of pretrained 2D diffusion models. Existing
text-to-3D methods typically optimize the 3D representation to ensure that the
rendered image aligns well with the given text, as evaluated by the pretrained
2D diffusion model. Nevertheless, a substantial domain gap exists between 2D
images and 3D assets, primarily attributed to variations in camera-related
attributes and the exclusive presence of foreground objects. Consequently,
employing 2D diffusion models directly for optimizing 3D representations may
lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a
novel approach for high-quality text-to-3D content creation that effectively
bridges the gap between text-to-2D and text-to-3D synthesis. The key components
of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation
(CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically
incorporates camera information into the pretrained diffusion models by
employing camera-dependent generation for trainable parameters. This
integration enhances the alignment between the generated 3D assets and the
camera's perspective. AMA loss guides the attention map of the pretrained
diffusion model using the binary mask of the 3D object, prioritizing the
creation of the foreground object. This module ensures that the model focuses
on generating accurate and detailed foreground objects. Extensive evaluations
demonstrate the effectiveness of our proposed method compared to existing
text-to-3D approaches. Our project webpage:
https://xmuxiaoma666.github.io/Projects/X-Dreamer .
X-Dreamer addresses the domain gap between 2D and 3D representations in text-to-3D generation by integrating camera information and foreground object focus using Camera-Guided Low-Rank Adaptation and Attention-Mask Alignment Loss.
pretrained 2D diffusion models Camera-Guided Low-Rank Adaptation Attention-Mask Alignment camera information foreground object focus