title: X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap
    Between Text-to-2D and Text-to-3D Generation
summary: In recent times, automatic text-to-3D content creation has made significant
  progress, driven by the development of pretrained 2D diffusion models. Existing
  text-to-3D methods typically optimize the 3D representation to ensure that the
  rendered image aligns well with the given text, as evaluated by the pretrained
  2D diffusion model. Nevertheless, a substantial domain gap exists between 2D
  images and 3D assets, primarily attributed to variations in camera-related
  attributes and the exclusive presence of foreground objects. Consequently,
  employing 2D diffusion models directly for optimizing 3D representations may
  lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a
  novel approach for high-quality text-to-3D content creation that effectively
  bridges the gap between text-to-2D and text-to-3D synthesis. The key components
  of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation
  (CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically
  incorporates camera information into the pretrained diffusion models by
  employing camera-dependent generation for trainable parameters. This
  integration enhances the alignment between the generated 3D assets and the
  camera's perspective. AMA loss guides the attention map of the pretrained
  diffusion model using the binary mask of the 3D object, prioritizing the
  creation of the foreground object. This module ensures that the model focuses
  on generating accurate and detailed foreground objects. Extensive evaluations
  demonstrate the effectiveness of our proposed method compared to existing
  text-to-3D approaches. Our project webpage:
  https://xmuxiaoma666.github.io/Projects/X-Dreamer .
paper_authors._id: 656d3a397825b3101031e7da
paper_authors.name: Yiwei Ma
paper_authors._id: 656d3a397825b3101031e7db
paper_authors.name: Yijun Fan
paper_authors._id: 656d3a397825b3101031e7dc
paper_authors.name: Jiayi Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:56:07.285Z
paper_authors._id: 61c7cb3fc4157c1d283da869
paper_authors.avatarUrl: /avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
paper_authors.fullname: Jiayi Ji
paper_authors.type: user
paper_authors.user: amy1254
paper_authors._id: 656d3a397825b3101031e7dd
paper_authors.name: Haowei Wang
paper_authors._id: 656d3a397825b3101031e7de
paper_authors.name: Xiaoshuai Sun
paper_authors._id: 656d3a397825b3101031e7df
paper_authors.name: Guannan Jiang
paper_authors._id: 656d3a397825b3101031e7e0
paper_authors.name: Annan Shu
paper_authors._id: 656d3a397825b3101031e7e1
paper_authors.name: Rongrong Ji
paper_title: X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap
    Between Text-to-2D and Text-to-3D Generation
paper_summary: In recent times, automatic text-to-3D content creation has made significant
  progress, driven by the development of pretrained 2D diffusion models. Existing
  text-to-3D methods typically optimize the 3D representation to ensure that the
  rendered image aligns well with the given text, as evaluated by the pretrained
  2D diffusion model. Nevertheless, a substantial domain gap exists between 2D
  images and 3D assets, primarily attributed to variations in camera-related
  attributes and the exclusive presence of foreground objects. Consequently,
  employing 2D diffusion models directly for optimizing 3D representations may
  lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a
  novel approach for high-quality text-to-3D content creation that effectively
  bridges the gap between text-to-2D and text-to-3D synthesis. The key components
  of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation
  (CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically
  incorporates camera information into the pretrained diffusion models by
  employing camera-dependent generation for trainable parameters. This
  integration enhances the alignment between the generated 3D assets and the
  camera's perspective. AMA loss guides the attention map of the pretrained
  diffusion model using the binary mask of the 3D object, prioritizing the
  creation of the foreground object. This module ensures that the model focuses
  on generating accurate and detailed foreground objects. Extensive evaluations
  demonstrate the effectiveness of our proposed method compared to existing
  text-to-3D approaches. Our project webpage:
  https://xmuxiaoma666.github.io/Projects/X-Dreamer .
paper_ai_keywords: diffusion models text-to-3D content creation synthesis 3D representation rendered image Camera-Guided Low-Rank Adaptation (CG-LoRA) Attention-Mask Alignment (AMA) Loss camera-dependent generation foreground object binary mask