title: Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
summary: Encouraged by the growing availability of pre-trained 2D diffusion models,
  image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
  making remarkable progress. Most existing methods combine novel-view lifting
  from 2D diffusion models which usually take the reference image as a condition
  while applying hard L2 image supervision at the reference view. Yet heavily
  adhering to the image is prone to corrupting the inductive knowledge of the 2D
  diffusion model leading to flat or distorted 3D generation frequently. In this
  work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
  an image-to-3D generation pipeline that takes only an image CLIP embedding as
  input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
  angle by solely resting on the SDS loss. The core of our framework lies in a
  two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
  diffusion model by substituting its text encoder with an image encoder, by
  which the model preliminarily acquires image-to-image capabilities. Secondly,
  we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
  combines noisy multi-view images with the noise-free reference image as an
  explicit condition. CLIP embedding is sent to the diffusion model throughout
  the whole process while reference images are discarded once after fine-tuning.
  As a result, with a single image CLIP embedding, Isotropic3D is capable of
  generating multi-view mutually consistent images and also a 3D model with more
  symmetrical and neat content, well-proportioned geometry, rich colored texture,
  and less distortion compared with existing image-to-3D methods while still
  preserving the similarity to the reference image to a large extent. The project
  page is available at https://isotropic3d.github.io/. The code and models are
  available at https://github.com/pkunliu/Isotropic3D.
paper_authors._id: 65f79f593a8814cf2f4fe9c8
paper_authors.name: Pengkun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:11:57.663Z
paper_authors._id: 63daf1700cc3bc12bc09de74
paper_authors.avatarUrl: /avatars/773a14ead497813c9d102bea1222d681.svg
paper_authors.fullname: Pengkun Liu
paper_authors.type: user
paper_authors.user: pengkunliu
paper_authors._id: 65f79f593a8814cf2f4fe9c9
paper_authors.name: Yikai Wang
paper_authors._id: 65f79f593a8814cf2f4fe9ca
paper_authors.name: Fuchun Sun
paper_authors._id: 65f79f593a8814cf2f4fe9cb
paper_authors.name: Jiafang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:12:44.232Z
paper_authors._id: 645633adc8f569b995d7628b
paper_authors.avatarUrl: /avatars/5ffd94289a53b6690f9f7db37557497d.svg
paper_authors.fullname: jiafang.li
paper_authors.type: user
paper_authors.user: jiafang
paper_authors._id: 65f79f593a8814cf2f4fe9cc
paper_authors.name: Hang Xiao
paper_authors._id: 65f79f593a8814cf2f4fe9cd
paper_authors.name: Hongxiang Xue
paper_authors._id: 65f79f593a8814cf2f4fe9ce
paper_authors.name: Xinzhou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:13:16.794Z
paper_authors._id: 6459c45b5a263a4c87b7c639
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t3iK1wPBOP-h_kLBzZA_h.png
paper_authors.fullname: Xinzhou Wang
paper_authors.type: user
paper_authors.user: zz7379
paper_title: Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
paper_summary: Encouraged by the growing availability of pre-trained 2D diffusion models,
  image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
  making remarkable progress. Most existing methods combine novel-view lifting
  from 2D diffusion models which usually take the reference image as a condition
  while applying hard L2 image supervision at the reference view. Yet heavily
  adhering to the image is prone to corrupting the inductive knowledge of the 2D
  diffusion model leading to flat or distorted 3D generation frequently. In this
  work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
  an image-to-3D generation pipeline that takes only an image CLIP embedding as
  input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
  angle by solely resting on the SDS loss. The core of our framework lies in a
  two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
  diffusion model by substituting its text encoder with an image encoder, by
  which the model preliminarily acquires image-to-image capabilities. Secondly,
  we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
  combines noisy multi-view images with the noise-free reference image as an
  explicit condition. CLIP embedding is sent to the diffusion model throughout
  the whole process while reference images are discarded once after fine-tuning.
  As a result, with a single image CLIP embedding, Isotropic3D is capable of
  generating multi-view mutually consistent images and also a 3D model with more
  symmetrical and neat content, well-proportioned geometry, rich colored texture,
  and less distortion compared with existing image-to-3D methods while still
  preserving the similarity to the reference image to a large extent. The project
  page is available at https://isotropic3d.github.io/. The code and models are
  available at https://github.com/pkunliu/Isotropic3D.
paper_ai_keywords: Score Distillation Sampling (SDS) novel-view lifting 2D diffusion models image CLIP embedding isotropic optimization azimuth angle SDS loss two-stage diffusion model fine-tuning text-to-3D diffusion model text encoder image encoder Explicit Multi-view Attention (EMA) noisy multi-view images noise-free reference image multi-view mutually consistent images symmetrical content neat content well-proportioned geometry rich colored texture