title: Phidias: A Generative Model for Creating 3D Content from Text, Image,
    and 3D Conditions with Reference-Augmented Diffusion
summary: In 3D modeling, designers often use an existing 3D model as a reference to
  create new ones. This practice has inspired the development of Phidias, a novel
  generative model that uses diffusion for reference-augmented 3D generation.
  Given an image, our method leverages a retrieved or user-provided 3D reference
  model to guide the generation process, thereby enhancing the generation
  quality, generalization ability, and controllability. Our model integrates
  three key components: 1) meta-ControlNet that dynamically modulates the
  conditioning strength, 2) dynamic reference routing that mitigates misalignment
  between the input image and 3D reference, and 3) self-reference augmentations
  that enable self-supervised training with a progressive curriculum.
  Collectively, these designs result in a clear improvement over existing
  methods. Phidias establishes a unified framework for 3D generation using text,
  image, and 3D conditions with versatile applications.
paper_authors._id: 66ea3def1e62ad9403ce2075
paper_authors.name: Zhenwei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-18T07:01:27.359Z
paper_authors._id: 6305e9471e731505a76b9099
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305e9471e731505a76b9099/JikVYIMZWoeKMd99Ruvej.jpeg
paper_authors.fullname: Zhenwei WANG (Zavier)
paper_authors.type: user
paper_authors.user: ZhenweiWang
paper_authors._id: 66ea3def1e62ad9403ce2076
paper_authors.name: Tengfei Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:55:49.095Z
paper_authors._id: 62e7c26236a8e8a827ff0891
paper_authors.avatarUrl: /avatars/15648407233bf5757262dbc04464d02e.svg
paper_authors.fullname: Tengfei Wang
paper_authors.type: user
paper_authors.user: tfwang
paper_authors._id: 66ea3def1e62ad9403ce2077
paper_authors.name: Zexin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:55:54.725Z
paper_authors._id: 63a5d18a8fb23d08bb30094c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671811420431-noauth.png
paper_authors.fullname: Zexin He
paper_authors.type: user
paper_authors.user: zxhezexin
paper_authors._id: 66ea3def1e62ad9403ce2078
paper_authors.name: Gerhard Hancke
paper_authors._id: 66ea3def1e62ad9403ce2079
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:55:18.257Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 66ea3def1e62ad9403ce207a
paper_authors.name: Rynson W. H. Lau
paper_title: Phidias: A Generative Model for Creating 3D Content from Text, Image,
    and 3D Conditions with Reference-Augmented Diffusion
paper_summary: In 3D modeling, designers often use an existing 3D model as a reference to
  create new ones. This practice has inspired the development of Phidias, a novel
  generative model that uses diffusion for reference-augmented 3D generation.
  Given an image, our method leverages a retrieved or user-provided 3D reference
  model to guide the generation process, thereby enhancing the generation
  quality, generalization ability, and controllability. Our model integrates
  three key components: 1) meta-ControlNet that dynamically modulates the
  conditioning strength, 2) dynamic reference routing that mitigates misalignment
  between the input image and 3D reference, and 3) self-reference augmentations
  that enable self-supervised training with a progressive curriculum.
  Collectively, these designs result in a clear improvement over existing
  methods. Phidias establishes a unified framework for 3D generation using text,
  image, and 3D conditions with versatile applications.
paper_ai_keywords: diffusion meta-ControlNet dynamic reference routing self-reference augmentations self-supervised training progressive curriculum