title: RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
summary: We present RodinHD, which can generate high-fidelity 3D avatars from a
  portrait image. Existing methods fail to capture intricate details such as
  hairstyles which we tackle in this paper. We first identify an overlooked
  problem of catastrophic forgetting that arises when fitting triplanes
  sequentially on many avatars, caused by the MLP decoder sharing scheme. To
  overcome this issue, we raise a novel data scheduling strategy and a weight
  consolidation regularization term, which improves the decoder's capability of
  rendering sharper details. Additionally, we optimize the guiding effect of the
  portrait image by computing a finer-grained hierarchical representation that
  captures rich 2D texture cues, and injecting them to the 3D diffusion model at
  multiple layers via cross-attention. When trained on 46K avatars with a noise
  schedule optimized for triplanes, the resulting model can generate 3D avatars
  with notably better details than previous methods and can generalize to
  in-the-wild portrait input.
paper_authors._id: 668df833835bf7e64bcaab44
paper_authors.name: Bowen Zhang
paper_authors._id: 668df833835bf7e64bcaab45
paper_authors.name: Yiji Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T14:57:21.913Z
paper_authors._id: 635e6b5d398ff343c4f4ed11
paper_authors.avatarUrl: /avatars/3b7809b12a8980cf948be51831c05317.svg
paper_authors.fullname: yijicheng
paper_authors.type: user
paper_authors.user: yiji
paper_authors._id: 668df833835bf7e64bcaab46
paper_authors.name: Chunyu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T14:57:33.957Z
paper_authors._id: 6342d0c01bdd3dfa55d5d6b0
paper_authors.avatarUrl: /avatars/8d0cb559db487c09eff1bc3863320f9b.svg
paper_authors.fullname: chunyuwang
paper_authors.type: user
paper_authors.user: chunyuwang
paper_authors._id: 668df833835bf7e64bcaab47
paper_authors.name: Ting Zhang
paper_authors._id: 668df833835bf7e64bcaab48
paper_authors.name: Jiaolong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:22:47.395Z
paper_authors._id: 649ac9ad69657756819c81af
paper_authors.avatarUrl: /avatars/1a2ca68cc14eeff772a00bd5198d5ac3.svg
paper_authors.fullname: yang
paper_authors.type: user
paper_authors.user: jiaolong
paper_authors._id: 668df833835bf7e64bcaab49
paper_authors.name: Yansong Tang
paper_authors._id: 668df833835bf7e64bcaab4a
paper_authors.name: Feng Zhao
paper_authors._id: 668df833835bf7e64bcaab4b
paper_authors.name: Dong Chen
paper_authors._id: 668df833835bf7e64bcaab4c
paper_authors.name: Baining Guo
paper_title: RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
paper_summary: We present RodinHD, which can generate high-fidelity 3D avatars from a
  portrait image. Existing methods fail to capture intricate details such as
  hairstyles which we tackle in this paper. We first identify an overlooked
  problem of catastrophic forgetting that arises when fitting triplanes
  sequentially on many avatars, caused by the MLP decoder sharing scheme. To
  overcome this issue, we raise a novel data scheduling strategy and a weight
  consolidation regularization term, which improves the decoder's capability of
  rendering sharper details. Additionally, we optimize the guiding effect of the
  portrait image by computing a finer-grained hierarchical representation that
  captures rich 2D texture cues, and injecting them to the 3D diffusion model at
  multiple layers via cross-attention. When trained on 46K avatars with a noise
  schedule optimized for triplanes, the resulting model can generate 3D avatars
  with notably better details than previous methods and can generalize to
  in-the-wild portrait input.
paper_ai_keywords: catastrophic forgetting triplanes MLP decoder weight consolidation hierarchical representation 2D texture cues 3D diffusion model cross-attention noise schedule