RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
We present RodinHD, which can generate high-fidelity 3D avatars from a
portrait image. Existing methods fail to capture intricate details such as
hairstyles which we tackle in this paper. We first identify an overlooked
problem of catastrophic forgetting that arises when fitting triplanes
sequentially on many avatars, caused by the MLP decoder sharing scheme. To
overcome this issue, we raise a novel data scheduling strategy and a weight
consolidation regularization term, which improves the decoder's capability of
rendering sharper details. Additionally, we optimize the guiding effect of the
portrait image by computing a finer-grained hierarchical representation that
captures rich 2D texture cues, and injecting them to the 3D diffusion model at
multiple layers via cross-attention. When trained on 46K avatars with a noise
schedule optimized for triplanes, the resulting model can generate 3D avatars
with notably better details than previous methods and can generalize to
in-the-wild portrait input.
668df833835bf7e64bcaab44
Bowen Zhang
668df833835bf7e64bcaab45
Yiji Cheng
admin_assigned
2024-07-10T14:57:21.913Z
635e6b5d398ff343c4f4ed11
/avatars/3b7809b12a8980cf948be51831c05317.svg
yijicheng
user
yiji
668df833835bf7e64bcaab46
Chunyu Wang
admin_assigned
2024-07-10T14:57:33.957Z
6342d0c01bdd3dfa55d5d6b0
/avatars/8d0cb559db487c09eff1bc3863320f9b.svg
chunyuwang
user
chunyuwang
668df833835bf7e64bcaab47
Ting Zhang
668df833835bf7e64bcaab48
Jiaolong Yang
admin_assigned
2024-07-10T10:22:47.395Z
649ac9ad69657756819c81af
/avatars/1a2ca68cc14eeff772a00bd5198d5ac3.svg
yang
user
jiaolong
668df833835bf7e64bcaab49
Yansong Tang
668df833835bf7e64bcaab4a
Feng Zhao
668df833835bf7e64bcaab4b
Dong Chen
668df833835bf7e64bcaab4c
Baining Guo
RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models
We present RodinHD, which can generate high-fidelity 3D avatars from a
portrait image. Existing methods fail to capture intricate details such as
hairstyles which we tackle in this paper. We first identify an overlooked
problem of catastrophic forgetting that arises when fitting triplanes
sequentially on many avatars, caused by the MLP decoder sharing scheme. To
overcome this issue, we raise a novel data scheduling strategy and a weight
consolidation regularization term, which improves the decoder's capability of
rendering sharper details. Additionally, we optimize the guiding effect of the
portrait image by computing a finer-grained hierarchical representation that
captures rich 2D texture cues, and injecting them to the 3D diffusion model at
multiple layers via cross-attention. When trained on 46K avatars with a noise
schedule optimized for triplanes, the resulting model can generate 3D avatars
with notably better details than previous methods and can generalize to
in-the-wild portrait input.
RodinHD generates high-fidelity 3D avatars from portraits by addressing catastrophic forgetting in triplanes and enhancing detail capture through hierarchical representation and cross-attention in 3D diffusion models.
triplanes MLP decoder weight consolidation hierarchical representation cross-attention 3D diffusion model