Instant3D: Fast Text-to-3D with Sparse-View Generation and Large
  Reconstruction Model
Text-to-3D with diffusion models have achieved remarkable progress in recent
years. However, existing methods either rely on score distillation-based
optimization which suffer from slow inference, low diversity and Janus
problems, or are feed-forward methods that generate low quality results due to
the scarcity of 3D training data. In this paper, we propose Instant3D, a novel
method that generates high-quality and diverse 3D assets from text prompts in a
feed-forward manner. We adopt a two-stage paradigm, which first generates a
sparse set of four structured and consistent views from text in one shot with a
fine-tuned 2D text-to-image diffusion model, and then directly regresses the
NeRF from the generated images with a novel transformer-based sparse-view
reconstructor. Through extensive experiments, we demonstrate that our method
can generate high-quality, diverse and Janus-free 3D assets within 20 seconds,
which is two order of magnitude faster than previous optimization-based methods
that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.
65519d760774258908da059a
Jiahao Li
admin_assigned
2023-11-22T13:27:25.219Z
630bffcf8b327c7b8b9dfcce
/avatars/12691f869f01217f7b92a326c30716b1.svg
Jiahao Li
user
54jiahao
65519d760774258908da059b
Hao Tan
65519d760774258908da059c
Kai Zhang
claimed_verified
2023-11-14T09:37:57.822Z
6351a882a4845f26d16a28d7
/avatars/4f8629fbf22faeb6511f4068f2828809.svg
Kai Zhang
user
Kai-46
65519d760774258908da059d
Zexiang Xu
claimed_verified
2023-11-17T07:41:44.064Z
65565b1b8616d8d740abb976
/avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
Zexiang Xu
user
zexiangxu
65519d760774258908da059e
Fujun Luan
65519d760774258908da059f
Yinghao Xu
admin_assigned
2023-11-22T13:26:29.903Z
6555da9adcf410fd0753569c
/avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
Yinghao Xu
user
justimyhxu
65519d760774258908da05a0
Yicong Hong
extracted_confirmed
2023-11-13T04:04:30.241Z
654ca66fe1671abcbc50a6e4
https://cdn-avatars.huggingface.co/v1/production/uploads/654ca66fe1671abcbc50a6e4/VqcnC17YcqC9Hk65_LxtE.jpeg
Yicong Hong
user
YicongHong
65519d760774258908da05a1
Kalyan Sunkavalli
65519d760774258908da05a2
Greg Shakhnarovich
65519d760774258908da05a3
Sai Bi
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large
  Reconstruction Model
Text-to-3D with diffusion models have achieved remarkable progress in recent
years. However, existing methods either rely on score distillation-based
optimization which suffer from slow inference, low diversity and Janus
problems, or are feed-forward methods that generate low quality results due to
the scarcity of 3D training data. In this paper, we propose Instant3D, a novel
method that generates high-quality and diverse 3D assets from text prompts in a
feed-forward manner. We adopt a two-stage paradigm, which first generates a
sparse set of four structured and consistent views from text in one shot with a
fine-tuned 2D text-to-image diffusion model, and then directly regresses the
NeRF from the generated images with a novel transformer-based sparse-view
reconstructor. Through extensive experiments, we demonstrate that our method
can generate high-quality, diverse and Janus-free 3D assets within 20 seconds,
which is two order of magnitude faster than previous optimization-based methods
that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.
Instant3D rapidly generates high-quality 3D assets from text using a two-stage approach involving a fine-tuned 2D diffusion model and a transformer-based reconstructor.
text-to-3D diffusion models score distillation-based optimization Janus problems feed-forward methods sparse set of views 2D text-to-image diffusion model NeRF transformer-based sparse-view reconstructor