HyperFields: Towards Zero-Shot Generation of NeRFs from Text
We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.
653b29afd994e992e27b800d
Sudarshan Babu
claimed_verified
2023-10-27T09:12:43.718Z
6341baf206a304bdc432d28b
/avatars/31fe822d1cd05d9bd7009b97873cb527.svg
sudarshan babu
user
sudarshan94
653b29afd994e992e27b800e
Richard Liu
653b29afd994e992e27b800f
Avery Zhou
admin_assigned
2023-10-27T09:15:38.847Z
63433036aa45ed8ecf97da55
/avatars/63566616130bddfae399ff52d600e9e6.svg
Avery Zhou
user
averyzhou
653b29afd994e992e27b8010
Michael Maire
admin_assigned
2023-10-27T09:15:17.865Z
63dcf657f37111482522cd56
https://cdn-avatars.huggingface.co/v1/production/uploads/1675425351343-noauth.png
Michael Maire
user
MikeStudios
653b29afd994e992e27b8011
Greg Shakhnarovich
653b29afd994e992e27b8012
Rana Hanocka
admin_assigned
2023-10-27T09:15:28.844Z
630495ffce6b12280b193f30
/avatars/55b201131fa798c2a33d17dcab72fd1f.svg
Rana Hanocka
user
ranahanocka
HyperFields: Towards Zero-Shot Generation of NeRFs from Text
We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.
HyperFields, a method using a dynamic hypernetwork and NeRF distillation, generates text-conditioned NeRFs efficiently across multiple scenes with potentially fast fine-tuning.
HyperFields dynamic hypernetwork NeRFs text token embeddings NeRF distillation zero-shot fine-tuning neural optimization-based methods