NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
We focus on reconstructing high-fidelity radiance fields of human heads,
capturing their animations over time, and synthesizing re-renderings from novel
viewpoints at arbitrary time steps. To this end, we propose a new multi-view
capture setup composed of 16 calibrated machine vision cameras that record
time-synchronized images at 7.1 MP resolution and 73 frames per second. With
our setup, we collect a new dataset of over 4700 high-resolution,
high-framerate sequences of more than 220 human heads, from which we introduce
a new human head reconstruction benchmark. The recorded sequences cover a wide
range of facial dynamics, including head motions, natural expressions,
emotions, and spoken language. In order to reconstruct high-fidelity human
heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles
(NeRSemble). We represent scene dynamics by combining a deformation field and
an ensemble of 3D multi-resolution hash encodings. The deformation field allows
for precise modeling of simple scene movements, while the ensemble of hash
encodings helps to represent complex dynamics. As a result, we obtain radiance
field representations of human heads that capture motion over time and
facilitate re-rendering of arbitrary novel viewpoints. In a series of
experiments, we explore the design choices of our method and demonstrate that
our approach outperforms state-of-the-art dynamic radiance field approaches by
a significant margin.
645483faec40bbbd012556c6
Tobias Kirschstein
claimed_verified
2023-06-07T19:39:30.809Z
64806288cacb1c4a06946084
/avatars/5101d2234279257d534d9e9d2f69527f.svg
Tobias Kirschstein
user
tobias-kirschstein
645483faec40bbbd012556c7
Shenhan Qian
extracted_pending
2024-11-04T13:22:30.403Z
6728ca5827fa1635d7b168aa
/avatars/39d4bf9539d4b066aa5a553338d9e31e.svg
Shenhan Qian
user
Shenhan
645483faec40bbbd012556c8
Simon Giebenhain
extracted_confirmed
2024-07-24T13:32:33.700Z
64dc981b96fbcd6b19f687b5
/avatars/6eca33e3fcca750d4f6e9b0f7d724d8e.svg
Giebenhain
user
SimonGiebenhain
645483faec40bbbd012556c9
Tim Walter
extracted_confirmed
2023-05-25T17:14:56.136Z
646f9768cc049b49e35e8147
/avatars/326ffea9f66112d547c6f77d2f505802.svg
Tim Walter
user
TimWalter
645483faec40bbbd012556ca
Matthias Nießner
NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
We focus on reconstructing high-fidelity radiance fields of human heads,
capturing their animations over time, and synthesizing re-renderings from novel
viewpoints at arbitrary time steps. To this end, we propose a new multi-view
capture setup composed of 16 calibrated machine vision cameras that record
time-synchronized images at 7.1 MP resolution and 73 frames per second. With
our setup, we collect a new dataset of over 4700 high-resolution,
high-framerate sequences of more than 220 human heads, from which we introduce
a new human head reconstruction benchmark. The recorded sequences cover a wide
range of facial dynamics, including head motions, natural expressions,
emotions, and spoken language. In order to reconstruct high-fidelity human
heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles
(NeRSemble). We represent scene dynamics by combining a deformation field and
an ensemble of 3D multi-resolution hash encodings. The deformation field allows
for precise modeling of simple scene movements, while the ensemble of hash
encodings helps to represent complex dynamics. As a result, we obtain radiance
field representations of human heads that capture motion over time and
facilitate re-rendering of arbitrary novel viewpoints. In a series of
experiments, we explore the design choices of our method and demonstrate that
our approach outperforms state-of-the-art dynamic radiance field approaches by
a significant margin.
A new multi-view capture setup and Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble) enable high-fidelity reconstruction and re-rendering of human heads over time.
Neural Radiance Fields Hash Ensembles NeRSemble deformation field multi-resolution hash encodings radiance field high-fidelity reconstruction re-rendering