HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
Representing human performance at high-fidelity is an essential building
block in diverse applications, such as film production, computer games or
videoconferencing. To close the gap to production-level quality, we introduce
HumanRF, a 4D dynamic neural scene representation that captures full-body
appearance in motion from multi-view video input, and enables playback from
novel, unseen viewpoints. Our novel representation acts as a dynamic video
encoding that captures fine details at high compression rates by factorizing
space-time into a temporal matrix-vector decomposition. This allows us to
obtain temporally coherent reconstructions of human actors for long sequences,
while representing high-resolution details even in the context of challenging
motion. While most research focuses on synthesizing at resolutions of 4MP or
lower, we address the challenge of operating at 12MP. To this end, we introduce
ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
demonstrate challenges that emerge from using such high-resolution data and
show that our newly introduced HumanRF effectively leverages this data, making
a significant step towards production-level quality novel view synthesis.
645c66ef1d2ba8a1cf4939c1
Mustafa Işık
claimed_verified
2023-05-18T02:56:23.764Z
64655fce5cdb9ab50f867916
/avatars/b4a08280abcb7447a46bc0aec90de3ff.svg
Mustafa Isik
user
mustafaisik
645c66ef1d2ba8a1cf4939c2
Martin Rünz
645c66ef1d2ba8a1cf4939c3
Markos Georgopoulos
645c66ef1d2ba8a1cf4939c4
Taras Khakhulin
admin_assigned
2023-05-17T13:58:33.396Z
6366434d64bcbbd03e2fb7c2
/avatars/6265b2a7cfaab736654a7259745608d0.svg
Taras Khakhulin
user
khakhulin
645c66ef1d2ba8a1cf4939c5
Jonathan Starck
645c66ef1d2ba8a1cf4939c6
Lourdes Agapito
645c66ef1d2ba8a1cf4939c7
Matthias Nießner
HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
Representing human performance at high-fidelity is an essential building
block in diverse applications, such as film production, computer games or
videoconferencing. To close the gap to production-level quality, we introduce
HumanRF, a 4D dynamic neural scene representation that captures full-body
appearance in motion from multi-view video input, and enables playback from
novel, unseen viewpoints. Our novel representation acts as a dynamic video
encoding that captures fine details at high compression rates by factorizing
space-time into a temporal matrix-vector decomposition. This allows us to
obtain temporally coherent reconstructions of human actors for long sequences,
while representing high-resolution details even in the context of challenging
motion. While most research focuses on synthesizing at resolutions of 4MP or
lower, we address the challenge of operating at 12MP. To this end, we introduce
ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
demonstrate challenges that emerge from using such high-resolution data and
show that our newly introduced HumanRF effectively leverages this data, making
a significant step towards production-level quality novel view synthesis.
HumanRF is a 4D neural scene representation that captures high-fidelity, full-body motion for novel view synthesis, utilizing a temporal matrix-vector decomposition for high-resolution data.
dynamic neural scene representation full-body appearance multi-view video input novel viewpoints temporal matrix-vector decomposition ActorsHQ per-frame mesh reconstructions novel view synthesis