title: HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
summary: Representing human performance at high-fidelity is an essential building
  block in diverse applications, such as film production, computer games or
  videoconferencing. To close the gap to production-level quality, we introduce
  HumanRF, a 4D dynamic neural scene representation that captures full-body
  appearance in motion from multi-view video input, and enables playback from
  novel, unseen viewpoints. Our novel representation acts as a dynamic video
  encoding that captures fine details at high compression rates by factorizing
  space-time into a temporal matrix-vector decomposition. This allows us to
  obtain temporally coherent reconstructions of human actors for long sequences,
  while representing high-resolution details even in the context of challenging
  motion. While most research focuses on synthesizing at resolutions of 4MP or
  lower, we address the challenge of operating at 12MP. To this end, we introduce
  ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
  cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
  demonstrate challenges that emerge from using such high-resolution data and
  show that our newly introduced HumanRF effectively leverages this data, making
  a significant step towards production-level quality novel view synthesis.
paper_authors._id: 645c66ef1d2ba8a1cf4939c1
paper_authors.name: Mustafa Işık
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-18T02:56:23.764Z
paper_authors._id: 64655fce5cdb9ab50f867916
paper_authors.avatarUrl: /avatars/b4a08280abcb7447a46bc0aec90de3ff.svg
paper_authors.fullname: Mustafa Isik
paper_authors.type: user
paper_authors.user: mustafaisik
paper_authors._id: 645c66ef1d2ba8a1cf4939c2
paper_authors.name: Martin Rünz
paper_authors._id: 645c66ef1d2ba8a1cf4939c3
paper_authors.name: Markos Georgopoulos
paper_authors._id: 645c66ef1d2ba8a1cf4939c4
paper_authors.name: Taras Khakhulin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-17T13:58:33.396Z
paper_authors._id: 6366434d64bcbbd03e2fb7c2
paper_authors.avatarUrl: /avatars/6265b2a7cfaab736654a7259745608d0.svg
paper_authors.fullname: Taras Khakhulin
paper_authors.type: user
paper_authors.user: khakhulin
paper_authors._id: 645c66ef1d2ba8a1cf4939c5
paper_authors.name: Jonathan Starck
paper_authors._id: 645c66ef1d2ba8a1cf4939c6
paper_authors.name: Lourdes Agapito
paper_authors._id: 645c66ef1d2ba8a1cf4939c7
paper_authors.name: Matthias Nießner
paper_title: HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
paper_summary: Representing human performance at high-fidelity is an essential building
  block in diverse applications, such as film production, computer games or
  videoconferencing. To close the gap to production-level quality, we introduce
  HumanRF, a 4D dynamic neural scene representation that captures full-body
  appearance in motion from multi-view video input, and enables playback from
  novel, unseen viewpoints. Our novel representation acts as a dynamic video
  encoding that captures fine details at high compression rates by factorizing
  space-time into a temporal matrix-vector decomposition. This allows us to
  obtain temporally coherent reconstructions of human actors for long sequences,
  while representing high-resolution details even in the context of challenging
  motion. While most research focuses on synthesizing at resolutions of 4MP or
  lower, we address the challenge of operating at 12MP. To this end, we introduce
  ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160
  cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We
  demonstrate challenges that emerge from using such high-resolution data and
  show that our newly introduced HumanRF effectively leverages this data, making
  a significant step towards production-level quality novel view synthesis.
paper_ai_summary: HumanRF is a 4D neural scene representation that captures high-fidelity, full-body motion for novel view synthesis, utilizing a temporal matrix-vector decomposition for high-resolution data.
paper_ai_keywords: dynamic neural scene representation full-body appearance multi-view video input novel viewpoints temporal matrix-vector decomposition ActorsHQ per-frame mesh reconstructions novel view synthesis