title: Relightable and Animatable Neural Avatar from Sparse-View Video
summary: This paper tackles the challenge of creating relightable and animatable
  neural avatars from sparse-view (or even monocular) videos of dynamic humans
  under unknown illumination. Compared to studio environments, this setting is
  more practical and accessible but poses an extremely challenging ill-posed
  problem. Previous neural human reconstruction methods are able to reconstruct
  animatable avatars from sparse views using deformed Signed Distance Fields
  (SDF) but cannot recover material parameters for relighting. While
  differentiable inverse rendering-based methods have succeeded in material
  recovery of static objects, it is not straightforward to extend them to dynamic
  humans as it is computationally intensive to compute pixel-surface intersection
  and light visibility on deformed SDFs for inverse rendering. To solve this
  challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to
  approximate the world space distances under arbitrary human poses.
  Specifically, we estimate coarse distances based on a parametric human model
  and compute fine distances by exploiting the local deformation invariance of
  SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently
  estimate the surface intersection and light visibility. This allows us to
  develop the first system to recover animatable and relightable neural avatars
  from sparse view (or monocular) inputs. Experiments demonstrate that our
  approach is able to produce superior results compared to state-of-the-art
  methods. Our code will be released for reproducibility.
paper_authors._id: 64dd3fecc79e3245ebb7613e
paper_authors.name: Zhen Xu
paper_authors._id: 64dd3fecc79e3245ebb7613f
paper_authors.name: Sida Peng
paper_authors._id: 64dd3fecc79e3245ebb76140
paper_authors.name: Chen Geng
paper_authors._id: 64dd3fecc79e3245ebb76141
paper_authors.name: Linzhan Mou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-20T14:07:54.036Z
paper_authors._id: 646df429ad20c6fa4f30b8f3
paper_authors.avatarUrl: /avatars/24e14788856a5ae7b383579bdc2a06b6.svg
paper_authors.fullname: Lin zhan Mou
paper_authors.type: user
paper_authors.user: moulz
paper_authors._id: 64dd3fecc79e3245ebb76142
paper_authors.name: Zihan Yan
paper_authors._id: 64dd3fecc79e3245ebb76143
paper_authors.name: Jiaming Sun
paper_authors._id: 64dd3fecc79e3245ebb76144
paper_authors.name: Hujun Bao
paper_authors._id: 64dd3fecc79e3245ebb76145
paper_authors.name: Xiaowei Zhou
paper_title: Relightable and Animatable Neural Avatar from Sparse-View Video
paper_summary: This paper tackles the challenge of creating relightable and animatable
  neural avatars from sparse-view (or even monocular) videos of dynamic humans
  under unknown illumination. Compared to studio environments, this setting is
  more practical and accessible but poses an extremely challenging ill-posed
  problem. Previous neural human reconstruction methods are able to reconstruct
  animatable avatars from sparse views using deformed Signed Distance Fields
  (SDF) but cannot recover material parameters for relighting. While
  differentiable inverse rendering-based methods have succeeded in material
  recovery of static objects, it is not straightforward to extend them to dynamic
  humans as it is computationally intensive to compute pixel-surface intersection
  and light visibility on deformed SDFs for inverse rendering. To solve this
  challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to
  approximate the world space distances under arbitrary human poses.
  Specifically, we estimate coarse distances based on a parametric human model
  and compute fine distances by exploiting the local deformation invariance of
  SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently
  estimate the surface intersection and light visibility. This allows us to
  develop the first system to recover animatable and relightable neural avatars
  from sparse view (or monocular) inputs. Experiments demonstrate that our
  approach is able to produce superior results compared to state-of-the-art
  methods. Our code will be released for reproducibility.
paper_ai_keywords: relightable animatable neural avatars sparse-view monocular dynamic humans unknown illumination signed distance fields (SDF) deformed SDF material parameters differentiable inverse rendering pixel-surface intersection light visibility hierarchical distance query (HDQ) parametric human model surface intersection sphere tracing