title: AvatarReX: Real-time Expressive Full-body Avatars
summary: We present AvatarReX, a new method for learning NeRF-based full-body avatars
  from video data. The learnt avatar not only provides expressive control of the
  body, hands and the face together, but also supports real-time animation and
  rendering. To this end, we propose a compositional avatar representation, where
  the body, hands and the face are separately modeled in a way that the
  structural prior from parametric mesh templates is properly utilized without
  compromising representation flexibility. Furthermore, we disentangle the
  geometry and appearance for each part. With these technical designs, we propose
  a dedicated deferred rendering pipeline, which can be executed in real-time
  framerate to synthesize high-quality free-view images. The disentanglement of
  geometry and appearance also allows us to design a two-pass training strategy
  that combines volume rendering and surface rendering for network training. In
  this way, patch-level supervision can be applied to force the network to learn
  sharp appearance details on the basis of geometry estimation. Overall, our
  method enables automatic construction of expressive full-body avatars with
  real-time rendering capability, and can generate photo-realistic images with
  dynamic details for novel body motions and facial expressions.
paper_authors._id: 6459c49d38d95f3427a5c23b
paper_authors.name: Zerong Zheng
paper_authors._id: 6459c49d38d95f3427a5c23c
paper_authors.name: Xiaochen Zhao
paper_authors._id: 6459c49d38d95f3427a5c23d
paper_authors.name: Hongwen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:04:47.541Z
paper_authors._id: 638715a1aaee5cd69dc0c401
paper_authors.avatarUrl: /avatars/f5976b3b02fe86191cb7f59c3dae102b.svg
paper_authors.fullname: Hongwen Zhang
paper_authors.type: user
paper_authors.user: HongwenZhang
paper_authors._id: 6459c49d38d95f3427a5c23e
paper_authors.name: Boning Liu
paper_authors._id: 6459c49d38d95f3427a5c23f
paper_authors.name: Yebin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:05:34.593Z
paper_authors._id: 62e14dbe4db2175cd2735a80
paper_authors.avatarUrl: /avatars/e6385dcedcb97e1b36281b49210321aa.svg
paper_authors.fullname: Yebin Liu
paper_authors.type: user
paper_authors.user: YebinLiu
paper_title: AvatarReX: Real-time Expressive Full-body Avatars
paper_summary: We present AvatarReX, a new method for learning NeRF-based full-body avatars
  from video data. The learnt avatar not only provides expressive control of the
  body, hands and the face together, but also supports real-time animation and
  rendering. To this end, we propose a compositional avatar representation, where
  the body, hands and the face are separately modeled in a way that the
  structural prior from parametric mesh templates is properly utilized without
  compromising representation flexibility. Furthermore, we disentangle the
  geometry and appearance for each part. With these technical designs, we propose
  a dedicated deferred rendering pipeline, which can be executed in real-time
  framerate to synthesize high-quality free-view images. The disentanglement of
  geometry and appearance also allows us to design a two-pass training strategy
  that combines volume rendering and surface rendering for network training. In
  this way, patch-level supervision can be applied to force the network to learn
  sharp appearance details on the basis of geometry estimation. Overall, our
  method enables automatic construction of expressive full-body avatars with
  real-time rendering capability, and can generate photo-realistic images with
  dynamic details for novel body motions and facial expressions.
paper_ai_summary: A new method called AvatarReX enables real-time rendering of expressive full-body avatars from video data using a compositional avatar representation and a deferred rendering pipeline.
paper_ai_keywords: NeRF-based full-body avatars real-time animation compositional avatar representation parametric mesh templates disentanglement geometry appearance deferred rendering pipeline real-time framerate high-quality free-view images patch-level supervision volume rendering surface rendering photo-realistic images dynamic details facial expressions