AvatarReX: Real-time Expressive Full-body Avatars
We present AvatarReX, a new method for learning NeRF-based full-body avatars
from video data. The learnt avatar not only provides expressive control of the
body, hands and the face together, but also supports real-time animation and
rendering. To this end, we propose a compositional avatar representation, where
the body, hands and the face are separately modeled in a way that the
structural prior from parametric mesh templates is properly utilized without
compromising representation flexibility. Furthermore, we disentangle the
geometry and appearance for each part. With these technical designs, we propose
a dedicated deferred rendering pipeline, which can be executed in real-time
framerate to synthesize high-quality free-view images. The disentanglement of
geometry and appearance also allows us to design a two-pass training strategy
that combines volume rendering and surface rendering for network training. In
this way, patch-level supervision can be applied to force the network to learn
sharp appearance details on the basis of geometry estimation. Overall, our
method enables automatic construction of expressive full-body avatars with
real-time rendering capability, and can generate photo-realistic images with
dynamic details for novel body motions and facial expressions.
6459c49d38d95f3427a5c23b
Zerong Zheng
6459c49d38d95f3427a5c23c
Xiaochen Zhao
6459c49d38d95f3427a5c23d
Hongwen Zhang
admin_assigned
2023-06-02T14:04:47.541Z
638715a1aaee5cd69dc0c401
/avatars/f5976b3b02fe86191cb7f59c3dae102b.svg
Hongwen Zhang
user
HongwenZhang
6459c49d38d95f3427a5c23e
Boning Liu
6459c49d38d95f3427a5c23f
Yebin Liu
admin_assigned
2023-06-02T14:05:34.593Z
62e14dbe4db2175cd2735a80
/avatars/e6385dcedcb97e1b36281b49210321aa.svg
Yebin Liu
user
YebinLiu
AvatarReX: Real-time Expressive Full-body Avatars
We present AvatarReX, a new method for learning NeRF-based full-body avatars
from video data. The learnt avatar not only provides expressive control of the
body, hands and the face together, but also supports real-time animation and
rendering. To this end, we propose a compositional avatar representation, where
the body, hands and the face are separately modeled in a way that the
structural prior from parametric mesh templates is properly utilized without
compromising representation flexibility. Furthermore, we disentangle the
geometry and appearance for each part. With these technical designs, we propose
a dedicated deferred rendering pipeline, which can be executed in real-time
framerate to synthesize high-quality free-view images. The disentanglement of
geometry and appearance also allows us to design a two-pass training strategy
that combines volume rendering and surface rendering for network training. In
this way, patch-level supervision can be applied to force the network to learn
sharp appearance details on the basis of geometry estimation. Overall, our
method enables automatic construction of expressive full-body avatars with
real-time rendering capability, and can generate photo-realistic images with
dynamic details for novel body motions and facial expressions.
A new method called AvatarReX enables real-time rendering of expressive full-body avatars from video data using a compositional avatar representation and a deferred rendering pipeline.
NeRF-based full-body avatars real-time animation compositional avatar representation parametric mesh templates disentanglement geometry appearance deferred rendering pipeline real-time framerate high-quality free-view images patch-level supervision volume rendering surface rendering photo-realistic images dynamic details facial expressions