title: One Shot, One Talk: Whole-body Talking Avatar from a Single Image
summary: Building realistic and animatable avatars still requires minutes of
  multi-view or monocular self-rotating videos, and most methods lack precise
  control over gestures and expressions. To push this boundary, we address the
  challenge of constructing a whole-body talking avatar from a single image. We
  propose a novel pipeline that tackles two critical issues: 1) complex dynamic
  modeling and 2) generalization to novel gestures and expressions. To achieve
  seamless generalization, we leverage recent pose-guided image-to-video
  diffusion models to generate imperfect video frames as pseudo-labels. To
  overcome the dynamic modeling challenge posed by inconsistent and noisy
  pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar
  representation and apply several key regularizations to mitigate
  inconsistencies caused by imperfect labels. Extensive experiments on diverse
  subjects demonstrate that our method enables the creation of a photorealistic,
  precisely animatable, and expressive whole-body talking avatar from just a
  single image.
paper_authors._id: 674e94bb320a043daeafc668
paper_authors.name: Jun Xiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:23.321Z
paper_authors._id: 657d8495e50ca9a699ae7a1a
paper_authors.avatarUrl: /avatars/afa2b9c30c37c1c2130cc3a2eabab19e.svg
paper_authors.fullname: Jun Xiang
paper_authors.type: user
paper_authors.user: xiangjun-xj
paper_authors._id: 674e94bb320a043daeafc669
paper_authors.name: Yudong Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:54.257Z
paper_authors._id: 6358a2cc1d66b442317f56c3
paper_authors.avatarUrl: /avatars/119b991a9dafefce4b9217346044f7ea.svg
paper_authors.fullname: Yudong Guo
paper_authors.type: user
paper_authors.user: YudongGuo
paper_authors._id: 674e94bb320a043daeafc66a
paper_authors.name: Leipeng Hu
paper_authors._id: 674e94bb320a043daeafc66b
paper_authors.name: Boyang Guo
paper_authors._id: 674e94bb320a043daeafc66c
paper_authors.name: Yancheng Yuan
paper_authors._id: 674e94bb320a043daeafc66d
paper_authors.name: Juyong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:44:32.580Z
paper_authors._id: 66fec5869b33512a6125ef2f
paper_authors.avatarUrl: /avatars/5e9762ab35de5dd64ba06573fdea2ea7.svg
paper_authors.fullname: Juyong Zhang
paper_authors.type: user
paper_authors.user: JuyongZhang
paper_title: One Shot, One Talk: Whole-body Talking Avatar from a Single Image
paper_summary: Building realistic and animatable avatars still requires minutes of
  multi-view or monocular self-rotating videos, and most methods lack precise
  control over gestures and expressions. To push this boundary, we address the
  challenge of constructing a whole-body talking avatar from a single image. We
  propose a novel pipeline that tackles two critical issues: 1) complex dynamic
  modeling and 2) generalization to novel gestures and expressions. To achieve
  seamless generalization, we leverage recent pose-guided image-to-video
  diffusion models to generate imperfect video frames as pseudo-labels. To
  overcome the dynamic modeling challenge posed by inconsistent and noisy
  pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar
  representation and apply several key regularizations to mitigate
  inconsistencies caused by imperfect labels. Extensive experiments on diverse
  subjects demonstrate that our method enables the creation of a photorealistic,
  precisely animatable, and expressive whole-body talking avatar from just a
  single image.
paper_ai_summary: A new method creates photorealistic, expressive whole-body talking avatars from a single image using pose-guided image-to-video diffusion models and a hybrid 3DGS-mesh representation with regularizations.
paper_ai_keywords: pose-guided image-to-video diffusion models pseudo-labels 3DGS-mesh regularizations whole-body talking avatar