title: ChatAnyone: Stylized Real-time Portrait Video Generation with
    Hierarchical Motion Diffusion Model
summary: Real-time interactive video-chat portraits have been increasingly recognized
  as the future trend, particularly due to the remarkable progress made in text
  and voice chat technologies. However, existing methods primarily focus on
  real-time generation of head movements, but struggle to produce synchronized
  body motions that match these head actions. Additionally, achieving
  fine-grained control over the speaking style and nuances of facial expressions
  remains a challenge. To address these limitations, we introduce a novel
  framework for stylized real-time portrait video generation, enabling expressive
  and flexible video chat that extends from talking head to upper-body
  interaction. Our approach consists of the following two stages. The first stage
  involves efficient hierarchical motion diffusion models, that take both
  explicit and implicit motion representations into account based on audio
  inputs, which can generate a diverse range of facial expressions with stylistic
  control and synchronization between head and body movements. The second stage
  aims to generate portrait video featuring upper-body movements, including hand
  gestures. We inject explicit hand control signals into the generator to produce
  more detailed hand movements, and further perform face refinement to enhance
  the overall realism and expressiveness of the portrait video. Additionally, our
  approach supports efficient and continuous generation of upper-body portrait
  video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting
  interactive video-chat in real-time. Experimental results demonstrate the
  capability of our approach to produce portrait videos with rich expressiveness
  and natural upper-body movements.
paper_authors._id: 67e60e315f20b94fcd0d1f9b
paper_authors.name: Jinwei Qi
paper_authors._id: 67e60e315f20b94fcd0d1f9c
paper_authors.name: Chaonan Ji
paper_authors._id: 67e60e315f20b94fcd0d1f9d
paper_authors.name: Sheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:10:43.747Z
paper_authors._id: 672d72751474234855223935
paper_authors.avatarUrl: /avatars/dc8a79b7d5e1175725334b337702e1fd.svg
paper_authors.fullname: Sheng Xu
paper_authors.type: user
paper_authors.user: shengxu97
paper_authors._id: 67e60e315f20b94fcd0d1f9e
paper_authors.name: Peng Zhang
paper_authors._id: 67e60e315f20b94fcd0d1f9f
paper_authors.name: Bang Zhang
paper_authors._id: 67e60e315f20b94fcd0d1fa0
paper_authors.name: Liefeng Bo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:11:02.629Z
paper_authors._id: 63d0cc736b985b0f25d0412c
paper_authors.avatarUrl: /avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
paper_authors.fullname: Bo
paper_authors.type: user
paper_authors.user: Liefeng
paper_title: ChatAnyone: Stylized Real-time Portrait Video Generation with
    Hierarchical Motion Diffusion Model
paper_summary: Real-time interactive video-chat portraits have been increasingly recognized
  as the future trend, particularly due to the remarkable progress made in text
  and voice chat technologies. However, existing methods primarily focus on
  real-time generation of head movements, but struggle to produce synchronized
  body motions that match these head actions. Additionally, achieving
  fine-grained control over the speaking style and nuances of facial expressions
  remains a challenge. To address these limitations, we introduce a novel
  framework for stylized real-time portrait video generation, enabling expressive
  and flexible video chat that extends from talking head to upper-body
  interaction. Our approach consists of the following two stages. The first stage
  involves efficient hierarchical motion diffusion models, that take both
  explicit and implicit motion representations into account based on audio
  inputs, which can generate a diverse range of facial expressions with stylistic
  control and synchronization between head and body movements. The second stage
  aims to generate portrait video featuring upper-body movements, including hand
  gestures. We inject explicit hand control signals into the generator to produce
  more detailed hand movements, and further perform face refinement to enhance
  the overall realism and expressiveness of the portrait video. Additionally, our
  approach supports efficient and continuous generation of upper-body portrait
  video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting
  interactive video-chat in real-time. Experimental results demonstrate the
  capability of our approach to produce portrait videos with rich expressiveness
  and natural upper-body movements.
paper_ai_summary: The novel framework for real-time portrait video generation uses hierarchical motion diffusion models and explicit hand control to produce expressive video chats with synchronized head and body movements.
paper_ai_keywords: motion diffusion models hierarchical motion diffusion models facial expressions synchronization upper-body movements hand gestures face refinement real-time generation interactive video-chat