ChatAnyone: Stylized Real-time Portrait Video Generation with
  Hierarchical Motion Diffusion Model
Real-time interactive video-chat portraits have been increasingly recognized
as the future trend, particularly due to the remarkable progress made in text
and voice chat technologies. However, existing methods primarily focus on
real-time generation of head movements, but struggle to produce synchronized
body motions that match these head actions. Additionally, achieving
fine-grained control over the speaking style and nuances of facial expressions
remains a challenge. To address these limitations, we introduce a novel
framework for stylized real-time portrait video generation, enabling expressive
and flexible video chat that extends from talking head to upper-body
interaction. Our approach consists of the following two stages. The first stage
involves efficient hierarchical motion diffusion models, that take both
explicit and implicit motion representations into account based on audio
inputs, which can generate a diverse range of facial expressions with stylistic
control and synchronization between head and body movements. The second stage
aims to generate portrait video featuring upper-body movements, including hand
gestures. We inject explicit hand control signals into the generator to produce
more detailed hand movements, and further perform face refinement to enhance
the overall realism and expressiveness of the portrait video. Additionally, our
approach supports efficient and continuous generation of upper-body portrait
video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting
interactive video-chat in real-time. Experimental results demonstrate the
capability of our approach to produce portrait videos with rich expressiveness
and natural upper-body movements.
67e60e315f20b94fcd0d1f9b
Jinwei Qi
67e60e315f20b94fcd0d1f9c
Chaonan Ji
67e60e315f20b94fcd0d1f9d
Sheng Xu
admin_assigned
2025-03-28T09:10:43.747Z
672d72751474234855223935
/avatars/dc8a79b7d5e1175725334b337702e1fd.svg
Sheng Xu
user
shengxu97
67e60e315f20b94fcd0d1f9e
Peng Zhang
67e60e315f20b94fcd0d1f9f
Bang Zhang
67e60e315f20b94fcd0d1fa0
Liefeng Bo
admin_assigned
2025-03-28T09:11:02.629Z
63d0cc736b985b0f25d0412c
/avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
Bo
user
Liefeng
ChatAnyone: Stylized Real-time Portrait Video Generation with
  Hierarchical Motion Diffusion Model
Real-time interactive video-chat portraits have been increasingly recognized
as the future trend, particularly due to the remarkable progress made in text
and voice chat technologies. However, existing methods primarily focus on
real-time generation of head movements, but struggle to produce synchronized
body motions that match these head actions. Additionally, achieving
fine-grained control over the speaking style and nuances of facial expressions
remains a challenge. To address these limitations, we introduce a novel
framework for stylized real-time portrait video generation, enabling expressive
and flexible video chat that extends from talking head to upper-body
interaction. Our approach consists of the following two stages. The first stage
involves efficient hierarchical motion diffusion models, that take both
explicit and implicit motion representations into account based on audio
inputs, which can generate a diverse range of facial expressions with stylistic
control and synchronization between head and body movements. The second stage
aims to generate portrait video featuring upper-body movements, including hand
gestures. We inject explicit hand control signals into the generator to produce
more detailed hand movements, and further perform face refinement to enhance
the overall realism and expressiveness of the portrait video. Additionally, our
approach supports efficient and continuous generation of upper-body portrait
video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting
interactive video-chat in real-time. Experimental results demonstrate the
capability of our approach to produce portrait videos with rich expressiveness
and natural upper-body movements.
The novel framework for real-time portrait video generation uses hierarchical motion diffusion models and explicit hand control to produce expressive video chats with synchronized head and body movements.
motion diffusion models hierarchical motion diffusion models facial expressions synchronization upper-body movements hand gestures face refinement real-time generation interactive video-chat