Audio-visual Controlled Video Diffusion with Masked Selective State
  Spaces Modeling for Natural Talking Head Generation
Talking head synthesis is vital for virtual avatars and human-computer
interaction. However, most existing methods are typically limited to accepting
control from a single primary modality, restricting their practical utility. To
this end, we introduce ACTalker, an end-to-end video diffusion
framework that supports both multi-signals control and single-signal control
for talking head video generation. For multiple control, we design a parallel
mamba structure with multiple branches, each utilizing a separate driving
signal to control specific facial regions. A gate mechanism is applied across
all branches, providing flexible control over video generation. To ensure
natural coordination of the controlled video both temporally and spatially, we
employ the mamba structure, which enables driving signals to manipulate feature
tokens across both dimensions in each branch. Additionally, we introduce a
mask-drop strategy that allows each driving signal to independently control its
corresponding facial region within the mamba structure, preventing control
conflicts. Experimental results demonstrate that our method produces
natural-looking facial videos driven by diverse signals and that the mamba
layer seamlessly integrates multiple driving modalities without conflict.
67ef3773ac0c701df7fd98aa
Fa-Ting Hong
claimed_verified
2025-04-04T07:09:38.641Z
6264a7dfc39850dc093eb68a
https://cdn-avatars.huggingface.co/v1/production/uploads/1650763566575-noauth.png
Fa-Ting Hong
user
HarlanHong
67ef3773ac0c701df7fd98ab
Zunnan Xu
claimed_verified
2025-04-04T07:09:23.733Z
6481523b3fb124fc9850afed
/avatars/ddde178c88713662800aafd2343647a4.svg
kkakkkka
user
kkakkkka
67ef3773ac0c701df7fd98ac
Zixiang Zhou
67ef3773ac0c701df7fd98ad
Jun Zhou
67ef3773ac0c701df7fd98ae
Xiu Li
67ef3773ac0c701df7fd98af
Qin Lin
67ef3773ac0c701df7fd98b0
Qinglin Lu
67ef3773ac0c701df7fd98b1
Dan Xu
claimed_verified
2025-04-04T07:09:20.987Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
user
danxuhk
Audio-visual Controlled Video Diffusion with Masked Selective State
  Spaces Modeling for Natural Talking Head Generation
Talking head synthesis is vital for virtual avatars and human-computer
interaction. However, most existing methods are typically limited to accepting
control from a single primary modality, restricting their practical utility. To
this end, we introduce ACTalker, an end-to-end video diffusion
framework that supports both multi-signals control and single-signal control
for talking head video generation. For multiple control, we design a parallel
mamba structure with multiple branches, each utilizing a separate driving
signal to control specific facial regions. A gate mechanism is applied across
all branches, providing flexible control over video generation. To ensure
natural coordination of the controlled video both temporally and spatially, we
employ the mamba structure, which enables driving signals to manipulate feature
tokens across both dimensions in each branch. Additionally, we introduce a
mask-drop strategy that allows each driving signal to independently control its
corresponding facial region within the mamba structure, preventing control
conflicts. Experimental results demonstrate that our method produces
natural-looking facial videos driven by diverse signals and that the mamba
layer seamlessly integrates multiple driving modalities without conflict.
ACTalker is an end-to-end video diffusion framework that uses a parallel mamba structure to generate talking head videos with flexible, multi-modal control, ensuring natural coordination across temporal and spatial dimensions.
video diffusion framework parallel mamba structure multi-signals control single-signal control gate mechanism feature tokens mask-drop strategy