title: Audio-visual Controlled Video Diffusion with Masked Selective State
    Spaces Modeling for Natural Talking Head Generation
summary: Talking head synthesis is vital for virtual avatars and human-computer
  interaction. However, most existing methods are typically limited to accepting
  control from a single primary modality, restricting their practical utility. To
  this end, we introduce ACTalker, an end-to-end video diffusion
  framework that supports both multi-signals control and single-signal control
  for talking head video generation. For multiple control, we design a parallel
  mamba structure with multiple branches, each utilizing a separate driving
  signal to control specific facial regions. A gate mechanism is applied across
  all branches, providing flexible control over video generation. To ensure
  natural coordination of the controlled video both temporally and spatially, we
  employ the mamba structure, which enables driving signals to manipulate feature
  tokens across both dimensions in each branch. Additionally, we introduce a
  mask-drop strategy that allows each driving signal to independently control its
  corresponding facial region within the mamba structure, preventing control
  conflicts. Experimental results demonstrate that our method produces
  natural-looking facial videos driven by diverse signals and that the mamba
  layer seamlessly integrates multiple driving modalities without conflict.
paper_authors._id: 67ef3773ac0c701df7fd98aa
paper_authors.name: Fa-Ting Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:38.641Z
paper_authors._id: 6264a7dfc39850dc093eb68a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650763566575-noauth.png
paper_authors.fullname: Fa-Ting Hong
paper_authors.type: user
paper_authors.user: HarlanHong
paper_authors._id: 67ef3773ac0c701df7fd98ab
paper_authors.name: Zunnan Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:23.733Z
paper_authors._id: 6481523b3fb124fc9850afed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481523b3fb124fc9850afed/a71_YLW9Gg_Lo1G8MRWQj.png
paper_authors.fullname: Xavier
paper_authors.type: user
paper_authors.user: kkakkkka
paper_authors._id: 67ef3773ac0c701df7fd98ac
paper_authors.name: Zixiang Zhou
paper_authors._id: 67ef3773ac0c701df7fd98ad
paper_authors.name: Jun Zhou
paper_authors._id: 67ef3773ac0c701df7fd98ae
paper_authors.name: Xiu Li
paper_authors._id: 67ef3773ac0c701df7fd98af
paper_authors.name: Qin Lin
paper_authors._id: 67ef3773ac0c701df7fd98b0
paper_authors.name: Qinglin Lu
paper_authors._id: 67ef3773ac0c701df7fd98b1
paper_authors.name: Dan Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:09:20.987Z
paper_authors._id: 66feab48651e00e22f33222e
paper_authors.avatarUrl: /avatars/7344377e2c796c7ec85194bb2fc78521.svg
paper_authors.fullname: Dan Xu
paper_authors.type: user
paper_authors.user: danxuhk
paper_title: Audio-visual Controlled Video Diffusion with Masked Selective State
    Spaces Modeling for Natural Talking Head Generation
paper_summary: Talking head synthesis is vital for virtual avatars and human-computer
  interaction. However, most existing methods are typically limited to accepting
  control from a single primary modality, restricting their practical utility. To
  this end, we introduce ACTalker, an end-to-end video diffusion
  framework that supports both multi-signals control and single-signal control
  for talking head video generation. For multiple control, we design a parallel
  mamba structure with multiple branches, each utilizing a separate driving
  signal to control specific facial regions. A gate mechanism is applied across
  all branches, providing flexible control over video generation. To ensure
  natural coordination of the controlled video both temporally and spatially, we
  employ the mamba structure, which enables driving signals to manipulate feature
  tokens across both dimensions in each branch. Additionally, we introduce a
  mask-drop strategy that allows each driving signal to independently control its
  corresponding facial region within the mamba structure, preventing control
  conflicts. Experimental results demonstrate that our method produces
  natural-looking facial videos driven by diverse signals and that the mamba
  layer seamlessly integrates multiple driving modalities without conflict.
paper_ai_summary: ACTalker is an end-to-end video diffusion framework that uses a parallel mamba structure to generate talking head videos with flexible, multi-modal control, ensuring natural coordination across temporal and spatial dimensions.
paper_ai_keywords: video diffusion framework parallel mamba structure multi-signals control single-signal control gate mechanism feature tokens mask-drop strategy