JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
The intrinsic link between facial motion and speech is often overlooked in
generative modeling, where talking head synthesis and text-to-speech (TTS) are
typically addressed as separate tasks. This paper introduces JAM-Flow, a
unified framework to simultaneously synthesize and condition on both facial
motion and speech. Our approach leverages flow matching and a novel Multi-Modal
Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT
and Audio-DiT modules. These are coupled via selective joint attention layers
and incorporate key architectural choices, such as temporally aligned
positional embeddings and localized joint attention masking, to enable
effective cross-modal interaction while preserving modality-specific strengths.
Trained with an inpainting-style objective, JAM-Flow supports a wide array of
conditioning inputs-including text, reference audio, and reference
motion-facilitating tasks such as synchronized talking head generation from
text, audio-driven animation, and much more, within a single, coherent model.
JAM-Flow significantly advances multi-modal generative modeling by providing a
practical solution for holistic audio-visual synthesis. project page:
https://joonghyuk.com/jamflow-web
6865e0148c83dab5f72d1e26
Mingi Kwon
6865e0148c83dab5f72d1e27
Joonghyuk Shin
claimed_verified
2025-07-03T07:16:56.605Z
631074d895c34b95407945f0
/avatars/699baf06ec818650dec5752aca87c5b4.svg
Joonghyuk Shin
user
alex4727
6865e0148c83dab5f72d1e28
Jaeseok Jung
6865e0148c83dab5f72d1e29
Jaesik Park
6865e0148c83dab5f72d1e2a
Youngjung Uh
JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching
The intrinsic link between facial motion and speech is often overlooked in
generative modeling, where talking head synthesis and text-to-speech (TTS) are
typically addressed as separate tasks. This paper introduces JAM-Flow, a
unified framework to simultaneously synthesize and condition on both facial
motion and speech. Our approach leverages flow matching and a novel Multi-Modal
Diffusion Transformer (MM-DiT) architecture, integrating specialized Motion-DiT
and Audio-DiT modules. These are coupled via selective joint attention layers
and incorporate key architectural choices, such as temporally aligned
positional embeddings and localized joint attention masking, to enable
effective cross-modal interaction while preserving modality-specific strengths.
Trained with an inpainting-style objective, JAM-Flow supports a wide array of
conditioning inputs-including text, reference audio, and reference
motion-facilitating tasks such as synchronized talking head generation from
text, audio-driven animation, and much more, within a single, coherent model.
JAM-Flow significantly advances multi-modal generative modeling by providing a
practical solution for holistic audio-visual synthesis. project page:
https://joonghyuk.com/jamflow-web
JAM-Flow, a unified framework using flow matching and a Multi-Modal Diffusion Transformer, integrates facial motion and speech synthesis with selective joint attention layers and temporally aligned positional embeddings.
flow matching Multi-Modal Diffusion Transformer MM-DiT Motion-DiT Audio-DiT selective joint attention layers temporally aligned positional embeddings localized joint attention masking inpainting-style objective synchronized talking head generation audio-driven animation