SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
  Diffusion Transformers
The generation and editing of audio-conditioned talking portraits guided by
multimodal inputs, including text, images, and videos, remains under explored.
In this paper, we present SkyReels-Audio, a unified framework for synthesizing
high-fidelity and temporally coherent talking portrait videos. Built upon
pretrained video diffusion transformers, our framework supports infinite-length
generation and editing, while enabling diverse and controllable conditioning
through multimodal inputs. We employ a hybrid curriculum learning strategy to
progressively align audio with facial motion, enabling fine-grained multimodal
control over long video sequences. To enhance local facial coherence, we
introduce a facial mask loss and an audio-guided classifier-free guidance
mechanism. A sliding-window denoising approach further fuses latent
representations across temporal segments, ensuring visual fidelity and temporal
consistency across extended durations and diverse identities. More importantly,
we construct a dedicated data pipeline for curating high-quality triplets
consisting of synchronized audio, video, and textual descriptions.
Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
performance in lip-sync accuracy, identity consistency, and realistic facial
dynamics, particularly under complex and challenging conditions.
684264d1a9584289f0053f5c
Zhengcong Fei
684264d1a9584289f0053f5d
Hao Jiang
684264d1a9584289f0053f5e
Di Qiu
claimed_verified
2025-06-06T07:40:29.181Z
65bef422fdb8d33cefeaccc3
/avatars/d40b0d7dda21fa1a68c291d11bc357ec.svg
Qiu Di
user
diqiu7
684264d1a9584289f0053f5f
Baoxuan Gu
684264d1a9584289f0053f60
Youqiang Zhang
684264d1a9584289f0053f61
Jiahua Wang
684264d1a9584289f0053f62
Jialin Bai
684264d1a9584289f0053f63
Debang Li
684264d1a9584289f0053f64
Mingyuan Fan
684264d1a9584289f0053f65
Guibin Chen
684264d1a9584289f0053f66
Yahui Zhou
SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
  Diffusion Transformers
The generation and editing of audio-conditioned talking portraits guided by
multimodal inputs, including text, images, and videos, remains under explored.
In this paper, we present SkyReels-Audio, a unified framework for synthesizing
high-fidelity and temporally coherent talking portrait videos. Built upon
pretrained video diffusion transformers, our framework supports infinite-length
generation and editing, while enabling diverse and controllable conditioning
through multimodal inputs. We employ a hybrid curriculum learning strategy to
progressively align audio with facial motion, enabling fine-grained multimodal
control over long video sequences. To enhance local facial coherence, we
introduce a facial mask loss and an audio-guided classifier-free guidance
mechanism. A sliding-window denoising approach further fuses latent
representations across temporal segments, ensuring visual fidelity and temporal
consistency across extended durations and diverse identities. More importantly,
we construct a dedicated data pipeline for curating high-quality triplets
consisting of synchronized audio, video, and textual descriptions.
Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
performance in lip-sync accuracy, identity consistency, and realistic facial
dynamics, particularly under complex and challenging conditions.
SkyReels-Audio is a unified framework using pretrained video diffusion transformers for generating high-fidelity and coherent audio-conditioned talking portrait videos, supported by a hybrid curriculum learning strategy and advanced loss mechanisms.
video diffusion transformers infinite-length generation multimodal inputs hybrid curriculum learning facial mask loss classifier-free guidance mechanism sliding-window denoising lip-sync accuracy identity consistency realistic facial dynamics