title: SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
    Diffusion Transformers
summary: The generation and editing of audio-conditioned talking portraits guided by
  multimodal inputs, including text, images, and videos, remains under explored.
  In this paper, we present SkyReels-Audio, a unified framework for synthesizing
  high-fidelity and temporally coherent talking portrait videos. Built upon
  pretrained video diffusion transformers, our framework supports infinite-length
  generation and editing, while enabling diverse and controllable conditioning
  through multimodal inputs. We employ a hybrid curriculum learning strategy to
  progressively align audio with facial motion, enabling fine-grained multimodal
  control over long video sequences. To enhance local facial coherence, we
  introduce a facial mask loss and an audio-guided classifier-free guidance
  mechanism. A sliding-window denoising approach further fuses latent
  representations across temporal segments, ensuring visual fidelity and temporal
  consistency across extended durations and diverse identities. More importantly,
  we construct a dedicated data pipeline for curating high-quality triplets
  consisting of synchronized audio, video, and textual descriptions.
  Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
  performance in lip-sync accuracy, identity consistency, and realistic facial
  dynamics, particularly under complex and challenging conditions.
paper_authors._id: 684264d1a9584289f0053f5c
paper_authors.name: Zhengcong Fei
paper_authors._id: 684264d1a9584289f0053f5d
paper_authors.name: Hao Jiang
paper_authors._id: 684264d1a9584289f0053f5e
paper_authors.name: Di Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:40:29.181Z
paper_authors._id: 65bef422fdb8d33cefeaccc3
paper_authors.avatarUrl: /avatars/d40b0d7dda21fa1a68c291d11bc357ec.svg
paper_authors.fullname: Qiu Di
paper_authors.type: user
paper_authors.user: diqiu7
paper_authors._id: 684264d1a9584289f0053f5f
paper_authors.name: Baoxuan Gu
paper_authors._id: 684264d1a9584289f0053f60
paper_authors.name: Youqiang Zhang
paper_authors._id: 684264d1a9584289f0053f61
paper_authors.name: Jiahua Wang
paper_authors._id: 684264d1a9584289f0053f62
paper_authors.name: Jialin Bai
paper_authors._id: 684264d1a9584289f0053f63
paper_authors.name: Debang Li
paper_authors._id: 684264d1a9584289f0053f64
paper_authors.name: Mingyuan Fan
paper_authors._id: 684264d1a9584289f0053f65
paper_authors.name: Guibin Chen
paper_authors._id: 684264d1a9584289f0053f66
paper_authors.name: Yahui Zhou
paper_title: SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video
    Diffusion Transformers
paper_summary: The generation and editing of audio-conditioned talking portraits guided by
  multimodal inputs, including text, images, and videos, remains under explored.
  In this paper, we present SkyReels-Audio, a unified framework for synthesizing
  high-fidelity and temporally coherent talking portrait videos. Built upon
  pretrained video diffusion transformers, our framework supports infinite-length
  generation and editing, while enabling diverse and controllable conditioning
  through multimodal inputs. We employ a hybrid curriculum learning strategy to
  progressively align audio with facial motion, enabling fine-grained multimodal
  control over long video sequences. To enhance local facial coherence, we
  introduce a facial mask loss and an audio-guided classifier-free guidance
  mechanism. A sliding-window denoising approach further fuses latent
  representations across temporal segments, ensuring visual fidelity and temporal
  consistency across extended durations and diverse identities. More importantly,
  we construct a dedicated data pipeline for curating high-quality triplets
  consisting of synchronized audio, video, and textual descriptions.
  Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior
  performance in lip-sync accuracy, identity consistency, and realistic facial
  dynamics, particularly under complex and challenging conditions.
paper_ai_summary: SkyReels-Audio is a unified framework using pretrained video diffusion transformers for generating high-fidelity and coherent audio-conditioned talking portrait videos, supported by a hybrid curriculum learning strategy and advanced loss mechanisms.
paper_ai_keywords: video diffusion transformers infinite-length generation multimodal inputs hybrid curriculum learning facial mask loss classifier-free guidance mechanism sliding-window denoising lip-sync accuracy identity consistency realistic facial dynamics