title: AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
    Video Generation
summary: Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
  generating high-quality single-modality content, including images, videos, and
  audio. However, it is still under-explored whether the transformer-based
  diffuser can efficiently denoise the Gaussian noises towards superb multimodal
  content creation. To bridge this gap, we introduce AV-DiT, a novel and
  efficient audio-visual diffusion transformer designed to generate high-quality,
  realistic videos with both visual and audio tracks. To minimize model
  complexity and computational costs, AV-DiT utilizes a shared DiT backbone
  pre-trained on image-only data, with only lightweight, newly inserted adapters
  being trainable. This shared backbone facilitates both audio and video
  generation. Specifically, the video branch incorporates a trainable temporal
  attention layer into a frozen pre-trained DiT block for temporal consistency.
  Additionally, a small number of trainable parameters adapt the image-based DiT
  block for audio generation. An extra shared DiT block, equipped with
  lightweight parameters, facilitates feature interaction between audio and
  visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
  Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
  performance in joint audio-visual generation with significantly fewer tunable
  parameters. Furthermore, our results highlight that a single shared image
  generative backbone with modality-specific adaptations is sufficient for
  constructing a joint audio-video generator. Our source code and pre-trained
  models will be released.
paper_authors._id: 666a52b2cf27eed0f20fbf5a
paper_authors.name: Kai Wang
paper_authors._id: 666a52b2cf27eed0f20fbf5b
paper_authors.name: Shijian Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T20:01:02.128Z
paper_authors._id: 668bafa02c3897266dcbacd0
paper_authors.avatarUrl: /avatars/ef41cd4a855ebbe6efc79f34c38ef136.svg
paper_authors.fullname: Shijian Deng
paper_authors.type: user
paper_authors.user: ShijianDeng
paper_authors._id: 666a52b2cf27eed0f20fbf5c
paper_authors.name: Jing Shi
paper_authors._id: 666a52b2cf27eed0f20fbf5d
paper_authors.name: Dimitrios Hatzinakos
paper_authors._id: 666a52b2cf27eed0f20fbf5e
paper_authors.name: Yapeng Tian
paper_title: AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
    Video Generation
paper_summary: Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
  generating high-quality single-modality content, including images, videos, and
  audio. However, it is still under-explored whether the transformer-based
  diffuser can efficiently denoise the Gaussian noises towards superb multimodal
  content creation. To bridge this gap, we introduce AV-DiT, a novel and
  efficient audio-visual diffusion transformer designed to generate high-quality,
  realistic videos with both visual and audio tracks. To minimize model
  complexity and computational costs, AV-DiT utilizes a shared DiT backbone
  pre-trained on image-only data, with only lightweight, newly inserted adapters
  being trainable. This shared backbone facilitates both audio and video
  generation. Specifically, the video branch incorporates a trainable temporal
  attention layer into a frozen pre-trained DiT block for temporal consistency.
  Additionally, a small number of trainable parameters adapt the image-based DiT
  block for audio generation. An extra shared DiT block, equipped with
  lightweight parameters, facilitates feature interaction between audio and
  visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
  Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
  performance in joint audio-visual generation with significantly fewer tunable
  parameters. Furthermore, our results highlight that a single shared image
  generative backbone with modality-specific adaptations is sufficient for
  constructing a joint audio-video generator. Our source code and pre-trained
  models will be released.
paper_ai_keywords: Diffusion Transformers AV-DiT Gaussian noises shared DiT backbone temporal attention layer modality-specific adaptations joint audio-visual generation feature interaction