title: AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
    Audio-Video Generation
summary: We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
  generation that leverages the activations of frozen video and audio diffusion
  models for temporally-aligned cross-modal conditioning. The key to our
  framework is a Fusion Block that enables bidirectional information exchange
  between our backbone video and audio diffusion models through a
  temporally-aligned self attention operation. Unlike prior work that uses
  feature extractors pretrained for other tasks for the conditioning signal,
  AV-Link can directly leverage features obtained by the complementary modality
  in a single framework i.e. video features to generate audio, or audio features
  to generate video. We extensively evaluate our design choices and demonstrate
  the ability of our method to achieve synchronized and high-quality audiovisual
  content, showcasing its potential for applications in immersive media
  generation. Project Page: snap-research.github.io/AVLink/
paper_authors._id: 6765d9f5bde4bc579f5b0603
paper_authors.name: Moayed Haji-Ali
paper_authors._id: 6765d9f5bde4bc579f5b0604
paper_authors.name: Willi Menapace
paper_authors._id: 6765d9f5bde4bc579f5b0605
paper_authors.name: Aliaksandr Siarohin
paper_authors._id: 6765d9f5bde4bc579f5b0606
paper_authors.name: Ivan Skorokhodov
paper_authors._id: 6765d9f5bde4bc579f5b0607
paper_authors.name: Alper Canberk
paper_authors._id: 6765d9f5bde4bc579f5b0608
paper_authors.name: Kwot Sin Lee
paper_authors._id: 6765d9f5bde4bc579f5b0609
paper_authors.name: Vicente Ordonez
paper_authors._id: 6765d9f5bde4bc579f5b060a
paper_authors.name: Sergey Tulyakov
paper_title: AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
    Audio-Video Generation
paper_summary: We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
  generation that leverages the activations of frozen video and audio diffusion
  models for temporally-aligned cross-modal conditioning. The key to our
  framework is a Fusion Block that enables bidirectional information exchange
  between our backbone video and audio diffusion models through a
  temporally-aligned self attention operation. Unlike prior work that uses
  feature extractors pretrained for other tasks for the conditioning signal,
  AV-Link can directly leverage features obtained by the complementary modality
  in a single framework i.e. video features to generate audio, or audio features
  to generate video. We extensively evaluate our design choices and demonstrate
  the ability of our method to achieve synchronized and high-quality audiovisual
  content, showcasing its potential for applications in immersive media
  generation. Project Page: snap-research.github.io/AVLink/
paper_ai_summary: AV-Link is a unified framework using frozen diffusion models and a Fusion Block with self-attention for generating temporally aligned audio-visual content.
paper_ai_keywords: diffusion models Fusion Block self attention cross-modal conditioning audio-to-video generation video-to-audio generation immersive media generation