title: Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion
    Latent Aligners
summary: Video and audio content creation serves as the core technique for the movie
  industry and professional users. Recently, existing diffusion-based methods
  tackle video and audio generation separately, which hinders the technique
  transfer from academia to industry. In this work, we aim at filling the gap,
  with a carefully designed optimization-based framework for cross-visual-audio
  and joint-visual-audio generation. We observe the powerful generation ability
  of off-the-shelf video or audio generation models. Thus, instead of training
  the giant models from scratch, we propose to bridge the existing strong models
  with a shared latent representation space. Specifically, we propose a
  multimodality latent aligner with the pre-trained ImageBind model. Our latent
  aligner shares a similar core as the classifier guidance that guides the
  diffusion denoising process during inference time. Through carefully designed
  optimization strategy and loss functions, we show the superior performance of
  our method on joint video-audio generation, visual-steered audio generation,
  and audio-steered visual generation tasks. The project website can be found at
  https://yzxing87.github.io/Seeing-and-Hearing/
paper_authors._id: 65deb3413aec0530561b9bc5
paper_authors.name: Yazhou Xing
paper_authors._id: 65deb3413aec0530561b9bc6
paper_authors.name: Yingqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:33:53.827Z
paper_authors._id: 630231bf7e137e3d6b3b0645
paper_authors.avatarUrl: /avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: Yingqing
paper_authors._id: 65deb3413aec0530561b9bc7
paper_authors.name: Zeyue Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:25:05.894Z
paper_authors._id: 6504409b52ca06fef957266d
paper_authors.avatarUrl: /avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
paper_authors.fullname: Zeyue
paper_authors.type: user
paper_authors.user: Zeyue7
paper_authors._id: 65deb3413aec0530561b9bc8
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:34:07.511Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65deb3413aec0530561b9bc9
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:34:13.723Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_title: Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion
    Latent Aligners
paper_summary: Video and audio content creation serves as the core technique for the movie
  industry and professional users. Recently, existing diffusion-based methods
  tackle video and audio generation separately, which hinders the technique
  transfer from academia to industry. In this work, we aim at filling the gap,
  with a carefully designed optimization-based framework for cross-visual-audio
  and joint-visual-audio generation. We observe the powerful generation ability
  of off-the-shelf video or audio generation models. Thus, instead of training
  the giant models from scratch, we propose to bridge the existing strong models
  with a shared latent representation space. Specifically, we propose a
  multimodality latent aligner with the pre-trained ImageBind model. Our latent
  aligner shares a similar core as the classifier guidance that guides the
  diffusion denoising process during inference time. Through carefully designed
  optimization strategy and loss functions, we show the superior performance of
  our method on joint video-audio generation, visual-steered audio generation,
  and audio-steered visual generation tasks. The project website can be found at
  https://yzxing87.github.io/Seeing-and-Hearing/
paper_ai_keywords: diffusion-based methods cross-visual-audio joint-visual-audio generation optimization-based framework latent representation space Experimentalimodal latent aligner ImageBind model classifier guidance diffusion denoising process shared latent representation optimization strategy loss functions joint video-audio generation visual-steered audio generation audio-steered visual generation