title: Video-to-Audio Generation with Hidden Alignment
summary: Generating semantically and temporally aligned audio content in accordance
  with video input has become a focal point for researchers, particularly
  following the remarkable breakthrough in text-to-video generation. In this
  work, we aim to offer insights into the video-to-audio generation paradigm,
  focusing on three crucial aspects: vision encoders, auxiliary embeddings, and
  data augmentation techniques. Beginning with a foundational model VTA-LDM built
  on a simple yet surprisingly effective intuition, we explore various vision
  encoders and auxiliary embeddings through ablation studies. Employing a
  comprehensive evaluation pipeline that emphasizes generation quality and
  video-audio synchronization alignment, we demonstrate that our model exhibits
  state-of-the-art video-to-audio generation capabilities. Furthermore, we
  provide critical insights into the impact of different data augmentation
  methods on enhancing the generation framework's overall capacity. We showcase
  possibilities to advance the challenge of generating synchronized audio from
  semantic and temporal perspectives. We hope these insights will serve as a
  stepping stone toward developing more realistic and accurate audio-visual
  generation models.
paper_authors._id: 668f41aa52a697a87e3c31e3
paper_authors.name: Manjie Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-11T13:45:44.890Z
paper_authors._id: 627df0bd72aed9ffb0a96a8a
paper_authors.avatarUrl: /avatars/a863f3cf791f3f525df6863f8d4c476b.svg
paper_authors.fullname: ManjieTsu
paper_authors.type: user
paper_authors.user: ariesssxu
paper_authors._id: 668f41aa52a697a87e3c31e4
paper_authors.name: Chenxing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:23:59.730Z
paper_authors._id: 64cc9972e60d2cddfadc5c35
paper_authors.avatarUrl: /avatars/5dac96288fa93fa965f1b57ad8d0a9b7.svg
paper_authors.fullname: Chenxing
paper_authors.type: user
paper_authors.user: Chenxinglili
paper_authors._id: 668f41aa52a697a87e3c31e5
paper_authors.name: Yong Ren
paper_authors._id: 668f41aa52a697a87e3c31e6
paper_authors.name: Rilin Chen
paper_authors._id: 668f41aa52a697a87e3c31e7
paper_authors.name: Yu Gu
paper_authors._id: 668f41aa52a697a87e3c31e8
paper_authors.name: Wei Liang
paper_authors._id: 668f41aa52a697a87e3c31e9
paper_authors.name: Dong Yu
paper_title: Video-to-Audio Generation with Hidden Alignment
paper_summary: Generating semantically and temporally aligned audio content in accordance
  with video input has become a focal point for researchers, particularly
  following the remarkable breakthrough in text-to-video generation. In this
  work, we aim to offer insights into the video-to-audio generation paradigm,
  focusing on three crucial aspects: vision encoders, auxiliary embeddings, and
  data augmentation techniques. Beginning with a foundational model VTA-LDM built
  on a simple yet surprisingly effective intuition, we explore various vision
  encoders and auxiliary embeddings through ablation studies. Employing a
  comprehensive evaluation pipeline that emphasizes generation quality and
  video-audio synchronization alignment, we demonstrate that our model exhibits
  state-of-the-art video-to-audio generation capabilities. Furthermore, we
  provide critical insights into the impact of different data augmentation
  methods on enhancing the generation framework's overall capacity. We showcase
  possibilities to advance the challenge of generating synchronized audio from
  semantic and temporal perspectives. We hope these insights will serve as a
  stepping stone toward developing more realistic and accurate audio-visual
  generation models.
paper_ai_keywords: vision encoders auxiliary embeddings data augmentation techniques VTA-LDM video-to-audio generation ablation studies encoder video-audio synchronization generation framework