title: Temporally Aligned Audio for Video with Autoregression
summary: We introduce V-AURA, the first autoregressive model to achieve high temporal
  alignment and relevance in video-to-audio generation. V-AURA uses a
  high-framerate visual feature extractor and a cross-modal audio-visual feature
  fusion strategy to capture fine-grained visual motion events and ensure precise
  temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
  with high audio-visual relevance. VisualSound is based on VGGSound, a video
  dataset consisting of in-the-wild samples extracted from YouTube. During the
  curation, we remove samples where auditory events are not aligned with the
  visual ones. V-AURA outperforms current state-of-the-art models in temporal
  alignment and semantic relevance while maintaining comparable audio quality.
  Code, samples, VisualSound and models are available at
  https://v-aura.notion.site
paper_authors._id: 66f11acd16bffdccd567e57c
paper_authors.name: Ilpo Viertola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:44:42.556Z
paper_authors._id: 64b7c3d5f72c6cd946dff515
paper_authors.avatarUrl: /avatars/f579bc83b529782bbe6109f18d89837a.svg
paper_authors.fullname: Ilpo Viertola
paper_authors.type: user
paper_authors.user: bilpo
paper_authors._id: 66f11acd16bffdccd567e57d
paper_authors.name: Vladimir Iashin
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-23T07:37:50.179Z
paper_authors._id: 64b7c3d5f72c6cd946dff515
paper_authors.avatarUrl: /avatars/f579bc83b529782bbe6109f18d89837a.svg
paper_authors.fullname: Ilpo Viertola
paper_authors.type: user
paper_authors.user: bilpo
paper_authors._id: 66f11acd16bffdccd567e57e
paper_authors.name: Esa Rahtu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-09-23T07:37:50.179Z
paper_authors._id: 651eb538fff19a2bc3aa2f1f
paper_authors.avatarUrl: /avatars/8332a6803a5ae3c7bf5d7f9aa1e35983.svg
paper_authors.fullname: Esa Rahtu
paper_authors.type: user
paper_authors.user: erahtu
paper_title: Temporally Aligned Audio for Video with Autoregression
paper_summary: We introduce V-AURA, the first autoregressive model to achieve high temporal
  alignment and relevance in video-to-audio generation. V-AURA uses a
  high-framerate visual feature extractor and a cross-modal audio-visual feature
  fusion strategy to capture fine-grained visual motion events and ensure precise
  temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
  with high audio-visual relevance. VisualSound is based on VGGSound, a video
  dataset consisting of in-the-wild samples extracted from YouTube. During the
  curation, we remove samples where auditory events are not aligned with the
  visual ones. V-AURA outperforms current state-of-the-art models in temporal
  alignment and semantic relevance while maintaining comparable audio quality.
  Code, samples, VisualSound and models are available at
  https://v-aura.notion.site
paper_ai_keywords: autoregressive model temporal alignment relevance video-to-audio generation high-framerate visual feature extractor cross-modal audio-visual feature fusion fine-grained visual motion events precise temporal alignment benchmark dataset VGGSound audio-visual relevance semantic relevance auditory events