Temporally Aligned Audio for Video with Autoregression
We introduce V-AURA, the first autoregressive model to achieve high temporal
alignment and relevance in video-to-audio generation. V-AURA uses a
high-framerate visual feature extractor and a cross-modal audio-visual feature
fusion strategy to capture fine-grained visual motion events and ensure precise
temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
with high audio-visual relevance. VisualSound is based on VGGSound, a video
dataset consisting of in-the-wild samples extracted from YouTube. During the
curation, we remove samples where auditory events are not aligned with the
visual ones. V-AURA outperforms current state-of-the-art models in temporal
alignment and semantic relevance while maintaining comparable audio quality.
Code, samples, VisualSound and models are available at
https://v-aura.notion.site
66f11acd16bffdccd567e57c
Ilpo Viertola
admin_assigned
2024-09-23T07:44:42.556Z
64b7c3d5f72c6cd946dff515
/avatars/f579bc83b529782bbe6109f18d89837a.svg
Ilpo Viertola
user
bilpo
66f11acd16bffdccd567e57d
Vladimir Iashin
extracted_pending
2024-09-23T07:37:50.179Z
64b7c3d5f72c6cd946dff515
/avatars/f579bc83b529782bbe6109f18d89837a.svg
Ilpo Viertola
user
bilpo
66f11acd16bffdccd567e57e
Esa Rahtu
extracted_pending
2024-09-23T07:37:50.179Z
651eb538fff19a2bc3aa2f1f
/avatars/8332a6803a5ae3c7bf5d7f9aa1e35983.svg
Esa Rahtu
user
erahtu
Temporally Aligned Audio for Video with Autoregression
We introduce V-AURA, the first autoregressive model to achieve high temporal
alignment and relevance in video-to-audio generation. V-AURA uses a
high-framerate visual feature extractor and a cross-modal audio-visual feature
fusion strategy to capture fine-grained visual motion events and ensure precise
temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
with high audio-visual relevance. VisualSound is based on VGGSound, a video
dataset consisting of in-the-wild samples extracted from YouTube. During the
curation, we remove samples where auditory events are not aligned with the
visual ones. V-AURA outperforms current state-of-the-art models in temporal
alignment and semantic relevance while maintaining comparable audio quality.
Code, samples, VisualSound and models are available at
https://v-aura.notion.site
V-AURA, an autoregressive model, achieves high temporal alignment and relevance in video-to-audio generation using a high-framerate visual feature extractor and cross-modal feature fusion, outperforming existing models while maintaining audio quality.
autoregressive model temporal alignment video-to-audio generation high-framerate visual feature extractor cross-modal audio-visual feature fusion VisualSound VGGSound