Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
Foley is a term commonly used in filmmaking, referring to the addition of
daily sound effects to silent films or videos to enhance the auditory
experience. Video-to-Audio (V2A), as a particular type of automatic foley task,
presents inherent challenges related to audio-visual synchronization. These
challenges encompass maintaining the content consistency between the input
video and the generated audio, as well as the alignment of temporal and
loudness properties within the video. To address these issues, we construct a
controllable video-to-audio synthesis model, termed Draw an Audio, which
supports multiple input instructions through drawn masks and loudness signals.
To ensure content consistency between the synthesized audio and target video,
we introduce the Mask-Attention Module (MAM), which employs masked video
instruction to enable the model to focus on regions of interest. Additionally,
we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness
signal to ensure the synthesis of sound that aligns with the video in both
loudness and temporal dimensions. Furthermore, we have extended a large-scale
V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive
experiments on challenging benchmarks across two large-scale V2A datasets
verify Draw an Audio achieves the state-of-the-art. Project page:
https://yannqi.github.io/Draw-an-Audio/.
66e0fde4e085beb03644841d
Qi Yang
claimed_verified
2024-09-11T09:57:11.730Z
643e45c4c639f8bc9727810a
https://cdn-avatars.huggingface.co/v1/production/uploads/643e45c4c639f8bc9727810a/BJR1cvSCxcqxr08iS7GsI.jpeg
YannQi
user
YannQi
66e0fde4e085beb03644841e
Binjie Mao
66e0fde4e085beb03644841f
Zili Wang
claimed_verified
2024-09-11T07:26:17.147Z
6410998db27543634e3aacc3
https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
Zili Wang
user
MarkWang
66e0fde4e085beb036448420
Xing Nie
66e0fde4e085beb036448421
Pengfei Gao
66e0fde4e085beb036448422
Ying Guo
66e0fde4e085beb036448423
Cheng Zhen
66e0fde4e085beb036448424
Pengfei Yan
66e0fde4e085beb036448425
Shiming Xiang
Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
Foley is a term commonly used in filmmaking, referring to the addition of
daily sound effects to silent films or videos to enhance the auditory
experience. Video-to-Audio (V2A), as a particular type of automatic foley task,
presents inherent challenges related to audio-visual synchronization. These
challenges encompass maintaining the content consistency between the input
video and the generated audio, as well as the alignment of temporal and
loudness properties within the video. To address these issues, we construct a
controllable video-to-audio synthesis model, termed Draw an Audio, which
supports multiple input instructions through drawn masks and loudness signals.
To ensure content consistency between the synthesized audio and target video,
we introduce the Mask-Attention Module (MAM), which employs masked video
instruction to enable the model to focus on regions of interest. Additionally,
we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness
signal to ensure the synthesis of sound that aligns with the video in both
loudness and temporal dimensions. Furthermore, we have extended a large-scale
V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive
experiments on challenging benchmarks across two large-scale V2A datasets
verify Draw an Audio achieves the state-of-the-art. Project page:
https://yannqi.github.io/Draw-an-Audio/.
A controllable video-to-audio synthesis model, Draw an Audio, uses masked instructions and loudness signals for audio-visual synchronization in challenging V2A tasks.
V2A video-to-audio Mask-Attention Module MAM Time-Loudness Module TLM VGGSound-Caption