title: Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
summary: Foley is a term commonly used in filmmaking, referring to the addition of
  daily sound effects to silent films or videos to enhance the auditory
  experience. Video-to-Audio (V2A), as a particular type of automatic foley task,
  presents inherent challenges related to audio-visual synchronization. These
  challenges encompass maintaining the content consistency between the input
  video and the generated audio, as well as the alignment of temporal and
  loudness properties within the video. To address these issues, we construct a
  controllable video-to-audio synthesis model, termed Draw an Audio, which
  supports multiple input instructions through drawn masks and loudness signals.
  To ensure content consistency between the synthesized audio and target video,
  we introduce the Mask-Attention Module (MAM), which employs masked video
  instruction to enable the model to focus on regions of interest. Additionally,
  we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness
  signal to ensure the synthesis of sound that aligns with the video in both
  loudness and temporal dimensions. Furthermore, we have extended a large-scale
  V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive
  experiments on challenging benchmarks across two large-scale V2A datasets
  verify Draw an Audio achieves the state-of-the-art. Project page:
  https://yannqi.github.io/Draw-an-Audio/.
paper_authors._id: 66e0fde4e085beb03644841d
paper_authors.name: Qi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T09:57:11.730Z
paper_authors._id: 643e45c4c639f8bc9727810a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643e45c4c639f8bc9727810a/BJR1cvSCxcqxr08iS7GsI.jpeg
paper_authors.fullname: YannQi
paper_authors.type: user
paper_authors.user: YannQi
paper_authors._id: 66e0fde4e085beb03644841e
paper_authors.name: Binjie Mao
paper_authors._id: 66e0fde4e085beb03644841f
paper_authors.name: Zili Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T07:26:17.147Z
paper_authors._id: 6410998db27543634e3aacc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678809388600-noauth.png
paper_authors.fullname: Zili Wang
paper_authors.type: user
paper_authors.user: MarkWang
paper_authors._id: 66e0fde4e085beb036448420
paper_authors.name: Xing Nie
paper_authors._id: 66e0fde4e085beb036448421
paper_authors.name: Pengfei Gao
paper_authors._id: 66e0fde4e085beb036448422
paper_authors.name: Ying Guo
paper_authors._id: 66e0fde4e085beb036448423
paper_authors.name: Cheng Zhen
paper_authors._id: 66e0fde4e085beb036448424
paper_authors.name: Pengfei Yan
paper_authors._id: 66e0fde4e085beb036448425
paper_authors.name: Shiming Xiang
paper_title: Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
paper_summary: Foley is a term commonly used in filmmaking, referring to the addition of
  daily sound effects to silent films or videos to enhance the auditory
  experience. Video-to-Audio (V2A), as a particular type of automatic foley task,
  presents inherent challenges related to audio-visual synchronization. These
  challenges encompass maintaining the content consistency between the input
  video and the generated audio, as well as the alignment of temporal and
  loudness properties within the video. To address these issues, we construct a
  controllable video-to-audio synthesis model, termed Draw an Audio, which
  supports multiple input instructions through drawn masks and loudness signals.
  To ensure content consistency between the synthesized audio and target video,
  we introduce the Mask-Attention Module (MAM), which employs masked video
  instruction to enable the model to focus on regions of interest. Additionally,
  we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness
  signal to ensure the synthesis of sound that aligns with the video in both
  loudness and temporal dimensions. Furthermore, we have extended a large-scale
  V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive
  experiments on challenging benchmarks across two large-scale V2A datasets
  verify Draw an Audio achieves the state-of-the-art. Project page:
  https://yannqi.github.io/Draw-an-Audio/.
paper_ai_keywords: video-to-audio (V2A) Mask-Attention Module (MAM) Time-Loudness Module (TLM) VGGSound-Caption