title: Video-Guided Foley Sound Generation with Multimodal Controls
summary: Generating sound effects for videos often requires creating artistic sound
  effects that diverge significantly from real-life sources and flexible control
  in the sound design. To address this problem, we introduce MultiFoley, a model
  designed for video-guided sound generation that supports multimodal
  conditioning through text, audio, and video. Given a silent video and a text
  prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels
  spinning without wind noise) or more whimsical sounds (e.g., making a lion's
  roar sound like a cat's meow). MultiFoley also allows users to choose reference
  audio from sound effects (SFX) libraries or partial videos for conditioning. A
  key novelty of our model lies in its joint training on both internet video
  datasets with low-quality audio and professional SFX recordings, enabling
  high-quality, full-bandwidth (48kHz) audio generation. Through automated
  evaluations and human studies, we demonstrate that MultiFoley successfully
  generates synchronized high-quality sounds across varied conditional inputs and
  outperforms existing methods. Please see our project page for video results:
  https://ificl.github.io/MultiFoley/
paper_authors._id: 6748776100cf30c297e7d286
paper_authors.name: Ziyang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T16:05:44.842Z
paper_authors._id: 6542bf99899592bf11847315
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6542bf99899592bf11847315/ll0tyrWbbymrhYKOZxRHX.jpeg
paper_authors.fullname: Ziyang Chen
paper_authors.type: user
paper_authors.user: czyang
paper_authors._id: 6748776100cf30c297e7d287
paper_authors.name: Prem Seetharaman
paper_authors._id: 6748776100cf30c297e7d288
paper_authors.name: Bryan Russell
paper_authors._id: 6748776100cf30c297e7d289
paper_authors.name: Oriol Nieto
paper_authors._id: 6748776100cf30c297e7d28a
paper_authors.name: David Bourgin
paper_authors._id: 6748776100cf30c297e7d28b
paper_authors.name: Andrew Owens
paper_authors._id: 6748776100cf30c297e7d28c
paper_authors.name: Justin Salamon
paper_title: Video-Guided Foley Sound Generation with Multimodal Controls
paper_summary: Generating sound effects for videos often requires creating artistic sound
  effects that diverge significantly from real-life sources and flexible control
  in the sound design. To address this problem, we introduce MultiFoley, a model
  designed for video-guided sound generation that supports multimodal
  conditioning through text, audio, and video. Given a silent video and a text
  prompt, MultiFoley allows users to create clean sounds (e.g., skateboard wheels
  spinning without wind noise) or more whimsical sounds (e.g., making a lion's
  roar sound like a cat's meow). MultiFoley also allows users to choose reference
  audio from sound effects (SFX) libraries or partial videos for conditioning. A
  key novelty of our model lies in its joint training on both internet video
  datasets with low-quality audio and professional SFX recordings, enabling
  high-quality, full-bandwidth (48kHz) audio generation. Through automated
  evaluations and human studies, we demonstrate that MultiFoley successfully
  generates synchronized high-quality sounds across varied conditional inputs and
  outperforms existing methods. Please see our project page for video results:
  https://ificl.github.io/MultiFoley/
paper_ai_summary: MultiFoley generates high-quality synchronized sounds for videos using text, audio, and video inputs, outperforming existing methods.
paper_ai_keywords: MultFoley multimodal conditioning video-guided sound generation internet video datasets professional SFX recordings full-bandwidth audio generation