title: FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
    Sounds
summary: We study Neural Foley, the automatic generation of high-quality sound effects
  synchronizing with videos, enabling an immersive audio-visual experience.
  Despite its wide range of applications, existing approaches encounter
  limitations when it comes to simultaneously synthesizing high-quality and
  video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
  overcome these limitations, we propose FoleyCrafter, a novel framework that
  leverages a pre-trained text-to-audio model to ensure high-quality audio
  generation. FoleyCrafter comprises two key components: the semantic adapter for
  semantic alignment and the temporal controller for precise audio-video
  synchronization. The semantic adapter utilizes parallel cross-attention layers
  to condition audio generation on video features, producing realistic sound
  effects that are semantically relevant to the visual content. Meanwhile, the
  temporal controller incorporates an onset detector and a timestampbased adapter
  to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
  is its compatibility with text prompts, enabling the use of text descriptions
  to achieve controllable and diverse video-to-audio generation according to user
  intents. We conduct extensive quantitative and qualitative experiments on
  standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
  codes are available at https://github.com/open-mmlab/FoleyCrafter.
paper_authors._id: 66836152e365c0f6665bcfd2
paper_authors.name: Yiming Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:41:12.636Z
paper_authors._id: 64ccaa4687ec96aa4752e754
paper_authors.avatarUrl: /avatars/d2dd2040a521de4f55c7335cb7771c75.svg
paper_authors.fullname: Yiming Zhang
paper_authors.type: user
paper_authors.user: ymzhang319
paper_authors._id: 66836152e365c0f6665bcfd3
paper_authors.name: Yicheng Gu
paper_authors._id: 66836152e365c0f6665bcfd4
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:24:44.922Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 66836152e365c0f6665bcfd5
paper_authors.name: Zhening Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:24:53.244Z
paper_authors._id: 62fb2a9dc95d426ff8f74c8d
paper_authors.avatarUrl: /avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
paper_authors.fullname: Zhening Xing
paper_authors.type: user
paper_authors.user: Leoxing
paper_authors._id: 66836152e365c0f6665bcfd6
paper_authors.name: Yuancheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T08:25:05.005Z
paper_authors._id: 6539cc2d48b006575d3f5da2
paper_authors.avatarUrl: /avatars/a2ac1973800e8bb6a72e35fefb436a27.svg
paper_authors.fullname: wangyuancheng
paper_authors.type: user
paper_authors.user: CXKyyds199
paper_authors._id: 66836152e365c0f6665bcfd7
paper_authors.name: Zhizheng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-05T06:55:51.593Z
paper_authors._id: 63b4dcefa50cfcefdaa121f3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b4dcefa50cfcefdaa121f3/MlIxOaTSCbZARVyo8Ly7r.jpeg
paper_authors.fullname: Dr Wuz
paper_authors.type: user
paper_authors.user: drwuz
paper_authors._id: 66836152e365c0f6665bcfd8
paper_authors.name: Kai Chen
paper_title: FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
    Sounds
paper_summary: We study Neural Foley, the automatic generation of high-quality sound effects
  synchronizing with videos, enabling an immersive audio-visual experience.
  Despite its wide range of applications, existing approaches encounter
  limitations when it comes to simultaneously synthesizing high-quality and
  video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
  overcome these limitations, we propose FoleyCrafter, a novel framework that
  leverages a pre-trained text-to-audio model to ensure high-quality audio
  generation. FoleyCrafter comprises two key components: the semantic adapter for
  semantic alignment and the temporal controller for precise audio-video
  synchronization. The semantic adapter utilizes parallel cross-attention layers
  to condition audio generation on video features, producing realistic sound
  effects that are semantically relevant to the visual content. Meanwhile, the
  temporal controller incorporates an onset detector and a timestampbased adapter
  to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
  is its compatibility with text prompts, enabling the use of text descriptions
  to achieve controllable and diverse video-to-audio generation according to user
  intents. We conduct extensive quantitative and qualitative experiments on
  standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
  codes are available at https://github.com/open-mmlab/FoleyCrafter.
paper_ai_summary: FoleyCrafter, a novel framework, generates high-quality, semantically aligned, and temporally synchronized sound effects for videos using parallel cross-attention layers and a timestamp-based adapter.
paper_ai_keywords: Neural Foley text-to-audio model semantic adapter parallel cross-attention layers temporal controller onset detector timestamp-based adapter video-to-audio generation