FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
  Sounds
We study Neural Foley, the automatic generation of high-quality sound effects
synchronizing with videos, enabling an immersive audio-visual experience.
Despite its wide range of applications, existing approaches encounter
limitations when it comes to simultaneously synthesizing high-quality and
video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
overcome these limitations, we propose FoleyCrafter, a novel framework that
leverages a pre-trained text-to-audio model to ensure high-quality audio
generation. FoleyCrafter comprises two key components: the semantic adapter for
semantic alignment and the temporal controller for precise audio-video
synchronization. The semantic adapter utilizes parallel cross-attention layers
to condition audio generation on video features, producing realistic sound
effects that are semantically relevant to the visual content. Meanwhile, the
temporal controller incorporates an onset detector and a timestampbased adapter
to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
is its compatibility with text prompts, enabling the use of text descriptions
to achieve controllable and diverse video-to-audio generation according to user
intents. We conduct extensive quantitative and qualitative experiments on
standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
codes are available at https://github.com/open-mmlab/FoleyCrafter.
66836152e365c0f6665bcfd2
Yiming Zhang
claimed_verified
2024-07-03T07:41:12.636Z
64ccaa4687ec96aa4752e754
/avatars/d2dd2040a521de4f55c7335cb7771c75.svg
Yiming Zhang
user
ymzhang319
66836152e365c0f6665bcfd3
Yicheng Gu
66836152e365c0f6665bcfd4
Yanhong Zeng
admin_assigned
2024-07-03T08:24:44.922Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
66836152e365c0f6665bcfd5
Zhening Xing
admin_assigned
2024-07-03T08:24:53.244Z
62fb2a9dc95d426ff8f74c8d
/avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
Zhening Xing
user
Leoxing
66836152e365c0f6665bcfd6
Yuancheng Wang
admin_assigned
2024-07-03T08:25:05.005Z
6539cc2d48b006575d3f5da2
/avatars/a2ac1973800e8bb6a72e35fefb436a27.svg
wangyuancheng
user
CXKyyds199
66836152e365c0f6665bcfd7
Zhizheng Wu
claimed_verified
2024-07-05T06:55:51.593Z
63b4dcefa50cfcefdaa121f3
https://cdn-avatars.huggingface.co/v1/production/uploads/63b4dcefa50cfcefdaa121f3/MlIxOaTSCbZARVyo8Ly7r.jpeg
Dr Wuz
user
drwuz
66836152e365c0f6665bcfd8
Kai Chen
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
  Sounds
We study Neural Foley, the automatic generation of high-quality sound effects
synchronizing with videos, enabling an immersive audio-visual experience.
Despite its wide range of applications, existing approaches encounter
limitations when it comes to simultaneously synthesizing high-quality and
video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
overcome these limitations, we propose FoleyCrafter, a novel framework that
leverages a pre-trained text-to-audio model to ensure high-quality audio
generation. FoleyCrafter comprises two key components: the semantic adapter for
semantic alignment and the temporal controller for precise audio-video
synchronization. The semantic adapter utilizes parallel cross-attention layers
to condition audio generation on video features, producing realistic sound
effects that are semantically relevant to the visual content. Meanwhile, the
temporal controller incorporates an onset detector and a timestampbased adapter
to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
is its compatibility with text prompts, enabling the use of text descriptions
to achieve controllable and diverse video-to-audio generation according to user
intents. We conduct extensive quantitative and qualitative experiments on
standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
codes are available at https://github.com/open-mmlab/FoleyCrafter.
FoleyCrafter, a novel framework, generates high-quality, semantically aligned, and temporally synchronized sound effects for videos using parallel cross-attention layers and a timestamp-based adapter.
Neural Foley text-to-audio model semantic adapter parallel cross-attention layers temporal controller onset detector timestamp-based adapter video-to-audio generation