title: Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
    Condition For Foley Sound
summary: Foley sound synthesis is crucial for multimedia production, enhancing user
  experience by synchronizing audio and video both temporally and semantically.
  Recent studies on automating this labor-intensive process through
  video-to-sound generation face significant challenges. Systems lacking explicit
  temporal features suffer from poor controllability and alignment, while
  timestamp-based models require costly and subjective human annotation. We
  propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
  temporal event condition with semantic timbre prompts (audio or text). RMS, a
  frame-level intensity envelope feature closely related to audio semantics,
  ensures high controllability and synchronization. The annotation-free
  self-supervised learning framework consists of two stages, Video2RMS and
  RMS2Sound, incorporating novel ideas including RMS discretization and
  RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
  shows that Video-Foley achieves state-of-the-art performance in audio-visual
  alignment and controllability for sound timing, intensity, timbre, and nuance.
  Code, model weights, and demonstrations are available on the accompanying
  website. (https://jnwnlee.github.io/video-foley-demo)
paper_authors._id: 66c7f43e9a13dc8fc2310324
paper_authors.name: Junwon Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:15:43.942Z
paper_authors._id: 6555d537f47c458e89e926cc
paper_authors.avatarUrl: /avatars/3df074b6257473e7c98fb9d97fac9196.svg
paper_authors.fullname: Junwon Lee
paper_authors.type: user
paper_authors.user: jnwnlee
paper_authors._id: 66c7f43e9a13dc8fc2310325
paper_authors.name: Jaekwon Im
paper_authors._id: 66c7f43e9a13dc8fc2310326
paper_authors.name: Dabin Kim
paper_authors._id: 66c7f43e9a13dc8fc2310327
paper_authors.name: Juhan Nam
paper_title: Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
    Condition For Foley Sound
paper_summary: Foley sound synthesis is crucial for multimedia production, enhancing user
  experience by synchronizing audio and video both temporally and semantically.
  Recent studies on automating this labor-intensive process through
  video-to-sound generation face significant challenges. Systems lacking explicit
  temporal features suffer from poor controllability and alignment, while
  timestamp-based models require costly and subjective human annotation. We
  propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
  temporal event condition with semantic timbre prompts (audio or text). RMS, a
  frame-level intensity envelope feature closely related to audio semantics,
  ensures high controllability and synchronization. The annotation-free
  self-supervised learning framework consists of two stages, Video2RMS and
  RMS2Sound, incorporating novel ideas including RMS discretization and
  RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
  shows that Video-Foley achieves state-of-the-art performance in audio-visual
  alignment and controllability for sound timing, intensity, timbre, and nuance.
  Code, model weights, and demonstrations are available on the accompanying
  website. (https://jnwnlee.github.io/video-foley-demo)
paper_ai_keywords: RMS frame-level intensity envelope temporal event condition semantic timbre prompts annotation-free self-supervised learning Video2RMS RMS2Sound RMS discretization RMS-ControlNet pretrained text-to-audio model audio-visual alignment sound timing sound intensity sound timbre sound nuance