Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound
<|reference_start|>Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound: Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video both temporally and semantically. Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges. Systems lacking explicit temporal features suffer from poor controllability and alignment, while timestamp-based models require costly and subjective human annotation. We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a temporal event condition with semantic timbre prompts (audio or text). RMS, a frame-level intensity envelope feature closely related to audio semantics, ensures high controllability and synchronization. The annotation-free self-supervised learning framework consists of two stages, Video2RMS and RMS2Sound, incorporating novel ideas including RMS discretization and RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation shows that Video-Foley achieves state-of-the-art performance in audio-visual alignment and controllability for sound timing, intensity, timbre, and nuance. Code, model weights, and demonstrations are available on the accompanying website. (https://jnwnlee.github.io/video-foley-demo)<|reference_end|>
@article{lee2024video-foley:,
    title={Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
  Condition For Foley Sound},
    author={Junwon Lee, Jaekwon Im, Dabin Kim, Juhan Nam},
    journal={arXiv preprint arXiv:2408.11915},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.11915},
    primaryClass={cs.SD cs.CV cs.LG cs.MM eess.AS}
}