title: FoleyGen: Visually-Guided Audio Generation
summary: Recent advancements in audio generation have been spurred by the evolution of
  large-scale deep learning models and expansive datasets. However, the task of
  video-to-audio (V2A) generation continues to be a challenge, principally
  because of the intricate relationship between the high-dimensional visual and
  auditory data, and the challenges associated with temporal synchronization. In
  this study, we introduce FoleyGen, an open-domain V2A generation system built
  on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural
  audio codec for bidirectional conversion between waveforms and discrete tokens.
  The generation of audio tokens is facilitated by a single Transformer model,
  which is conditioned on visual features extracted from a visual encoder. A
  prevalent problem in V2A generation is the misalignment of generated audio with
  the visible actions in the video. To address this, we explore three novel
  visual attention mechanisms. We further undertake an exhaustive evaluation of
  multiple visual encoders, each pretrained on either single-modal or multi-modal
  tasks. The experimental results on VGGSound dataset show that our proposed
  FoleyGen outperforms previous systems across all objective metrics and human
  evaluations.
paper_authors._id: 650a5260e0850b3ff055a141
paper_authors.name: Xinhao Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:07:02.821Z
paper_authors._id: 626c5f8af451470f862534c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626c5f8af451470f862534c6/qL5ubYstO0Won5VOFHviF.jpeg
paper_authors.fullname: Xinhao Mei
paper_authors.type: user
paper_authors.user: Xinhao
paper_authors._id: 650a5260e0850b3ff055a142
paper_authors.name: Varun Nagaraja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:07:39.205Z
paper_authors._id: 6435e32a2d0ed796668f9700
paper_authors.avatarUrl: /avatars/ef85b2f33d85433eb88e033c302fc68a.svg
paper_authors.fullname: Varun Nagaraja
paper_authors.type: user
paper_authors.user: varun-nagaraja
paper_authors._id: 650a5260e0850b3ff055a143
paper_authors.name: Gael Le Lan
paper_authors._id: 650a5260e0850b3ff055a144
paper_authors.name: Zhaoheng Ni
paper_authors._id: 650a5260e0850b3ff055a145
paper_authors.name: Ernie Chang
paper_authors._id: 650a5260e0850b3ff055a146
paper_authors.name: Yangyang Shi
paper_authors._id: 650a5260e0850b3ff055a147
paper_authors.name: Vikas Chandra
paper_title: FoleyGen: Visually-Guided Audio Generation
paper_summary: Recent advancements in audio generation have been spurred by the evolution of
  large-scale deep learning models and expansive datasets. However, the task of
  video-to-audio (V2A) generation continues to be a challenge, principally
  because of the intricate relationship between the high-dimensional visual and
  auditory data, and the challenges associated with temporal synchronization. In
  this study, we introduce FoleyGen, an open-domain V2A generation system built
  on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural
  audio codec for bidirectional conversion between waveforms and discrete tokens.
  The generation of audio tokens is facilitated by a single Transformer model,
  which is conditioned on visual features extracted from a visual encoder. A
  prevalent problem in V2A generation is the misalignment of generated audio with
  the visible actions in the video. To address this, we explore three novel
  visual attention mechanisms. We further undertake an exhaustive evaluation of
  multiple visual encoders, each pretrained on either single-modal or multi-modal
  tasks. The experimental results on VGGSound dataset show that our proposed
  FoleyGen outperforms previous systems across all objective metrics and human
  evaluations.
paper_ai_keywords: FoleyGen language modeling paradigm neural audio codec bidirectional conversion waveforms discrete tokens Transformer model visual features visual encoder visual attention mechanisms single-modal tasks multi-modal tasks VGGSound dataset