title: Masked Generative Video-to-Audio Transformers with Enhanced
    Synchronicity
summary: Video-to-audio (V2A) generation leverages visual-only video features to
  render plausible sounds that match the scene. Importantly, the generated sound
  onsets should match the visual actions that are aligned with them, otherwise
  unnatural synchronization artifacts arise. Recent works have explored the
  progression of conditioning sound generators on still images and then video
  features, focusing on quality and semantic matching while ignoring
  synchronization, or by sacrificing some amount of quality to focus on improving
  synchronization only. In this work, we propose a V2A generative model, named
  MaskVAT, that interconnects a full-band high-quality general audio codec with a
  sequence-to-sequence masked generative model. This combination allows modeling
  both high audio quality, semantic matching, and temporal synchronicity at the
  same time. Our results show that, by combining a high-quality codec with the
  proper pre-trained audio-visual features and a sequence-to-sequence parallel
  structure, we are able to yield highly synchronized results on one hand, whilst
  being competitive with the state of the art of non-codec generative audio
  models. Sample videos and generated audios are available at
  https://maskvat.github.io .
paper_authors._id: 6695ec67659a52fa274d8939
paper_authors.name: Santiago Pascual
paper_authors._id: 6695ec67659a52fa274d893a
paper_authors.name: Chunghsin Yeh
paper_authors._id: 6695ec67659a52fa274d893b
paper_authors.name: Ioannis Tsiamas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:08:01.245Z
paper_authors._id: 667572848c58c5a12a18692d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/667572848c58c5a12a18692d/V-mtLKoOLdIedcSTYZCjn.jpeg
paper_authors.fullname: Ioannis Tsiamas
paper_authors.type: user
paper_authors.user: johntsi
paper_authors._id: 6695ec67659a52fa274d893c
paper_authors.name: Joan Serrà
paper_title: Masked Generative Video-to-Audio Transformers with Enhanced
    Synchronicity
paper_summary: Video-to-audio (V2A) generation leverages visual-only video features to
  render plausible sounds that match the scene. Importantly, the generated sound
  onsets should match the visual actions that are aligned with them, otherwise
  unnatural synchronization artifacts arise. Recent works have explored the
  progression of conditioning sound generators on still images and then video
  features, focusing on quality and semantic matching while ignoring
  synchronization, or by sacrificing some amount of quality to focus on improving
  synchronization only. In this work, we propose a V2A generative model, named
  MaskVAT, that interconnects a full-band high-quality general audio codec with a
  sequence-to-sequence masked generative model. This combination allows modeling
  both high audio quality, semantic matching, and temporal synchronicity at the
  same time. Our results show that, by combining a high-quality codec with the
  proper pre-trained audio-visual features and a sequence-to-sequence parallel
  structure, we are able to yield highly synchronized results on one hand, whilst
  being competitive with the state of the art of non-codec generative audio
  models. Sample videos and generated audios are available at
  https://maskvat.github.io .
paper_ai_summary: MaskVAT implements a high-quality codec with a sequence-to-sequence masked generative model to produce synchronized and semantically accurate audio from visual video inputs.
paper_ai_keywords: video-to-audio generation full-band high-quality audio codec sequence-to-sequence masked generative model temporal synchronicity pre-trained audio-visual features