title: Taming Multimodal Joint Training for High-Quality Video-to-Audio
    Synthesis
summary: We propose to synthesize high-quality and synchronized audio, given video and
  optional text conditions, using a novel multimodal joint training framework
  MMAudio. In contrast to single-modality training conditioned on (limited) video
  data only, MMAudio is jointly trained with larger-scale, readily available
  text-audio data to learn to generate semantically aligned high-quality audio
  samples. Additionally, we improve audio-visual synchrony with a conditional
  synchronization module that aligns video conditions with audio latents at the
  frame level. Trained with a flow matching objective, MMAudio achieves new
  video-to-audio state-of-the-art among public models in terms of audio quality,
  semantic alignment, and audio-visual synchronization, while having a low
  inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio
  also achieves surprisingly competitive performance in text-to-audio generation,
  showing that joint training does not hinder single-modality performance. Code
  and demo are available at: https://hkchengrex.github.io/MMAudio
paper_authors._id: 6768d18826eb881162077014
paper_authors.name: Ho Kei Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:09:58.995Z
paper_authors._id: 63041b541dd5d3c62486c294
paper_authors.avatarUrl: /avatars/a5286d562f7b9082730f760e66c3bf29.svg
paper_authors.fullname: Ho Kei Cheng
paper_authors.type: user
paper_authors.user: hkchengrex
paper_authors._id: 6768d18826eb881162077015
paper_authors.name: Masato Ishii
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:33:45.656Z
paper_authors._id: 674545617e0ea169b0c471d1
paper_authors.avatarUrl: /avatars/a422e3efa5fb1c3f2c6c0997c412b088.svg
paper_authors.fullname: Masato Ishii
paper_authors.type: user
paper_authors.user: mi141
paper_authors._id: 6768d18826eb881162077016
paper_authors.name: Akio Hayakawa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:17.816Z
paper_authors._id: 66d54244b4396d43c356ea2e
paper_authors.avatarUrl: /avatars/c90dde293cbda625db73b35d8f191328.svg
paper_authors.fullname: Akio Hayakawa
paper_authors.type: user
paper_authors.user: AkHyk
paper_authors._id: 6768d18826eb881162077017
paper_authors.name: Takashi Shibuya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:09:56.433Z
paper_authors._id: 6650773ca6acfdd2aba7d486
paper_authors.avatarUrl: /avatars/d297886ea60dbff98a043caf825820ed.svg
paper_authors.fullname: Takashi Shibuya
paper_authors.type: user
paper_authors.user: TakashiShibuyaSony
paper_authors._id: 6768d18826eb881162077018
paper_authors.name: Alexander Schwing
paper_authors._id: 6768d18826eb881162077019
paper_authors.name: Yuki Mitsufuji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:34:10.186Z
paper_authors._id: 665e32384ecc8a7181634f6d
paper_authors.avatarUrl: /avatars/8752f952010540d14f45eac849e91371.svg
paper_authors.fullname: Yuki Mitsufuji
paper_authors.type: user
paper_authors.user: mittu1204
paper_title: Taming Multimodal Joint Training for High-Quality Video-to-Audio
    Synthesis
paper_summary: We propose to synthesize high-quality and synchronized audio, given video and
  optional text conditions, using a novel multimodal joint training framework
  MMAudio. In contrast to single-modality training conditioned on (limited) video
  data only, MMAudio is jointly trained with larger-scale, readily available
  text-audio data to learn to generate semantically aligned high-quality audio
  samples. Additionally, we improve audio-visual synchrony with a conditional
  synchronization module that aligns video conditions with audio latents at the
  frame level. Trained with a flow matching objective, MMAudio achieves new
  video-to-audio state-of-the-art among public models in terms of audio quality,
  semantic alignment, and audio-visual synchronization, while having a low
  inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio
  also achieves surprisingly competitive performance in text-to-audio generation,
  showing that joint training does not hinder single-modality performance. Code
  and demo are available at: https://hkchengrex.github.io/MMAudio
paper_ai_summary: A multimodal framework jointly trains with video and text to generate high-quality, semantically aligned, and visually synchronized audio, achieving state-of-the-art performance.
paper_ai_keywords: multimodal joint training MMAudio text-audio data semantic alignment conditional synchronization module audio latents frame level flow matching objective audio quality audio-visual synchronization text-to-audio generation low inference time