title: Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
    Adaptation
summary: We consider the task of generating diverse and realistic videos guided by
  natural audio samples from a wide variety of semantic classes. For this task,
  the videos are required to be aligned both globally and temporally with the
  input audio: globally, the input audio is semantically associated with the
  entire output video, and temporally, each segment of the input audio is
  associated with a corresponding segment of that video. We utilize an existing
  text-conditioned video generation model and a pre-trained audio encoder model.
  The proposed method is based on a lightweight adaptor network, which learns to
  map the audio-based representation to the input representation expected by the
  text-to-video generation model. As such, it also enables video generation
  conditioned on text, audio, and, for the first time as far as we can ascertain,
  on both text and audio. We validate our method extensively on three datasets
  demonstrating significant semantic diversity of audio-video samples and further
  propose a novel evaluation metric (AV-Align) to assess the alignment of
  generated videos with input audio samples. AV-Align is based on the detection
  and comparison of energy peaks in both modalities. In comparison to recent
  state-of-the-art approaches, our method generates videos that are better
  aligned with the input sound, both with respect to content and temporal axis.
  We also show that videos produced by our method present higher visual quality
  and are more diverse.
paper_authors._id: 65163fa0c661dc405bd8d841
paper_authors.name: Guy Yariv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-29T07:04:21.968Z
paper_authors._id: 646d239f4220471ca0c6471c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
paper_authors.fullname: Guy Yariv
paper_authors.type: user
paper_authors.user: GuyYariv
paper_authors._id: 65163fa0c661dc405bd8d842
paper_authors.name: Itai Gat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:48:21.716Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 65163fa0c661dc405bd8d843
paper_authors.name: Sagie Benaim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:48:30.102Z
paper_authors._id: 6345a9b9a8c2ff9f1377faab
paper_authors.avatarUrl: /avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
paper_authors.fullname: Sagie Benaim
paper_authors.type: user
paper_authors.user: sagiebenaim
paper_authors._id: 65163fa0c661dc405bd8d844
paper_authors.name: Lior Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:48:52.763Z
paper_authors._id: 62f9f170363251ee40a08046
paper_authors.avatarUrl: /avatars/67ad459db44b605be0c1f6bd33363d6e.svg
paper_authors.fullname: Lior Wolf
paper_authors.type: user
paper_authors.user: liorwolf
paper_authors._id: 65163fa0c661dc405bd8d845
paper_authors.name: Idan Schwartz
paper_authors._id: 65163fa0c661dc405bd8d846
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:49:11.717Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_title: Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model
    Adaptation
paper_summary: We consider the task of generating diverse and realistic videos guided by
  natural audio samples from a wide variety of semantic classes. For this task,
  the videos are required to be aligned both globally and temporally with the
  input audio: globally, the input audio is semantically associated with the
  entire output video, and temporally, each segment of the input audio is
  associated with a corresponding segment of that video. We utilize an existing
  text-conditioned video generation model and a pre-trained audio encoder model.
  The proposed method is based on a lightweight adaptor network, which learns to
  map the audio-based representation to the input representation expected by the
  text-to-video generation model. As such, it also enables video generation
  conditioned on text, audio, and, for the first time as far as we can ascertain,
  on both text and audio. We validate our method extensively on three datasets
  demonstrating significant semantic diversity of audio-video samples and further
  propose a novel evaluation metric (AV-Align) to assess the alignment of
  generated videos with input audio samples. AV-Align is based on the detection
  and comparison of energy peaks in both modalities. In comparison to recent
  state-of-the-art approaches, our method generates videos that are better
  aligned with the input sound, both with respect to content and temporal axis.
  We also show that videos produced by our method present higher visual quality
  and are more diverse.
paper_ai_summary: A lightweight adaptor network improves video generation conditioned on audio and text by aligning audio segments with video segments and enhancing alignment and diversity.
paper_ai_keywords: text-conditioned video generation model pre-trained audio encoder model lightweight adaptor network audio-based representation input representation text-to-video generation model AV-Align energy peaks detection and comparison audio-video samples semantic diversity visual quality