title: Moonshot: Towards Controllable Video Generation and Editing with
    Multimodal Conditions
summary: Most existing video diffusion models (VDMs) are limited to mere text
  conditions. Thereby, they are usually lacking in control over visual appearance
  and geometry structure of the generated videos. This work presents Moonshot, a
  new video generation model that conditions simultaneously on multimodal inputs
  of image and text. The model builts upon a core module, called multimodal video
  block (MVB), which consists of conventional spatialtemporal layers for
  representing video features, and a decoupled cross-attention layer to address
  image and text inputs for appearance conditioning. In addition, we carefully
  design the model architecture such that it can optionally integrate with
  pre-trained image ControlNet modules for geometry visual conditions, without
  needing of extra training overhead as opposed to prior methods. Experiments
  show that with versatile multimodal conditioning mechanisms, Moonshot
  demonstrates significant improvement on visual quality and temporal consistency
  compared to existing models. In addition, the model can be easily repurposed
  for a variety of generative applications, such as personalized video
  generation, image animation and video editing, unveiling its potential to serve
  as a fundamental architecture for controllable video generation. Models will be
  made public on https://github.com/salesforce/LAVIS.
paper_authors._id: 65960fe832aa91b5edb800f4
paper_authors.name: David Junhao Zhang
paper_authors._id: 65960fe832aa91b5edb800f5
paper_authors.name: Dongxu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:03:25.894Z
paper_authors._id: 6357362f811ee2fa05070f64
paper_authors.avatarUrl: /avatars/2cf37efb80f5cfb3e4e9d08674de6dd1.svg
paper_authors.fullname: Dongxu Li
paper_authors.type: user
paper_authors.user: dxli1
paper_authors._id: 65960fe832aa91b5edb800f6
paper_authors.name: Hung Le
paper_authors._id: 65960fe832aa91b5edb800f7
paper_authors.name: Mike Zheng Shou
paper_authors._id: 65960fe832aa91b5edb800f8
paper_authors.name: Caiming Xiong
paper_authors._id: 65960fe832aa91b5edb800f9
paper_authors.name: Doyen Sahoo
paper_title: Moonshot: Towards Controllable Video Generation and Editing with
    Multimodal Conditions
paper_summary: Most existing video diffusion models (VDMs) are limited to mere text
  conditions. Thereby, they are usually lacking in control over visual appearance
  and geometry structure of the generated videos. This work presents Moonshot, a
  new video generation model that conditions simultaneously on multimodal inputs
  of image and text. The model builts upon a core module, called multimodal video
  block (MVB), which consists of conventional spatialtemporal layers for
  representing video features, and a decoupled cross-attention layer to address
  image and text inputs for appearance conditioning. In addition, we carefully
  design the model architecture such that it can optionally integrate with
  pre-trained image ControlNet modules for geometry visual conditions, without
  needing of extra training overhead as opposed to prior methods. Experiments
  show that with versatile multimodal conditioning mechanisms, Moonshot
  demonstrates significant improvement on visual quality and temporal consistency
  compared to existing models. In addition, the model can be easily repurposed
  for a variety of generative applications, such as personalized video
  generation, image animation and video editing, unveiling its potential to serve
  as a fundamental architecture for controllable video generation. Models will be
  made public on https://github.com/salesforce/LAVIS.
paper_ai_summary: Moonshot, a new video generation model, uses multimodal conditioning with images and text, along with an optional pre-trained ControlNet for geometry, to improve video quality and temporal consistency.
paper_ai_keywords: video diffusion models multimodal inputs multimodal video block spatialtemporal layers cross-attention layer pre-trained ControlNet image animation video editing