Moonshot: Towards Controllable Video Generation and Editing with
  Multimodal Conditions
Most existing video diffusion models (VDMs) are limited to mere text
conditions. Thereby, they are usually lacking in control over visual appearance
and geometry structure of the generated videos. This work presents Moonshot, a
new video generation model that conditions simultaneously on multimodal inputs
of image and text. The model builts upon a core module, called multimodal video
block (MVB), which consists of conventional spatialtemporal layers for
representing video features, and a decoupled cross-attention layer to address
image and text inputs for appearance conditioning. In addition, we carefully
design the model architecture such that it can optionally integrate with
pre-trained image ControlNet modules for geometry visual conditions, without
needing of extra training overhead as opposed to prior methods. Experiments
show that with versatile multimodal conditioning mechanisms, Moonshot
demonstrates significant improvement on visual quality and temporal consistency
compared to existing models. In addition, the model can be easily repurposed
for a variety of generative applications, such as personalized video
generation, image animation and video editing, unveiling its potential to serve
as a fundamental architecture for controllable video generation. Models will be
made public on https://github.com/salesforce/LAVIS.
65960fe832aa91b5edb800f4
David Junhao Zhang
65960fe832aa91b5edb800f5
Dongxu Li
claimed_verified
2024-10-10T08:03:25.894Z
6357362f811ee2fa05070f64
/avatars/2cf37efb80f5cfb3e4e9d08674de6dd1.svg
Dongxu Li
user
dxli1
65960fe832aa91b5edb800f6
Hung Le
65960fe832aa91b5edb800f7
Mike Zheng Shou
65960fe832aa91b5edb800f8
Caiming Xiong
65960fe832aa91b5edb800f9
Doyen Sahoo
Moonshot: Towards Controllable Video Generation and Editing with
  Multimodal Conditions
Most existing video diffusion models (VDMs) are limited to mere text
conditions. Thereby, they are usually lacking in control over visual appearance
and geometry structure of the generated videos. This work presents Moonshot, a
new video generation model that conditions simultaneously on multimodal inputs
of image and text. The model builts upon a core module, called multimodal video
block (MVB), which consists of conventional spatialtemporal layers for
representing video features, and a decoupled cross-attention layer to address
image and text inputs for appearance conditioning. In addition, we carefully
design the model architecture such that it can optionally integrate with
pre-trained image ControlNet modules for geometry visual conditions, without
needing of extra training overhead as opposed to prior methods. Experiments
show that with versatile multimodal conditioning mechanisms, Moonshot
demonstrates significant improvement on visual quality and temporal consistency
compared to existing models. In addition, the model can be easily repurposed
for a variety of generative applications, such as personalized video
generation, image animation and video editing, unveiling its potential to serve
as a fundamental architecture for controllable video generation. Models will be
made public on https://github.com/salesforce/LAVIS.
Moonshot, a new video generation model, uses multimodal conditioning with images and text, along with an optional pre-trained ControlNet for geometry, to improve video quality and temporal consistency.
video diffusion models multimodal inputs multimodal video block spatialtemporal layers cross-attention layer pre-trained ControlNet image animation video editing