title: VIMI: Grounding Video Generation through Multi-modal Instruction
summary: Existing text-to-video diffusion models rely solely on text-only encoders for
  their pretraining. This limitation stems from the absence of large-scale
  multimodal prompt video datasets, resulting in a lack of visual grounding and
  restricting their versatility and application in multimodal integration. To
  address this, we construct a large-scale multimodal prompt dataset by employing
  retrieval methods to pair in-context examples with the given text prompts and
  then utilize a two-stage training strategy to enable diverse video generation
  tasks within the same model. In the first stage, we propose a multimodal
  conditional video generation framework for pretraining on these augmented
  datasets, establishing a foundational model for grounded video generation.
  Secondly, we finetune the model from the first stage on three video generation
  tasks, incorporating multi-modal instructions. This process further refines the
  model's ability to handle diverse inputs and tasks, ensuring seamless
  integration of multi-modal information. After this two-stage train-ing process,
  VIMI demonstrates multimodal understanding capabilities, producing contextually
  rich and personalized videos grounded in the provided inputs, as shown in
  Figure 1. Compared to previous visual grounded video generation methods, VIMI
  can synthesize consistent and temporally coherent videos with large motion
  while retaining the semantic control. Lastly, VIMI also achieves
  state-of-the-art text-to-video generation results on UCF101 benchmark.
paper_authors._id: 668df26e3d0bc7c475c42380
paper_authors.name: Yuwei Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:48:35.613Z
paper_authors._id: 6406d1dcc3ab325efa9db198
paper_authors.avatarUrl: /avatars/781f3cd455596f7cffa0906cc487f79b.svg
paper_authors.fullname: Yuwei Fang
paper_authors.type: user
paper_authors.user: studyfang
paper_authors._id: 668df26e3d0bc7c475c42381
paper_authors.name: Willi Menapace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:48:41.447Z
paper_authors._id: 6315358a362e3e95ea538081
paper_authors.avatarUrl: /avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
paper_authors.fullname: Willi Menapace
paper_authors.type: user
paper_authors.user: willi-menapace
paper_authors._id: 668df26e3d0bc7c475c42382
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:48:46.539Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 668df26e3d0bc7c475c42383
paper_authors.name: Tsai-Shien Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:48:51.931Z
paper_authors._id: 6330f570b68c7453d2ee8945
paper_authors.avatarUrl: /avatars/5b394a72f277aa6c886b21b477010ab0.svg
paper_authors.fullname: Tsai-Shien Chen
paper_authors.type: user
paper_authors.user: tschen
paper_authors._id: 668df26e3d0bc7c475c42384
paper_authors.name: Kuan-Chien Wang
paper_authors._id: 668df26e3d0bc7c475c42385
paper_authors.name: Ivan Skorokhodov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:49:11.388Z
paper_authors._id: 63610db13c7147fae7de88e3
paper_authors.avatarUrl: /avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
paper_authors.fullname: Ivan Skorokhodov
paper_authors.type: user
paper_authors.user: universome
paper_authors._id: 668df26e3d0bc7c475c42386
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-10T10:49:17.329Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 668df26e3d0bc7c475c42387
paper_authors.name: Sergey Tulyakov
paper_title: VIMI: Grounding Video Generation through Multi-modal Instruction
paper_summary: Existing text-to-video diffusion models rely solely on text-only encoders for
  their pretraining. This limitation stems from the absence of large-scale
  multimodal prompt video datasets, resulting in a lack of visual grounding and
  restricting their versatility and application in multimodal integration. To
  address this, we construct a large-scale multimodal prompt dataset by employing
  retrieval methods to pair in-context examples with the given text prompts and
  then utilize a two-stage training strategy to enable diverse video generation
  tasks within the same model. In the first stage, we propose a multimodal
  conditional video generation framework for pretraining on these augmented
  datasets, establishing a foundational model for grounded video generation.
  Secondly, we finetune the model from the first stage on three video generation
  tasks, incorporating multi-modal instructions. This process further refines the
  model's ability to handle diverse inputs and tasks, ensuring seamless
  integration of multi-modal information. After this two-stage train-ing process,
  VIMI demonstrates multimodal understanding capabilities, producing contextually
  rich and personalized videos grounded in the provided inputs, as shown in
  Figure 1. Compared to previous visual grounded video generation methods, VIMI
  can synthesize consistent and temporally coherent videos with large motion
  while retaining the semantic control. Lastly, VIMI also achieves
  state-of-the-art text-to-video generation results on UCF101 benchmark.
paper_ai_keywords: diffusion models pretraining multimodal datasets retrieval methods two-stage training strategy multimodal conditional video generation framework finetuning multi-modal instructions multimodal understanding capabilities contextually rich temporally coherent semantic control UCF101 benchmark