title: Training-free Guidance in Text-to-Video Generation via Multimodal
    Planning and Structured Noise Initialization
summary: Recent advancements in text-to-video (T2V) diffusion models have
  significantly enhanced the visual quality of the generated videos. However,
  even recent T2V models find it challenging to follow text descriptions
  accurately, especially when the prompt requires accurate control of spatial
  layouts or object trajectories. A recent line of research uses layout guidance
  for T2V models that require fine-tuning or iterative manipulation of the
  attention map during inference time. This significantly increases the memory
  requirement, making it difficult to adopt a large T2V model as a backbone. To
  address this, we introduce Video-MSG, a training-free Guidance method for T2V
  generation based on Multimodal planning and Structured noise initialization.
  Video-MSG consists of three steps, where in the first two steps, Video-MSG
  creates Video Sketch, a fine-grained spatio-temporal plan for the final video,
  specifying background, foreground, and object trajectories, in the form of
  draft video frames. In the last step, Video-MSG guides a downstream T2V
  diffusion model with Video Sketch through noise inversion and denoising.
  Notably, Video-MSG does not need fine-tuning or attention manipulation with
  additional memory during inference time, making it easier to adopt large T2V
  models. Video-MSG demonstrates its effectiveness in enhancing text alignment
  with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V
  generation benchmarks (T2VCompBench and VBench). We provide comprehensive
  ablation studies about noise inversion ratio, different background generators,
  background object detection, and foreground object segmentation.
paper_authors._id: 67fd26108a74846fc0ff3065
paper_authors.name: Jialu Li
paper_authors._id: 67fd26108a74846fc0ff3066
paper_authors.name: Shoubin Yu
paper_authors._id: 67fd26108a74846fc0ff3067
paper_authors.name: Han Lin
paper_authors._id: 67fd26108a74846fc0ff3068
paper_authors.name: Jaemin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-24T09:11:27.939Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 67fd26108a74846fc0ff3069
paper_authors.name: Jaehong Yoon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T17:08:21.863Z
paper_authors._id: 652066649004117947e46ed6
paper_authors.avatarUrl: /avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
paper_authors.fullname: Jaehong Yoon
paper_authors.type: user
paper_authors.user: jaehong31
paper_authors._id: 67fd26108a74846fc0ff306a
paper_authors.name: Mohit Bansal
paper_title: Training-free Guidance in Text-to-Video Generation via Multimodal
    Planning and Structured Noise Initialization
paper_summary: Recent advancements in text-to-video (T2V) diffusion models have
  significantly enhanced the visual quality of the generated videos. However,
  even recent T2V models find it challenging to follow text descriptions
  accurately, especially when the prompt requires accurate control of spatial
  layouts or object trajectories. A recent line of research uses layout guidance
  for T2V models that require fine-tuning or iterative manipulation of the
  attention map during inference time. This significantly increases the memory
  requirement, making it difficult to adopt a large T2V model as a backbone. To
  address this, we introduce Video-MSG, a training-free Guidance method for T2V
  generation based on Multimodal planning and Structured noise initialization.
  Video-MSG consists of three steps, where in the first two steps, Video-MSG
  creates Video Sketch, a fine-grained spatio-temporal plan for the final video,
  specifying background, foreground, and object trajectories, in the form of
  draft video frames. In the last step, Video-MSG guides a downstream T2V
  diffusion model with Video Sketch through noise inversion and denoising.
  Notably, Video-MSG does not need fine-tuning or attention manipulation with
  additional memory during inference time, making it easier to adopt large T2V
  models. Video-MSG demonstrates its effectiveness in enhancing text alignment
  with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V
  generation benchmarks (T2VCompBench and VBench). We provide comprehensive
  ablation studies about noise inversion ratio, different background generators,
  background object detection, and foreground object segmentation.
paper_ai_summary: Video-MSG enhances text-to-video generation by creating a video sketch for guidance without additional memory or fine-tuning requirements.
paper_ai_keywords: text-to-video (T2V) diffusion models layout guidance attention map inference time multimodal planning structured noise initialization Video Sketch spatio-temporal plan draft video frames noise inversion denoising VideoCrafter2 CogVideoX-5B T2VCompBench VBench