title: VideoRepair: Improving Text-to-Video Generation via Misalignment
    Evaluation and Localized Refinement
summary: Recent text-to-video (T2V) diffusion models have demonstrated impressive
  generation capabilities across various domains. However, these models often
  generate videos that have misalignments with text prompts, especially when the
  prompts describe complex scenes with multiple objects and attributes. To
  address this, we introduce VideoRepair, a novel model-agnostic, training-free
  video refinement framework that automatically identifies fine-grained
  text-video misalignments and generates explicit spatial and textual feedback,
  enabling a T2V diffusion model to perform targeted, localized refinements.
  VideoRepair consists of four stages: In (1) video evaluation, we detect
  misalignments by generating fine-grained evaluation questions and answering
  those questions with MLLM. In (2) refinement planning, we identify accurately
  generated objects and then create localized prompts to refine other areas in
  the video. Next, in (3) region decomposition, we segment the correctly
  generated area using a combined grounding module. We regenerate the video by
  adjusting the misaligned regions while preserving the correct regions in (4)
  localized refinement. On two popular video generation benchmarks (EvalCrafter
  and T2V-CompBench), VideoRepair substantially outperforms recent baselines
  across various text-video alignment metrics. We provide a comprehensive
  analysis of VideoRepair components and qualitative examples.
paper_authors._id: 6743d80aa05eb026d723f89f
paper_authors.name: Daeun Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:21:26.100Z
paper_authors._id: 647585665aba8edfb2f393d7
paper_authors.avatarUrl: /avatars/5f3282a7f4e11ee967006e580183f3f8.svg
paper_authors.fullname: Daeun Lee
paper_authors.type: user
paper_authors.user: danaleee
paper_authors._id: 6743d80aa05eb026d723f8a0
paper_authors.name: Jaehong Yoon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T20:39:20.260Z
paper_authors._id: 652066649004117947e46ed6
paper_authors.avatarUrl: /avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
paper_authors.fullname: Jaehong Yoon
paper_authors.type: user
paper_authors.user: jaehong31
paper_authors._id: 6743d80aa05eb026d723f8a1
paper_authors.name: Jaemin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-25T08:34:20.962Z
paper_authors._id: 5ffe32d8942cf3533d364449
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
paper_authors.fullname: Jaemin Cho
paper_authors.type: user
paper_authors.user: j-min
paper_authors._id: 6743d80aa05eb026d723f8a2
paper_authors.name: Mohit Bansal
paper_title: VideoRepair: Improving Text-to-Video Generation via Misalignment
    Evaluation and Localized Refinement
paper_summary: Recent text-to-video (T2V) diffusion models have demonstrated impressive
  generation capabilities across various domains. However, these models often
  generate videos that have misalignments with text prompts, especially when the
  prompts describe complex scenes with multiple objects and attributes. To
  address this, we introduce VideoRepair, a novel model-agnostic, training-free
  video refinement framework that automatically identifies fine-grained
  text-video misalignments and generates explicit spatial and textual feedback,
  enabling a T2V diffusion model to perform targeted, localized refinements.
  VideoRepair consists of four stages: In (1) video evaluation, we detect
  misalignments by generating fine-grained evaluation questions and answering
  those questions with MLLM. In (2) refinement planning, we identify accurately
  generated objects and then create localized prompts to refine other areas in
  the video. Next, in (3) region decomposition, we segment the correctly
  generated area using a combined grounding module. We regenerate the video by
  adjusting the misaligned regions while preserving the correct regions in (4)
  localized refinement. On two popular video generation benchmarks (EvalCrafter
  and T2V-CompBench), VideoRepair substantially outperforms recent baselines
  across various text-video alignment metrics. We provide a comprehensive
  analysis of VideoRepair components and qualitative examples.
paper_ai_summary: VideoRepair is a model-agnostic framework that refines text-to-video generation by identifying misalignments and providing localized feedback, improving alignment metrics on video generation benchmarks.
paper_ai_keywords: T2V diffusion models video refinement fine-grained text-video misalignments MLLM refinement planning region decomposition grounding module localized refinement EvalCrafter T2V-CompBench text-video alignment metrics