VideoRepair: Improving Text-to-Video Generation via Misalignment
  Evaluation and Localized Refinement
Recent text-to-video (T2V) diffusion models have demonstrated impressive
generation capabilities across various domains. However, these models often
generate videos that have misalignments with text prompts, especially when the
prompts describe complex scenes with multiple objects and attributes. To
address this, we introduce VideoRepair, a novel model-agnostic, training-free
video refinement framework that automatically identifies fine-grained
text-video misalignments and generates explicit spatial and textual feedback,
enabling a T2V diffusion model to perform targeted, localized refinements.
VideoRepair consists of four stages: In (1) video evaluation, we detect
misalignments by generating fine-grained evaluation questions and answering
those questions with MLLM. In (2) refinement planning, we identify accurately
generated objects and then create localized prompts to refine other areas in
the video. Next, in (3) region decomposition, we segment the correctly
generated area using a combined grounding module. We regenerate the video by
adjusting the misaligned regions while preserving the correct regions in (4)
localized refinement. On two popular video generation benchmarks (EvalCrafter
and T2V-CompBench), VideoRepair substantially outperforms recent baselines
across various text-video alignment metrics. We provide a comprehensive
analysis of VideoRepair components and qualitative examples.
6743d80aa05eb026d723f89f
Daeun Lee
claimed_verified
2024-11-26T21:21:26.100Z
647585665aba8edfb2f393d7
/avatars/5f3282a7f4e11ee967006e580183f3f8.svg
Daeun Lee
user
danaleee
6743d80aa05eb026d723f8a0
Jaehong Yoon
claimed_verified
2024-11-25T20:39:20.260Z
652066649004117947e46ed6
/avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
Jaehong Yoon
user
jaehong31
6743d80aa05eb026d723f8a1
Jaemin Cho
claimed_verified
2024-11-25T08:34:20.962Z
5ffe32d8942cf3533d364449
https://cdn-avatars.huggingface.co/v1/production/uploads/1654821969191-5ffe32d8942cf3533d364449.jpeg
Jaemin Cho
user
j-min
6743d80aa05eb026d723f8a2
Mohit Bansal
VideoRepair: Improving Text-to-Video Generation via Misalignment
  Evaluation and Localized Refinement
Recent text-to-video (T2V) diffusion models have demonstrated impressive
generation capabilities across various domains. However, these models often
generate videos that have misalignments with text prompts, especially when the
prompts describe complex scenes with multiple objects and attributes. To
address this, we introduce VideoRepair, a novel model-agnostic, training-free
video refinement framework that automatically identifies fine-grained
text-video misalignments and generates explicit spatial and textual feedback,
enabling a T2V diffusion model to perform targeted, localized refinements.
VideoRepair consists of four stages: In (1) video evaluation, we detect
misalignments by generating fine-grained evaluation questions and answering
those questions with MLLM. In (2) refinement planning, we identify accurately
generated objects and then create localized prompts to refine other areas in
the video. Next, in (3) region decomposition, we segment the correctly
generated area using a combined grounding module. We regenerate the video by
adjusting the misaligned regions while preserving the correct regions in (4)
localized refinement. On two popular video generation benchmarks (EvalCrafter
and T2V-CompBench), VideoRepair substantially outperforms recent baselines
across various text-video alignment metrics. We provide a comprehensive
analysis of VideoRepair components and qualitative examples.
VideoRepair is a model-agnostic framework that refines text-to-video generation by identifying misalignments and providing localized feedback, improving alignment metrics on video generation benchmarks.
T2V diffusion models video refinement fine-grained text-video misalignments MLLM refinement planning region decomposition grounding module localized refinement EvalCrafter T2V-CompBench text-video alignment metrics