title: Compositional Video Generation as Flow Equalization
summary: Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated
  unprecedented capability to transform natural language descriptions into
  stunning and photorealistic videos. Despite the promising results, a
  significant challenge remains: these models struggle to fully grasp complex
  compositional interactions between multiple concepts and actions. This issue
  arises when some words dominantly influence the final video, overshadowing
  other concepts.To tackle this problem, we introduce Vico, a generic
  framework for compositional video generation that explicitly ensures all
  concepts are represented properly. At its core, Vico analyzes how input tokens
  influence the generated video, and adjusts the model to prevent any single
  concept from dominating. Specifically, Vico extracts attention weights from all
  layers to build a spatial-temporal attention graph, and then estimates the
  influence as the max-flow from the source text token to the video target
  token. Although the direct computation of attention flow in diffusion models is
  typically infeasible, we devise an efficient approximation based on subgraph
  flows and employ a fast and vectorized implementation, which in turn makes the
  flow computation manageable and differentiable. By updating the noisy latent to
  balance these flows, Vico captures complex interactions and consequently
  produces videos that closely adhere to textual descriptions. We apply our
  method to multiple diffusion-based video models for compositional T2V and video
  editing. Empirical results demonstrate that our framework significantly
  enhances the compositional richness and accuracy of the generated videos. Visit
  our website
  at~https://adamdad.github.io/vico/{https://adamdad.github.io/vico/}.
paper_authors._id: 668c9d937dbe0ab21d9a65f3
paper_authors.name: Xingyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:07:35.186Z
paper_authors._id: 634cfebc350bcee9bed20a4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634cfebc350bcee9bed20a4d/fN47nN5rhw-HJaFLBZWQy.png
paper_authors.fullname: Xingyi Yang
paper_authors.type: user
paper_authors.user: adamdad
paper_authors._id: 668c9d937dbe0ab21d9a65f4
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T21:08:16.858Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: Compositional Video Generation as Flow Equalization
paper_summary: Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated
  unprecedented capability to transform natural language descriptions into
  stunning and photorealistic videos. Despite the promising results, a
  significant challenge remains: these models struggle to fully grasp complex
  compositional interactions between multiple concepts and actions. This issue
  arises when some words dominantly influence the final video, overshadowing
  other concepts.To tackle this problem, we introduce Vico, a generic
  framework for compositional video generation that explicitly ensures all
  concepts are represented properly. At its core, Vico analyzes how input tokens
  influence the generated video, and adjusts the model to prevent any single
  concept from dominating. Specifically, Vico extracts attention weights from all
  layers to build a spatial-temporal attention graph, and then estimates the
  influence as the max-flow from the source text token to the video target
  token. Although the direct computation of attention flow in diffusion models is
  typically infeasible, we devise an efficient approximation based on subgraph
  flows and employ a fast and vectorized implementation, which in turn makes the
  flow computation manageable and differentiable. By updating the noisy latent to
  balance these flows, Vico captures complex interactions and consequently
  produces videos that closely adhere to textual descriptions. We apply our
  method to multiple diffusion-based video models for compositional T2V and video
  editing. Empirical results demonstrate that our framework significantly
  enhances the compositional richness and accuracy of the generated videos. Visit
  our website
  at~https://adamdad.github.io/vico/{https://adamdad.github.io/vico/}.
paper_ai_summary: Vico is a framework that enhances text-to-video diffusion models by ensuring balanced representation of all concepts in the generated video through spatial-temporal attention graphs and max-flow approximations.
paper_ai_keywords: diffusion models compositional video generation input tokens attention weights spatial-temporal attention graph max-flow subgraph flows vectorized implementation noisy latent compositional T2V video editing