title: ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question
    Generation and Answering
summary: Precisely evaluating semantic alignment between text prompts and generated
  videos remains a challenge in Text-to-Video (T2V) Generation. Existing
  text-to-video alignment metrics like CLIPScore only generate coarse-grained
  scores without fine-grained alignment details, failing to align with human
  preference. To address this limitation, we propose ETVA, a novel Evaluation
  method of Text-to-Video Alignment via fine-grained question generation and
  answering. First, a multi-agent system parses prompts into semantic scene
  graphs to generate atomic questions. Then we design a knowledge-augmented
  multi-stage reasoning framework for question answering, where an auxiliary LLM
  first retrieves relevant common-sense knowledge (e.g., physical laws), and then
  video LLM answers the generated questions through a multi-stage reasoning
  mechanism. Extensive experiments demonstrate that ETVA achieves a Spearman's
  correlation coefficient of 58.47, showing a much higher correlation with human
  judgment than existing metrics which attain only 31.0. We also construct a
  comprehensive benchmark specifically designed for text-to-video alignment
  evaluation, featuring 2k diverse prompts and 12k atomic questions spanning 10
  categories. Through a systematic evaluation of 15 existing text-to-video
  models, we identify their key capabilities and limitations, paving the way for
  next-generation T2V generation.
paper_authors._id: 67e0d31f151ca9ed92898fff
paper_authors.name: Kaisi Guan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:06:57.786Z
paper_authors._id: 63bbf071d8d676a2299c7d0b
paper_authors.avatarUrl: /avatars/4bb1c86ef8651c75b9761afee2865267.svg
paper_authors.fullname: Guan
paper_authors.type: user
paper_authors.user: Guan123
paper_authors._id: 67e0d31f151ca9ed92899000
paper_authors.name: Zhengfeng Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:49:46.588Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 67e0d31f151ca9ed92899001
paper_authors.name: Yuchong Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:49:53.004Z
paper_authors._id: 645e0f1aa45b4182d7f69176
paper_authors.avatarUrl: /avatars/6b87f203e781b6ec734d2b58084ab013.svg
paper_authors.fullname: yuchong sun
paper_authors.type: user
paper_authors.user: ycsun
paper_authors._id: 67e0d31f151ca9ed92899002
paper_authors.name: Peng Zhang
paper_authors._id: 67e0d31f151ca9ed92899003
paper_authors.name: Wei Liu
paper_authors._id: 67e0d31f151ca9ed92899004
paper_authors.name: Kieran Liu
paper_authors._id: 67e0d31f151ca9ed92899005
paper_authors.name: Meng Cao
paper_authors._id: 67e0d31f151ca9ed92899006
paper_authors.name: Ruihua Song
paper_title: ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question
    Generation and Answering
paper_summary: Precisely evaluating semantic alignment between text prompts and generated
  videos remains a challenge in Text-to-Video (T2V) Generation. Existing
  text-to-video alignment metrics like CLIPScore only generate coarse-grained
  scores without fine-grained alignment details, failing to align with human
  preference. To address this limitation, we propose ETVA, a novel Evaluation
  method of Text-to-Video Alignment via fine-grained question generation and
  answering. First, a multi-agent system parses prompts into semantic scene
  graphs to generate atomic questions. Then we design a knowledge-augmented
  multi-stage reasoning framework for question answering, where an auxiliary LLM
  first retrieves relevant common-sense knowledge (e.g., physical laws), and then
  video LLM answers the generated questions through a multi-stage reasoning
  mechanism. Extensive experiments demonstrate that ETVA achieves a Spearman's
  correlation coefficient of 58.47, showing a much higher correlation with human
  judgment than existing metrics which attain only 31.0. We also construct a
  comprehensive benchmark specifically designed for text-to-video alignment
  evaluation, featuring 2k diverse prompts and 12k atomic questions spanning 10
  categories. Through a systematic evaluation of 15 existing text-to-video
  models, we identify their key capabilities and limitations, paving the way for
  next-generation T2V generation.
paper_ai_summary: ETVA, a novel evaluation method for text-to-video alignment, uses fine-grained question generation and answering to provide detailed alignment, significantly outperforming existing metrics like CLIPScore.
paper_ai_keywords: ETVA Evaluation method Text-to-Video Alignment fine-grained question generation question answering multi-agent system semantic scene graphs knowledge-augmented multi-stage reasoning auxiliary LLM video LLM multi-stage reasoning Spearman's correlation coefficient text-to-video alignment benchmark diverse prompts atomic questions text-to-video models