title: Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing
    Reliability,Reproducibility, and Practicality
summary: Recent text-to-video (T2V) technology advancements, as demonstrated by models
  such as Gen2, Pika, and Sora, have significantly broadened its applicability
  and popularity. Despite these strides, evaluating these models poses
  substantial challenges. Primarily, due to the limitations inherent in automatic
  metrics, manual evaluation is often considered a superior method for assessing
  T2V generation. However, existing manual evaluation protocols face
  reproducibility, reliability, and practicality issues. To address these
  challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
  protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
  protocol includes well-defined metrics, thorough annotator training, and an
  effective dynamic evaluation module. Experimental results demonstrate that this
  protocol not only ensures high-quality annotations but can also reduce
  evaluation costs by nearly 50%. We will open-source the entire setup of the
  T2VHE protocol, including the complete protocol workflow, the dynamic
  evaluation component details, and the annotation interface code. This will help
  communities establish more sophisticated human assessment protocols.
paper_authors._id: 666fe04bc16d62cbb0462c4e
paper_authors.name: Tianle Zhang
paper_authors._id: 666fe04bc16d62cbb0462c4f
paper_authors.name: Langtian Ma
paper_authors._id: 666fe04bc16d62cbb0462c50
paper_authors.name: Yuchen Yan
paper_authors._id: 666fe04bc16d62cbb0462c51
paper_authors.name: Yuchen Zhang
paper_authors._id: 666fe04bc16d62cbb0462c52
paper_authors.name: Kai Wang
paper_authors._id: 666fe04bc16d62cbb0462c53
paper_authors.name: Yue Yang
paper_authors._id: 666fe04bc16d62cbb0462c54
paper_authors.name: Ziyao Guo
paper_authors._id: 666fe04bc16d62cbb0462c55
paper_authors.name: Wenqi Shao
paper_authors._id: 666fe04bc16d62cbb0462c56
paper_authors.name: Yang You
paper_authors._id: 666fe04bc16d62cbb0462c57
paper_authors.name: Yu Qiao
paper_authors._id: 666fe04bc16d62cbb0462c58
paper_authors.name: Ping Luo
paper_authors._id: 666fe04bc16d62cbb0462c59
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:16:41.682Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_title: Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing
    Reliability,Reproducibility, and Practicality
paper_summary: Recent text-to-video (T2V) technology advancements, as demonstrated by models
  such as Gen2, Pika, and Sora, have significantly broadened its applicability
  and popularity. Despite these strides, evaluating these models poses
  substantial challenges. Primarily, due to the limitations inherent in automatic
  metrics, manual evaluation is often considered a superior method for assessing
  T2V generation. However, existing manual evaluation protocols face
  reproducibility, reliability, and practicality issues. To address these
  challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
  protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
  protocol includes well-defined metrics, thorough annotator training, and an
  effective dynamic evaluation module. Experimental results demonstrate that this
  protocol not only ensures high-quality annotations but can also reduce
  evaluation costs by nearly 50%. We will open-source the entire setup of the
  T2VHE protocol, including the complete protocol workflow, the dynamic
  evaluation component details, and the annotation interface code. This will help
  communities establish more sophisticated human assessment protocols.
paper_ai_summary: A new standardized Text-to-Video Human Evaluation (T2VHE) protocol improves manual assessment of text-to-video models in terms of quality and cost.
paper_ai_keywords: text-to-video T2V Gen2 Pika Sora automatic metrics manual evaluation T2VHE protocol dynamic evaluation module annotations