Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing
  Reliability,Reproducibility, and Practicality
Recent text-to-video (T2V) technology advancements, as demonstrated by models
such as Gen2, Pika, and Sora, have significantly broadened its applicability
and popularity. Despite these strides, evaluating these models poses
substantial challenges. Primarily, due to the limitations inherent in automatic
metrics, manual evaluation is often considered a superior method for assessing
T2V generation. However, existing manual evaluation protocols face
reproducibility, reliability, and practicality issues. To address these
challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
protocol includes well-defined metrics, thorough annotator training, and an
effective dynamic evaluation module. Experimental results demonstrate that this
protocol not only ensures high-quality annotations but can also reduce
evaluation costs by nearly 50%. We will open-source the entire setup of the
T2VHE protocol, including the complete protocol workflow, the dynamic
evaluation component details, and the annotation interface code. This will help
communities establish more sophisticated human assessment protocols.
666fe04bc16d62cbb0462c4e
Tianle Zhang
666fe04bc16d62cbb0462c4f
Langtian Ma
666fe04bc16d62cbb0462c50
Yuchen Yan
666fe04bc16d62cbb0462c51
Yuchen Zhang
666fe04bc16d62cbb0462c52
Kai Wang
666fe04bc16d62cbb0462c53
Yue Yang
666fe04bc16d62cbb0462c54
Ziyao Guo
666fe04bc16d62cbb0462c55
Wenqi Shao
666fe04bc16d62cbb0462c56
Yang You
666fe04bc16d62cbb0462c57
Yu Qiao
666fe04bc16d62cbb0462c58
Ping Luo
666fe04bc16d62cbb0462c59
Kaipeng Zhang
admin_assigned
2024-06-17T14:16:41.682Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing
  Reliability,Reproducibility, and Practicality
Recent text-to-video (T2V) technology advancements, as demonstrated by models
such as Gen2, Pika, and Sora, have significantly broadened its applicability
and popularity. Despite these strides, evaluating these models poses
substantial challenges. Primarily, due to the limitations inherent in automatic
metrics, manual evaluation is often considered a superior method for assessing
T2V generation. However, existing manual evaluation protocols face
reproducibility, reliability, and practicality issues. To address these
challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
protocol includes well-defined metrics, thorough annotator training, and an
effective dynamic evaluation module. Experimental results demonstrate that this
protocol not only ensures high-quality annotations but can also reduce
evaluation costs by nearly 50%. We will open-source the entire setup of the
T2VHE protocol, including the complete protocol workflow, the dynamic
evaluation component details, and the annotation interface code. This will help
communities establish more sophisticated human assessment protocols.
A new standardized Text-to-Video Human Evaluation (T2VHE) protocol improves manual assessment of text-to-video models in terms of quality and cost.
text-to-video T2V Gen2 Pika Sora automatic metrics manual evaluation T2VHE protocol dynamic evaluation module annotations