Towards A Better Metric for Text-to-Video Generation
Generative models have demonstrated remarkable capability in synthesizing
high-quality text, images, and videos. For video generation, contemporary
text-to-video models exhibit impressive capabilities, crafting visually
stunning videos. Nonetheless, evaluating such videos poses significant
challenges. Current research predominantly employs automated metrics such as
FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis,
particularly in the temporal assessment of video content, thus rendering them
unreliable indicators of true video quality. Furthermore, while user studies
have the potential to reflect human perception accurately, they are hampered by
their time-intensive and laborious nature, with outcomes that are often tainted
by subjective bias. In this paper, we investigate the limitations inherent in
existing metrics and introduce a novel evaluation pipeline, the Text-to-Video
Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video
Alignment, which scrutinizes the fidelity of the video in representing the
given text description, and (2) Video Quality, which evaluates the video's
overall production caliber with a mixture of experts. Moreover, to evaluate the
proposed metrics and facilitate future improvements on them, we present the
TVGE dataset, collecting human judgements of 2,543 text-to-video generated
videos on the two criteria. Experiments on the TVGE dataset demonstrate the
superiority of the proposed T2VScore on offering a better metric for
text-to-video generation.
65a76a9db4b9c261f7213bef
Jay Zhangjie Wu
claimed_verified
2024-01-17T08:48:59.173Z
633aaf695df91da9cea92960
https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
Jay Wu
user
jayw
65a76a9db4b9c261f7213bf0
Guian Fang
65a76a9db4b9c261f7213bf1
Haoning Wu
claimed_verified
2024-01-18T11:54:12.422Z
63047ed2412a1b9d381b09c9
https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
Haoning Wu, Teo
user
teowu
65a76a9db4b9c261f7213bf2
Xintao Wang
admin_assigned
2024-01-17T09:34:43.604Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
65a76a9db4b9c261f7213bf3
Yixiao Ge
admin_assigned
2024-01-17T09:34:54.861Z
640e9762b03f4cd29f58d982
/avatars/81da37d628163fe3e094b247c7c3a3b5.svg
Yixiao Ge
user
yxgeee
65a76a9db4b9c261f7213bf4
Xiaodong Cun
admin_assigned
2024-01-17T09:35:06.208Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
65a76a9db4b9c261f7213bf5
David Junhao Zhang
admin_assigned
2024-01-17T09:35:12.573Z
6468855763a564ba347e1efa
/avatars/caa99c3318c1d6d21be44d2d32795a62.svg
David Junhao ZHANG
user
Junhao233
65a76a9db4b9c261f7213bf6
Jia-Wei Liu
admin_assigned
2024-01-17T09:35:31.822Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
65a76a9db4b9c261f7213bf7
Yuchao Gu
admin_assigned
2024-01-17T09:35:59.418Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
65a76a9db4b9c261f7213bf8
Rui Zhao
admin_assigned
2024-01-17T09:36:28.581Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
65a76a9db4b9c261f7213bf9
Weisi Lin
65a76a9db4b9c261f7213bfa
Wynne Hsu
65a76a9db4b9c261f7213bfb
Ying Shan
admin_assigned
2024-01-17T09:37:06.242Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
65a76a9db4b9c261f7213bfc
Mike Zheng Shou
admin_assigned
2024-01-17T09:38:02.872Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
Towards A Better Metric for Text-to-Video Generation
Generative models have demonstrated remarkable capability in synthesizing
high-quality text, images, and videos. For video generation, contemporary
text-to-video models exhibit impressive capabilities, crafting visually
stunning videos. Nonetheless, evaluating such videos poses significant
challenges. Current research predominantly employs automated metrics such as
FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis,
particularly in the temporal assessment of video content, thus rendering them
unreliable indicators of true video quality. Furthermore, while user studies
have the potential to reflect human perception accurately, they are hampered by
their time-intensive and laborious nature, with outcomes that are often tainted
by subjective bias. In this paper, we investigate the limitations inherent in
existing metrics and introduce a novel evaluation pipeline, the Text-to-Video
Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video
Alignment, which scrutinizes the fidelity of the video in representing the
given text description, and (2) Video Quality, which evaluates the video's
overall production caliber with a mixture of experts. Moreover, to evaluate the
proposed metrics and facilitate future improvements on them, we present the
TVGE dataset, collecting human judgements of 2,543 text-to-video generated
videos on the two criteria. Experiments on the TVGE dataset demonstrate the
superiority of the proposed T2VScore on offering a better metric for
text-to-video generation.
A new metric, T2VScore, is introduced to evaluate text-to-video generation by combining text-video alignment and video quality, addressing the limitations of current automated metrics.
generative models text-to-video models FVD IS CLIP Score Text-Video Alignment Video Quality TVGE dataset