title: Towards A Better Metric for Text-to-Video Generation
summary: Generative models have demonstrated remarkable capability in synthesizing
  high-quality text, images, and videos. For video generation, contemporary
  text-to-video models exhibit impressive capabilities, crafting visually
  stunning videos. Nonetheless, evaluating such videos poses significant
  challenges. Current research predominantly employs automated metrics such as
  FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis,
  particularly in the temporal assessment of video content, thus rendering them
  unreliable indicators of true video quality. Furthermore, while user studies
  have the potential to reflect human perception accurately, they are hampered by
  their time-intensive and laborious nature, with outcomes that are often tainted
  by subjective bias. In this paper, we investigate the limitations inherent in
  existing metrics and introduce a novel evaluation pipeline, the Text-to-Video
  Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video
  Alignment, which scrutinizes the fidelity of the video in representing the
  given text description, and (2) Video Quality, which evaluates the video's
  overall production caliber with a mixture of experts. Moreover, to evaluate the
  proposed metrics and facilitate future improvements on them, we present the
  TVGE dataset, collecting human judgements of 2,543 text-to-video generated
  videos on the two criteria. Experiments on the TVGE dataset demonstrate the
  superiority of the proposed T2VScore on offering a better metric for
  text-to-video generation.
paper_authors._id: 65a76a9db4b9c261f7213bef
paper_authors.name: Jay Zhangjie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-17T08:48:59.173Z
paper_authors._id: 633aaf695df91da9cea92960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
paper_authors.fullname: Jay Wu
paper_authors.type: user
paper_authors.user: jayw
paper_authors._id: 65a76a9db4b9c261f7213bf0
paper_authors.name: Guian Fang
paper_authors._id: 65a76a9db4b9c261f7213bf1
paper_authors.name: Haoning Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T11:54:12.422Z
paper_authors._id: 63047ed2412a1b9d381b09c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
paper_authors.fullname: Haoning Wu, Teo
paper_authors.type: user
paper_authors.user: teowu
paper_authors._id: 65a76a9db4b9c261f7213bf2
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:34:43.604Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65a76a9db4b9c261f7213bf3
paper_authors.name: Yixiao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:34:54.861Z
paper_authors._id: 640e9762b03f4cd29f58d982
paper_authors.avatarUrl: /avatars/81da37d628163fe3e094b247c7c3a3b5.svg
paper_authors.fullname: Yixiao Ge
paper_authors.type: user
paper_authors.user: yxgeee
paper_authors._id: 65a76a9db4b9c261f7213bf4
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:35:06.208Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 65a76a9db4b9c261f7213bf5
paper_authors.name: David Junhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:35:12.573Z
paper_authors._id: 6468855763a564ba347e1efa
paper_authors.avatarUrl: /avatars/caa99c3318c1d6d21be44d2d32795a62.svg
paper_authors.fullname: David Junhao ZHANG
paper_authors.type: user
paper_authors.user: Junhao233
paper_authors._id: 65a76a9db4b9c261f7213bf6
paper_authors.name: Jia-Wei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:35:31.822Z
paper_authors._id: 64230048a73327caad9d8241
paper_authors.avatarUrl: /avatars/ac115f1a21c743db7c925f0e18451145.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: JiaweiLIU
paper_authors._id: 65a76a9db4b9c261f7213bf7
paper_authors.name: Yuchao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:35:59.418Z
paper_authors._id: 63021630a35b21bd8a53305a
paper_authors.avatarUrl: /avatars/7a7e8b39749eda61e57d8a1908726558.svg
paper_authors.fullname: Gu Yuchao
paper_authors.type: user
paper_authors.user: guyuchao
paper_authors._id: 65a76a9db4b9c261f7213bf8
paper_authors.name: Rui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:36:28.581Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 65a76a9db4b9c261f7213bf9
paper_authors.name: Weisi Lin
paper_authors._id: 65a76a9db4b9c261f7213bfa
paper_authors.name: Wynne Hsu
paper_authors._id: 65a76a9db4b9c261f7213bfb
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:37:06.242Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 65a76a9db4b9c261f7213bfc
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:38:02.872Z
paper_authors._id: 63a55320ce5763e06f78519c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
paper_authors.fullname: Mike Shou
paper_authors.type: user
paper_authors.user: mikeshou
paper_title: Towards A Better Metric for Text-to-Video Generation
paper_summary: Generative models have demonstrated remarkable capability in synthesizing
  high-quality text, images, and videos. For video generation, contemporary
  text-to-video models exhibit impressive capabilities, crafting visually
  stunning videos. Nonetheless, evaluating such videos poses significant
  challenges. Current research predominantly employs automated metrics such as
  FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis,
  particularly in the temporal assessment of video content, thus rendering them
  unreliable indicators of true video quality. Furthermore, while user studies
  have the potential to reflect human perception accurately, they are hampered by
  their time-intensive and laborious nature, with outcomes that are often tainted
  by subjective bias. In this paper, we investigate the limitations inherent in
  existing metrics and introduce a novel evaluation pipeline, the Text-to-Video
  Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video
  Alignment, which scrutinizes the fidelity of the video in representing the
  given text description, and (2) Video Quality, which evaluates the video's
  overall production caliber with a mixture of experts. Moreover, to evaluate the
  proposed metrics and facilitate future improvements on them, we present the
  TVGE dataset, collecting human judgements of 2,543 text-to-video generated
  videos on the two criteria. Experiments on the TVGE dataset demonstrate the
  superiority of the proposed T2VScore on offering a better metric for
  text-to-video generation.
paper_ai_summary: A new metric, T2VScore, is introduced to evaluate text-to-video generation by combining text-video alignment and video quality, addressing the limitations of current automated metrics.
paper_ai_keywords: generative models text-to-video models FVD IS CLIP Score Text-Video Alignment Video Quality TVGE dataset