title: MantisScore: Building Automatic Metrics to Simulate Fine-grained Human
    Feedback for Video Generation
summary: The recent years have witnessed great advances in video generation. However,
  the development of automatic video metrics is lagging significantly behind.
  None of the existing metric is able to provide reliable scores over generated
  videos. The main barrier is the lack of large-scale human-annotated dataset. In
  this paper, we release VideoFeedback, the first large-scale dataset containing
  human-provided multi-aspect score over 37.6K synthesized videos from 11
  existing video generative models. We train MantisScore (initialized from
  Mantis) based on VideoFeedback to enable automatic video quality assessment.
  Experiments show that the Spearman correlation between MantisScore and humans
  can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about
  50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and
  VBench show that MantisScore has consistently much higher correlation with
  human judges than other metrics. Due to these results, we believe MantisScore
  can serve as a great proxy for human raters to (1) rate different video models
  to track progress (2) simulate fine-grained human feedback in Reinforcement
  Learning with Human Feedback (RLHF) to improve current video generation models.
paper_authors._id: 6678e7e5405bf950a0f06973
paper_authors.name: Xuan He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:35:33.469Z
paper_authors._id: 655c3953ce055ed40a7de0ba
paper_authors.avatarUrl: /avatars/1a0a746130fd9b95e20184f2fcf4c0ba.svg
paper_authors.fullname: Xuan He
paper_authors.type: user
paper_authors.user: hexuan21
paper_authors._id: 6678e7e5405bf950a0f06974
paper_authors.name: Dongfu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:13:28.524Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 6678e7e5405bf950a0f06975
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:13:41.373Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6678e7e5405bf950a0f06976
paper_authors.name: Max Ku
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:34:02.711Z
paper_authors._id: 631d760344503b7227837242
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631d760344503b7227837242/3b6JRusFX6GKJpsN9ZdeJ.png
paper_authors.fullname: Max Ku
paper_authors.type: user
paper_authors.user: vinesmsuic
paper_authors._id: 6678e7e5405bf950a0f06977
paper_authors.name: Achint Soni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:13:56.236Z
paper_authors._id: 6584bd1a5b94130d87c5f701
paper_authors.avatarUrl: /avatars/615e392f98e763144736c363f980a85a.svg
paper_authors.fullname: Achint Soni
paper_authors.type: user
paper_authors.user: Trickyjustice
paper_authors._id: 6678e7e5405bf950a0f06978
paper_authors.name: Sherman Siu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:14:04.893Z
paper_authors._id: 630d73387dacb93b335b00e9
paper_authors.avatarUrl: /avatars/2ded3550aedf77a58175a66319d01820.svg
paper_authors.fullname: Sherman Siu
paper_authors.type: user
paper_authors.user: shermansiu
paper_authors._id: 6678e7e5405bf950a0f06979
paper_authors.name: Haonan Chen
paper_authors._id: 6678e7e5405bf950a0f0697a
paper_authors.name: Abhranil Chandra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:14:36.268Z
paper_authors._id: 6154ee656181394cc00cb990
paper_authors.avatarUrl: /avatars/25d3b6911c5991b1869f5d76ca2c4069.svg
paper_authors.fullname: Abhranil Chandra
paper_authors.type: user
paper_authors.user: abhranil14
paper_authors._id: 6678e7e5405bf950a0f0697b
paper_authors.name: Ziyan Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:27:45.178Z
paper_authors._id: 64778fb8168cb428e00f69b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
paper_authors.fullname: Ziyan Jiang
paper_authors.type: user
paper_authors.user: ziyjiang
paper_authors._id: 6678e7e5405bf950a0f0697c
paper_authors.name: Aaran Arulraj
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:14:42.671Z
paper_authors._id: 65fb7212fc9132a2dfb80061
paper_authors.avatarUrl: /avatars/7955a4c96cba4e5b25481943f7b84be1.svg
paper_authors.fullname: Aaran Arulraj
paper_authors.type: user
paper_authors.user: AaranArulraj
paper_authors._id: 6678e7e5405bf950a0f0697d
paper_authors.name: Kai Wang
paper_authors._id: 6678e7e5405bf950a0f0697e
paper_authors.name: Quy Duc Do
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:14:51.954Z
paper_authors._id: 65fa5fa1180d06d37c845d51
paper_authors.avatarUrl: /avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
paper_authors.fullname: Quy Duc Do
paper_authors.type: user
paper_authors.user: qd2do
paper_authors._id: 6678e7e5405bf950a0f0697f
paper_authors.name: Yuansheng Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:14:57.984Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 6678e7e5405bf950a0f06980
paper_authors.name: Bohan Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:36:33.224Z
paper_authors._id: 650267e7e751d03da933a24a
paper_authors.avatarUrl: /avatars/f047a047d1de304cd97027463541bdf3.svg
paper_authors.fullname: Bohan22
paper_authors.type: user
paper_authors.user: Bohan22
paper_authors._id: 6678e7e5405bf950a0f06981
paper_authors.name: Yaswanth Narsupalli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:22:11.239Z
paper_authors._id: 64f6dc785f2dee8a6b9fcada
paper_authors.avatarUrl: /avatars/7e9930c141eebdc9fcaddf2fd1d3a53c.svg
paper_authors.fullname: Narsupalli Yaswanth
paper_authors.type: user
paper_authors.user: yaswanth-iitkgp
paper_authors._id: 6678e7e5405bf950a0f06982
paper_authors.name: Rongqi Fan
paper_authors._id: 6678e7e5405bf950a0f06983
paper_authors.name: Zhiheng Lyu
paper_authors._id: 6678e7e5405bf950a0f06984
paper_authors.name: Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:15:33.764Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 6678e7e5405bf950a0f06985
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:15:39.383Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: MantisScore: Building Automatic Metrics to Simulate Fine-grained Human
    Feedback for Video Generation
paper_summary: The recent years have witnessed great advances in video generation. However,
  the development of automatic video metrics is lagging significantly behind.
  None of the existing metric is able to provide reliable scores over generated
  videos. The main barrier is the lack of large-scale human-annotated dataset. In
  this paper, we release VideoFeedback, the first large-scale dataset containing
  human-provided multi-aspect score over 37.6K synthesized videos from 11
  existing video generative models. We train MantisScore (initialized from
  Mantis) based on VideoFeedback to enable automatic video quality assessment.
  Experiments show that the Spearman correlation between MantisScore and humans
  can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about
  50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and
  VBench show that MantisScore has consistently much higher correlation with
  human judges than other metrics. Due to these results, we believe MantisScore
  can serve as a great proxy for human raters to (1) rate different video models
  to track progress (2) simulate fine-grained human feedback in Reinforcement
  Learning with Human Feedback (RLHF) to improve current video generation models.
paper_ai_summary: MantisScore, trained on the VideoFeedback dataset, provides highly accurate automatic video quality assessment, outperforming existing metrics and serving as a reliable human-rater proxy.
paper_ai_keywords: VideoFeedback MantisScore large-scale dataset human-provided scores video generative models Spearman correlation EvalCrafter GenAI-Bench VBench Reinforcement Learning with Human Feedback (RLHF)