title: Q-Eval-100K: Evaluating Visual Quality and Alignment Level for
    Text-to-Vision Content
summary: Evaluating text-to-vision content hinges on two crucial aspects: visual
  quality and alignment. While significant progress has been made in developing
  objective models to assess these dimensions, the performance of such models
  heavily relies on the scale and quality of human annotations. According to
  Scaling Law, increasing the number of human-labeled instances follows a
  predictable pattern that enhances the performance of evaluation models.
  Therefore, we introduce a comprehensive dataset designed to Evaluate Visual
  quality and Alignment Level for text-to-vision content (Q-EVAL-100K), featuring
  the largest collection of human-labeled Mean Opinion Scores (MOS) for the
  mentioned two aspects. The Q-EVAL-100K dataset encompasses both text-to-image
  and text-to-video models, with 960K human annotations specifically focused on
  visual quality and alignment for 100K instances (60K images and 40K videos).
  Leveraging this dataset with context prompt, we propose Q-Eval-Score, a unified
  model capable of evaluating both visual quality and alignment with special
  improvements for handling long-text prompt alignment. Experimental results
  indicate that the proposed Q-Eval-Score achieves superior performance on both
  visual quality and alignment, with strong generalization capabilities across
  other benchmarks. These findings highlight the significant value of the
  Q-EVAL-100K dataset. Data and codes will be available at
  https://github.com/zzc-1998/Q-Eval.
paper_authors._id: 67c830966f68759bf38567a5
paper_authors.name: Zicheng Zhang
paper_authors._id: 67c830966f68759bf38567a6
paper_authors.name: Tengchuan Kou
paper_authors._id: 67c830966f68759bf38567a7
paper_authors.name: Shushi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:36:18.075Z
paper_authors._id: 64687fb52a180a1378359c46
paper_authors.avatarUrl: /avatars/746ea4d52801227fd44b9a9f380738a2.svg
paper_authors.fullname: shushi wang 
paper_authors.type: user
paper_authors.user: wcain
paper_authors._id: 67c830966f68759bf38567a8
paper_authors.name: Chunyi Li
paper_authors._id: 67c830966f68759bf38567a9
paper_authors.name: Wei Sun
paper_authors._id: 67c830966f68759bf38567aa
paper_authors.name: Wei Wang
paper_authors._id: 67c830966f68759bf38567ab
paper_authors.name: Xiaoyu Li
paper_authors._id: 67c830966f68759bf38567ac
paper_authors.name: Zongyu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:36:37.427Z
paper_authors._id: 6459f4f9667a889f3d35f51c
paper_authors.avatarUrl: /avatars/a4a407b8f0d6e3616c8e7ae7b4c78a7d.svg
paper_authors.fullname: zongyu wang
paper_authors.type: user
paper_authors.user: sl2782087
paper_authors._id: 67c830966f68759bf38567ad
paper_authors.name: Xuezhi Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:36:30.233Z
paper_authors._id: 647095cdcf0a68ad56aa7181
paper_authors.avatarUrl: /avatars/65e87b38b90c57b6db82e51f48a15fcf.svg
paper_authors.fullname: xuezhi cao
paper_authors.type: user
paper_authors.user: a9108
paper_authors._id: 67c830966f68759bf38567ae
paper_authors.name: Xiongkuo Min
paper_authors._id: 67c830966f68759bf38567af
paper_authors.name: Xiaohong Liu
paper_authors._id: 67c830966f68759bf38567b0
paper_authors.name: Guangtao Zhai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:37:03.268Z
paper_authors._id: 65535b125413c1a54e6fb243
paper_authors.avatarUrl: /avatars/03bcf1d58865f5406aff49a415e78bdc.svg
paper_authors.fullname: Guangtao Zhai
paper_authors.type: user
paper_authors.user: GTZhai
paper_title: Q-Eval-100K: Evaluating Visual Quality and Alignment Level for
    Text-to-Vision Content
paper_summary: Evaluating text-to-vision content hinges on two crucial aspects: visual
  quality and alignment. While significant progress has been made in developing
  objective models to assess these dimensions, the performance of such models
  heavily relies on the scale and quality of human annotations. According to
  Scaling Law, increasing the number of human-labeled instances follows a
  predictable pattern that enhances the performance of evaluation models.
  Therefore, we introduce a comprehensive dataset designed to Evaluate Visual
  quality and Alignment Level for text-to-vision content (Q-EVAL-100K), featuring
  the largest collection of human-labeled Mean Opinion Scores (MOS) for the
  mentioned two aspects. The Q-EVAL-100K dataset encompasses both text-to-image
  and text-to-video models, with 960K human annotations specifically focused on
  visual quality and alignment for 100K instances (60K images and 40K videos).
  Leveraging this dataset with context prompt, we propose Q-Eval-Score, a unified
  model capable of evaluating both visual quality and alignment with special
  improvements for handling long-text prompt alignment. Experimental results
  indicate that the proposed Q-Eval-Score achieves superior performance on both
  visual quality and alignment, with strong generalization capabilities across
  other benchmarks. These findings highlight the significant value of the
  Q-EVAL-100K dataset. Data and codes will be available at
  https://github.com/zzc-1998/Q-Eval.
paper_ai_keywords: Mean Opinion Scores (MOS) Q-EVAL-100K text-to-image text-to-video Q-Eval-Score context prompt long-text prompt alignment