title: MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for
    Text-to-Image Generation?
summary: While text-to-image models like DALLE-3 and Stable Diffusion are rapidly
  proliferating, they often encounter challenges such as hallucination, bias, and
  the production of unsafe, low-quality output. To effectively address these
  issues, it is crucial to align these models with desired behaviors based on
  feedback from a multimodal judge. Despite their significance, current
  multimodal judges frequently undergo inadequate evaluation of their
  capabilities and limitations, potentially leading to misalignment and unsafe
  fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel
  benchmark which incorporates a comprehensive preference dataset to evaluate
  multimodal judges in providing feedback for image generation models across four
  key perspectives: alignment, safety, image quality, and bias. Specifically, we
  evaluate a large variety of multimodal judges including smaller-sized
  CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and
  close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our
  preference dataset. Experiments reveal that close-source VLMs generally provide
  better feedback, with GPT-4o outperforming other judges in average. Compared
  with open-source VLMs, smaller-sized scoring models can provide better feedback
  regarding text-image alignment and image quality, while VLMs provide more
  accurate feedback regarding safety and generation bias due to their stronger
  reasoning capabilities. Further studies in feedback scale reveal that VLM
  judges can generally provide more accurate and stable feedback in natural
  language (Likert-scale) than numerical scales. Notably, human evaluations on
  end-to-end fine-tuned models using separate feedback from these multimodal
  judges provide similar conclusions, further confirming the effectiveness of
  MJ-Bench. All data, code, models are available at
  https://huggingface.co/MJ-Bench.
paper_authors._id: 668cbe15bddba3c354e0e7bc
paper_authors.name: Zhaorun Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:41:52.006Z
paper_authors._id: 64d448d3d095077728f7c740
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
paper_authors.fullname: Zhaorun Chen
paper_authors.type: user
paper_authors.user: Zhaorun
paper_authors._id: 668cbe15bddba3c354e0e7bd
paper_authors.name: Yichao Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:41:59.325Z
paper_authors._id: 63e1d3451e5a4f34b7a728ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e1d3451e5a4f34b7a728ef/kHE5JHF9iZJOG0uBkvJr-.jpeg
paper_authors.fullname: yichaodu
paper_authors.type: user
paper_authors.user: yichaodu
paper_authors._id: 668cbe15bddba3c354e0e7be
paper_authors.name: Zichen Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:34:41.704Z
paper_authors._id: 653b8c3e97a4d71d950e2f20
paper_authors.avatarUrl: /avatars/b68880022e14556d0be58c69615db3be.svg
paper_authors.fullname: Zichen Wen
paper_authors.type: user
paper_authors.user: zichenwen
paper_authors._id: 668cbe15bddba3c354e0e7bf
paper_authors.name: Yiyang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:35:14.296Z
paper_authors._id: 6433722c5277e3b24ef49055
paper_authors.avatarUrl: /avatars/f3c5560d500c699e452986a6a45ba3ee.svg
paper_authors.fullname: Yiyang Zhou
paper_authors.type: user
paper_authors.user: YiyangAiLab
paper_authors._id: 668cbe15bddba3c354e0e7c0
paper_authors.name: Chenhang Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:35:28.734Z
paper_authors._id: 64a26627f09ad861c85487e0
paper_authors.avatarUrl: /avatars/baefd153c59f3a3238fd6034c6daf0d4.svg
paper_authors.fullname: ChenhangCui
paper_authors.type: user
paper_authors.user: Chenhangcui
paper_authors._id: 668cbe15bddba3c354e0e7c1
paper_authors.name: Zhenzhen Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:35:35.672Z
paper_authors._id: 630834fbf0dc38fb47c245b2
paper_authors.avatarUrl: /avatars/579d083fdbfba7331b07941f0bef83a4.svg
paper_authors.fullname: Zhenzhen Weng
paper_authors.type: user
paper_authors.user: zzweng
paper_authors._id: 668cbe15bddba3c354e0e7c2
paper_authors.name: Haoqin Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:35:41.704Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 668cbe15bddba3c354e0e7c3
paper_authors.name: Chaoqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:35:48.911Z
paper_authors._id: 64fac4e92d20ced4e9f67525
paper_authors.avatarUrl: /avatars/3c33cf7bac9fab07aa200e553eb832e9.svg
paper_authors.fullname: Chaoqi Wang
paper_authors.type: user
paper_authors.user: alecwangcq
paper_authors._id: 668cbe15bddba3c354e0e7c4
paper_authors.name: Zhengwei Tong
paper_authors._id: 668cbe15bddba3c354e0e7c5
paper_authors.name: Qinglan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:36:09.433Z
paper_authors._id: 6507fac29622235d7dddc430
paper_authors.avatarUrl: /avatars/0fd23bc7db683a18cac8d2cc52131840.svg
paper_authors.fullname: hql
paper_authors.type: user
paper_authors.user: huangqinglan
paper_authors._id: 668cbe15bddba3c354e0e7c6
paper_authors.name: Canyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T07:41:55.281Z
paper_authors._id: 6483af58571c2dcfa98cae82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
paper_authors.fullname: Canyu Chen
paper_authors.type: user
paper_authors.user: canyuchen
paper_authors._id: 668cbe15bddba3c354e0e7c7
paper_authors.name: Qinghao Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T14:05:11.232Z
paper_authors._id: 64530fc01a57e1179c1fe4c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
paper_authors.fullname: QinghaoYe
paper_authors.type: user
paper_authors.user: MAGAer13
paper_authors._id: 668cbe15bddba3c354e0e7c8
paper_authors.name: Zhihong Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:36:25.102Z
paper_authors._id: 64b78e70da8017900e7d1972
paper_authors.avatarUrl: /avatars/f862e7f98cfc3ffb820309ffe7bc73d3.svg
paper_authors.fullname: zhihong zhu
paper_authors.type: user
paper_authors.user: PKUGGBond
paper_authors._id: 668cbe15bddba3c354e0e7c9
paper_authors.name: Yuqing Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T21:09:13.885Z
paper_authors._id: 668d4fde7dbe0ab21ddba8ed
paper_authors.avatarUrl: /avatars/ee5671cc0020cc1b7c6847d7af7252a0.svg
paper_authors.fullname: Yuqing  Zhang
paper_authors.type: user
paper_authors.user: yuqingzhang
paper_authors._id: 668cbe15bddba3c354e0e7ca
paper_authors.name: Jiawei Zhou
paper_authors._id: 668cbe15bddba3c354e0e7cb
paper_authors.name: Zhuokai Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:37:03.316Z
paper_authors._id: 63af25605fe9db73f67a0fb7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63af25605fe9db73f67a0fb7/mcQHbTdJVxi2GdHzRB9ad.jpeg
paper_authors.fullname: Zhuokai Zhao
paper_authors.type: user
paper_authors.user: zhuokai
paper_authors._id: 668cbe15bddba3c354e0e7cc
paper_authors.name: Rafael Rafailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:37:13.962Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 668cbe15bddba3c354e0e7cd
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:37:19.483Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 668cbe15bddba3c354e0e7ce
paper_authors.name: Huaxiu Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-09T08:37:27.914Z
paper_authors._id: 65f667a455009c4ad9e6ac4c
paper_authors.avatarUrl: /avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
paper_authors.fullname: Yao
paper_authors.type: user
paper_authors.user: Huaxiu
paper_title: MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for
    Text-to-Image Generation?
paper_summary: While text-to-image models like DALLE-3 and Stable Diffusion are rapidly
  proliferating, they often encounter challenges such as hallucination, bias, and
  the production of unsafe, low-quality output. To effectively address these
  issues, it is crucial to align these models with desired behaviors based on
  feedback from a multimodal judge. Despite their significance, current
  multimodal judges frequently undergo inadequate evaluation of their
  capabilities and limitations, potentially leading to misalignment and unsafe
  fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel
  benchmark which incorporates a comprehensive preference dataset to evaluate
  multimodal judges in providing feedback for image generation models across four
  key perspectives: alignment, safety, image quality, and bias. Specifically, we
  evaluate a large variety of multimodal judges including smaller-sized
  CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and
  close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our
  preference dataset. Experiments reveal that close-source VLMs generally provide
  better feedback, with GPT-4o outperforming other judges in average. Compared
  with open-source VLMs, smaller-sized scoring models can provide better feedback
  regarding text-image alignment and image quality, while VLMs provide more
  accurate feedback regarding safety and generation bias due to their stronger
  reasoning capabilities. Further studies in feedback scale reveal that VLM
  judges can generally provide more accurate and stable feedback in natural
  language (Likert-scale) than numerical scales. Notably, human evaluations on
  end-to-end fine-tuned models using separate feedback from these multimodal
  judges provide similar conclusions, further confirming the effectiveness of
  MJ-Bench. All data, code, models are available at
  https://huggingface.co/MJ-Bench.
paper_ai_keywords: multimodal judge hallucination bias unsafe output preference dataset image generation models alignment safety image quality bias CLIP-based scoring models open-source VLMs close-source VLMs GPT-4o Claude 3 end-to-end fine-tuned models human evaluations Likert-scale