title: LLaVA-Critic: Learning to Evaluate Multimodal Models
summary: We introduce LLaVA-Critic, the first open-source large multimodal model (LMM)
  designed as a generalist evaluator to assess performance across a wide range of
  multimodal tasks. LLaVA-Critic is trained using a high-quality critic
  instruction-following dataset that incorporates diverse evaluation criteria and
  scenarios. Our experiments demonstrate the model's effectiveness in two key
  areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation
  scores, performing on par with or surpassing GPT models on multiple evaluation
  benchmarks; and (2) Preference Learning, where it generates reward signals for
  preference learning, enhancing model alignment capabilities. This work
  underscores the potential of open-source LMMs in self-critique and evaluation,
  setting the stage for future research into scalable, superhuman alignment
  feedback mechanisms for LMMs.
paper_authors._id: 66ff51ed9e1143bff207d587
paper_authors.name: Tianyi Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:41:36.661Z
paper_authors._id: 6570977f87a92b76922c9950
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6570977f87a92b76922c9950/AQGto1w6ugBvH2yCV46YU.jpeg
paper_authors.fullname: Tianyi Xiong
paper_authors.type: user
paper_authors.user: txiong23
paper_authors._id: 66ff51ed9e1143bff207d588
paper_authors.name: Xiyao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:44:04.812Z
paper_authors._id: 655fed9fdef5905d38b84af3
paper_authors.avatarUrl: /avatars/2cda4182dfd11a1e94743639e62328ea.svg
paper_authors.fullname: Xiyao Wang
paper_authors.type: user
paper_authors.user: russwang
paper_authors._id: 66ff51ed9e1143bff207d589
paper_authors.name: Dong Guo
paper_authors._id: 66ff51ed9e1143bff207d58a
paper_authors.name: Qinghao Ye
paper_authors._id: 66ff51ed9e1143bff207d58b
paper_authors.name: Haoqi Fan
paper_authors._id: 66ff51ed9e1143bff207d58c
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:41:11.820Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_authors._id: 66ff51ed9e1143bff207d58d
paper_authors.name: Heng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:43:29.377Z
paper_authors._id: 6527bc6b34bf5ece73da426d
paper_authors.avatarUrl: /avatars/120739a9ac84e7319d9ea157a63dc547.svg
paper_authors.fullname: henghuang
paper_authors.type: user
paper_authors.user: henghuang
paper_authors._id: 66ff51ed9e1143bff207d58e
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T08:43:01.990Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_title: LLaVA-Critic: Learning to Evaluate Multimodal Models
paper_summary: We introduce LLaVA-Critic, the first open-source large multimodal model (LMM)
  designed as a generalist evaluator to assess performance across a wide range of
  multimodal tasks. LLaVA-Critic is trained using a high-quality critic
  instruction-following dataset that incorporates diverse evaluation criteria and
  scenarios. Our experiments demonstrate the model's effectiveness in two key
  areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation
  scores, performing on par with or surpassing GPT models on multiple evaluation
  benchmarks; and (2) Preference Learning, where it generates reward signals for
  preference learning, enhancing model alignment capabilities. This work
  underscores the potential of open-source LMMs in self-critique and evaluation,
  setting the stage for future research into scalable, superhuman alignment
  feedback mechanisms for LMMs.
paper_ai_keywords: large multimodal model (LMM) generalist evaluator critic instruction-following dataset evaluation criteria evaluation benchmarks LMM-as-a-Judge evaluation labels GPT models Preference Learning reward signals model alignment capabilities self-critique superhuman alignment feedback mechanisms