title: HumaniBench: A Human-Centric Framework for Large Multimodal Models
    Evaluation
summary: Large multimodal models (LMMs) now excel on many vision language benchmarks,
  however, they still struggle with human centered criteria such as fairness,
  ethics, empathy, and inclusivity, key to aligning with human values. We
  introduce HumaniBench, a holistic benchmark of 32K real-world image question
  pairs, annotated via a scalable GPT4o assisted pipeline and exhaustively
  verified by domain experts. HumaniBench evaluates seven Human Centered AI
  (HCAI) principles: fairness, ethics, understanding, reasoning, language
  inclusivity, empathy, and robustness, across seven diverse tasks, including
  open and closed ended visual question answering (VQA), multilingual QA, visual
  grounding, empathetic captioning, and robustness tests. Benchmarking 15 state
  of the art LMMs (open and closed source) reveals that proprietary models
  generally lead, though robustness and visual grounding remain weak points. Some
  open-source models also struggle to balance accuracy with adherence to
  human-aligned principles. HumaniBench is the first benchmark purpose built
  around HCAI principles. It provides a rigorous testbed for diagnosing alignment
  gaps and guiding LMMs toward behavior that is both accurate and socially
  responsible. Dataset, annotation prompts, and evaluation code are available at:
  https://vectorinstitute.github.io/HumaniBench
paper_authors._id: 682bd33f04f4f8e7cf57fba5
paper_authors.name: Shaina Raza
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:22:00.248Z
paper_authors._id: 5e466ca12d2efc729dc309ad
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1601989543860-5e466ca12d2efc729dc309ad.png
paper_authors.fullname: shaina
paper_authors.type: user
paper_authors.user: shainaraza
paper_authors._id: 682bd33f04f4f8e7cf57fba6
paper_authors.name: Aravind Narayanan
paper_authors._id: 682bd33f04f4f8e7cf57fba7
paper_authors.name: Vahid Reza Khazaie
paper_authors._id: 682bd33f04f4f8e7cf57fba8
paper_authors.name: Ashmal Vayani
paper_authors._id: 682bd33f04f4f8e7cf57fba9
paper_authors.name: Mukund S. Chettiar
paper_authors._id: 682bd33f04f4f8e7cf57fbaa
paper_authors.name: Amandeep Singh
paper_authors._id: 682bd33f04f4f8e7cf57fbab
paper_authors.name: Mubarak Shah
paper_authors._id: 682bd33f04f4f8e7cf57fbac
paper_authors.name: Deval Pandya
paper_title: HumaniBench: A Human-Centric Framework for Large Multimodal Models
    Evaluation
paper_summary: Large multimodal models (LMMs) now excel on many vision language benchmarks,
  however, they still struggle with human centered criteria such as fairness,
  ethics, empathy, and inclusivity, key to aligning with human values. We
  introduce HumaniBench, a holistic benchmark of 32K real-world image question
  pairs, annotated via a scalable GPT4o assisted pipeline and exhaustively
  verified by domain experts. HumaniBench evaluates seven Human Centered AI
  (HCAI) principles: fairness, ethics, understanding, reasoning, language
  inclusivity, empathy, and robustness, across seven diverse tasks, including
  open and closed ended visual question answering (VQA), multilingual QA, visual
  grounding, empathetic captioning, and robustness tests. Benchmarking 15 state
  of the art LMMs (open and closed source) reveals that proprietary models
  generally lead, though robustness and visual grounding remain weak points. Some
  open-source models also struggle to balance accuracy with adherence to
  human-aligned principles. HumaniBench is the first benchmark purpose built
  around HCAI principles. It provides a rigorous testbed for diagnosing alignment
  gaps and guiding LMMs toward behavior that is both accurate and socially
  responsible. Dataset, annotation prompts, and evaluation code are available at:
  https://vectorinstitute.github.io/HumaniBench
paper_ai_summary: HumaniBench evaluates state-of-the-art LMMs on seven human-centered AI principles using 32K real-world image-question pairs to ensure fairness, ethics, empathy, and inclusivity.
paper_ai_keywords: LMMs HumaniBench GPT4o Human Centered AI fairness ethics understanding reasoning language inclusivity empathy robustness visual question answering multilingual QA visual grounding empathetic captioning alignment gaps socially responsible