HumaniBench: A Human-Centric Framework for Large Multimodal Models
  Evaluation
Large multimodal models (LMMs) now excel on many vision language benchmarks,
however, they still struggle with human centered criteria such as fairness,
ethics, empathy, and inclusivity, key to aligning with human values. We
introduce HumaniBench, a holistic benchmark of 32K real-world image question
pairs, annotated via a scalable GPT4o assisted pipeline and exhaustively
verified by domain experts. HumaniBench evaluates seven Human Centered AI
(HCAI) principles: fairness, ethics, understanding, reasoning, language
inclusivity, empathy, and robustness, across seven diverse tasks, including
open and closed ended visual question answering (VQA), multilingual QA, visual
grounding, empathetic captioning, and robustness tests. Benchmarking 15 state
of the art LMMs (open and closed source) reveals that proprietary models
generally lead, though robustness and visual grounding remain weak points. Some
open-source models also struggle to balance accuracy with adherence to
human-aligned principles. HumaniBench is the first benchmark purpose built
around HCAI principles. It provides a rigorous testbed for diagnosing alignment
gaps and guiding LMMs toward behavior that is both accurate and socially
responsible. Dataset, annotation prompts, and evaluation code are available at:
https://vectorinstitute.github.io/HumaniBench
682bd33f04f4f8e7cf57fba5
Shaina Raza
claimed_verified
2025-05-20T07:22:00.248Z
5e466ca12d2efc729dc309ad
https://cdn-avatars.huggingface.co/v1/production/uploads/1601989543860-5e466ca12d2efc729dc309ad.png
shaina
user
shainaraza
682bd33f04f4f8e7cf57fba6
Aravind Narayanan
682bd33f04f4f8e7cf57fba7
Vahid Reza Khazaie
682bd33f04f4f8e7cf57fba8
Ashmal Vayani
682bd33f04f4f8e7cf57fba9
Mukund S. Chettiar
682bd33f04f4f8e7cf57fbaa
Amandeep Singh
682bd33f04f4f8e7cf57fbab
Mubarak Shah
682bd33f04f4f8e7cf57fbac
Deval Pandya
HumaniBench: A Human-Centric Framework for Large Multimodal Models
  Evaluation
Large multimodal models (LMMs) now excel on many vision language benchmarks,
however, they still struggle with human centered criteria such as fairness,
ethics, empathy, and inclusivity, key to aligning with human values. We
introduce HumaniBench, a holistic benchmark of 32K real-world image question
pairs, annotated via a scalable GPT4o assisted pipeline and exhaustively
verified by domain experts. HumaniBench evaluates seven Human Centered AI
(HCAI) principles: fairness, ethics, understanding, reasoning, language
inclusivity, empathy, and robustness, across seven diverse tasks, including
open and closed ended visual question answering (VQA), multilingual QA, visual
grounding, empathetic captioning, and robustness tests. Benchmarking 15 state
of the art LMMs (open and closed source) reveals that proprietary models
generally lead, though robustness and visual grounding remain weak points. Some
open-source models also struggle to balance accuracy with adherence to
human-aligned principles. HumaniBench is the first benchmark purpose built
around HCAI principles. It provides a rigorous testbed for diagnosing alignment
gaps and guiding LMMs toward behavior that is both accurate and socially
responsible. Dataset, annotation prompts, and evaluation code are available at:
https://vectorinstitute.github.io/HumaniBench
HumaniBench evaluates state-of-the-art LMMs on seven human-centered AI principles using 32K real-world image-question pairs to ensure fairness, ethics, empathy, and inclusivity.
LMMs HumaniBench GPT4o Human Centered AI fairness ethics understanding reasoning language inclusivity empathy robustness visual question answering multilingual QA visual grounding empathetic captioning alignment gaps socially responsible