title: Multi-Dimensional Insights: Benchmarking Real-World Personalization in
    Large Multimodal Models
summary: The rapidly developing field of large multimodal models (LMMs) has led to the
  emergence of diverse models with remarkable capabilities. However, existing
  benchmarks fail to comprehensively, objectively and accurately evaluate whether
  LMMs align with the diverse needs of humans in real-world scenarios. To bridge
  this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which
  includes over 500 images covering six common scenarios of human life. Notably,
  the MDI-Benchmark offers two significant advantages over existing evaluations:
  (1) Each image is accompanied by two types of questions: simple questions to
  assess the model's understanding of the image, and complex questions to
  evaluate the model's ability to analyze and reason beyond basic content. (2)
  Recognizing that people of different age groups have varying needs and
  perspectives when faced with the same scenario, our benchmark stratifies
  questions into three age categories: young people, middle-aged people, and
  older people. This design allows for a detailed assessment of LMMs'
  capabilities in meeting the preferences and needs of different age groups. With
  MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related
  tasks, indicating that existing LMMs still have considerable room for
  improvement in addressing real-world applications. Looking ahead, we anticipate
  that the MDI-Benchmark will open new pathways for aligning real-world
  personalization in LMMs. The MDI-Benchmark data and evaluation code are
  available at https://mdi-benchmark.github.io/
paper_authors._id: 676240aeb57a82f47d1a81f2
paper_authors.name: YiFan Zhang
paper_authors._id: 676240aeb57a82f47d1a81f3
paper_authors.name: Shanglin Lei
paper_authors._id: 676240aeb57a82f47d1a81f4
paper_authors.name: Runqi Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:03:40.564Z
paper_authors._id: 6683a05e74fb1736a4b7c934
paper_authors.avatarUrl: /avatars/35292253cf1882080be353c4b8c9ca3b.svg
paper_authors.fullname: QRQ
paper_authors.type: user
paper_authors.user: RichardQRQ
paper_authors._id: 676240aeb57a82f47d1a81f5
paper_authors.name: Zhuoma GongQue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:43.390Z
paper_authors._id: 6762b973686e574cab4717ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
paper_authors.fullname: zhuoma
paper_authors.type: user
paper_authors.user: Ethereal1l
paper_authors._id: 676240aeb57a82f47d1a81f6
paper_authors.name: Xiaoshuai Song
paper_authors._id: 676240aeb57a82f47d1a81f7
paper_authors.name: Guanting Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:53:40.027Z
paper_authors._id: 61cd4b833dd34ba1985e0753
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
paper_authors.fullname: KABI
paper_authors.type: user
paper_authors.user: dongguanting
paper_authors._id: 676240aeb57a82f47d1a81f8
paper_authors.name: Qiuna Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T14:32:33.953Z
paper_authors._id: 66615dd1c8bbe034b4b82029
paper_authors.avatarUrl: /avatars/7ef1ddf2fa356efbeb2b3aa12284a1f0.svg
paper_authors.fullname: Zhang YiFan
paper_authors.type: user
paper_authors.user: zhangyifan666
paper_authors._id: 676240aeb57a82f47d1a81f9
paper_authors.name: Zhe Wei
paper_authors._id: 676240aeb57a82f47d1a81fa
paper_authors.name: Peiqing Yang
paper_authors._id: 676240aeb57a82f47d1a81fb
paper_authors.name: Ye Tian
paper_authors._id: 676240aeb57a82f47d1a81fc
paper_authors.name: Yadong Xue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:52:43.539Z
paper_authors._id: 64d4b5808b65d477e68f2fba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/upUtVjLXZ9lAulngeFh_i.jpeg
paper_authors.fullname: Xue Yadong
paper_authors.type: user
paper_authors.user: ataraxy3
paper_authors._id: 676240aeb57a82f47d1a81fd
paper_authors.name: Xiaofei Wang
paper_authors._id: 676240aeb57a82f47d1a81fe
paper_authors.name: Honggang Zhang
paper_title: Multi-Dimensional Insights: Benchmarking Real-World Personalization in
    Large Multimodal Models
paper_summary: The rapidly developing field of large multimodal models (LMMs) has led to the
  emergence of diverse models with remarkable capabilities. However, existing
  benchmarks fail to comprehensively, objectively and accurately evaluate whether
  LMMs align with the diverse needs of humans in real-world scenarios. To bridge
  this gap, we propose the Multi-Dimensional Insights (MDI) benchmark, which
  includes over 500 images covering six common scenarios of human life. Notably,
  the MDI-Benchmark offers two significant advantages over existing evaluations:
  (1) Each image is accompanied by two types of questions: simple questions to
  assess the model's understanding of the image, and complex questions to
  evaluate the model's ability to analyze and reason beyond basic content. (2)
  Recognizing that people of different age groups have varying needs and
  perspectives when faced with the same scenario, our benchmark stratifies
  questions into three age categories: young people, middle-aged people, and
  older people. This design allows for a detailed assessment of LMMs'
  capabilities in meeting the preferences and needs of different age groups. With
  MDI-Benchmark, the strong model like GPT-4o achieve 79% accuracy on age-related
  tasks, indicating that existing LMMs still have considerable room for
  improvement in addressing real-world applications. Looking ahead, we anticipate
  that the MDI-Benchmark will open new pathways for aligning real-world
  personalization in LMMs. The MDI-Benchmark data and evaluation code are
  available at https://mdi-benchmark.github.io/
paper_ai_keywords: large multimodal models (LMMs) Multi-Dimensional Insights (MDI) benchmark kullbackbackleibler divergence (implied by "Kullbackbackleibler") age-related tasks large multimodal models (LMMs) Multi-Dimensional Insights (MDI) benchmark kullbackbackleibler divergence (implied by "Kullbackbackleibler") age-related tasks