title: MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution
    Real-World Scenarios that are Difficult for Humans?
summary: Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has
  recently garnered widespread attention in the research community. However, we
  observe that existing benchmarks present several common barriers that make it
  difficult to measure the significant challenges that models face in the real
  world, including: 1) small data scale leads to a large performance variance; 2)
  reliance on model-based annotations results in restricted data quality; 3)
  insufficient task difficulty, especially caused by the limited image
  resolution. To tackle these issues, we introduce MME-RealWorld. Specifically,
  we collect more than 300K images from public datasets and the Internet,
  filtering 13,366 high-quality images for annotation. This involves the
  efforts of professional 25 annotators and 7 experts in MLLMs, contributing
  to 29,429 question-answer pairs that cover 43 subtasks across 5
  real-world scenarios, extremely challenging even for humans. As far as we know,
  MME-RealWorld is the largest manually annotated benchmark to date, featuring
  the highest resolution and a targeted focus on real-world applications. We
  further conduct a thorough evaluation involving 28 prominent MLLMs, such as
  GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the
  most advanced models struggle with our benchmarks, where none of them reach
  60% accuracy. The challenges of perceiving high-resolution images and
  understanding complex real-world scenarios remain urgent issues to be
  addressed. The data and evaluation code are released at
  https://mme-realworld.github.io/ .
paper_authors._id: 66cbf6dab395ddc37b56e9c8
paper_authors.name: Yi-Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:43:53.881Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 66cbf6dab395ddc37b56e9c9
paper_authors.name: Huanyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:14:54.194Z
paper_authors._id: 65e816bbcfd12cd15b052a0e
paper_authors.avatarUrl: /avatars/4d92da469afdba8cd7dc645b98236011.svg
paper_authors.fullname: Huanyu_Zhang
paper_authors.type: user
paper_authors.user: huanyu112
paper_authors._id: 66cbf6dab395ddc37b56e9ca
paper_authors.name: Haochen Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-24T22:07:22.152Z
paper_authors._id: 6448baa3e780dbfc89058bc3
paper_authors.avatarUrl: /avatars/44c243c8eb4714cdcdaa7cd04e5a9716.svg
paper_authors.fullname: Micheal Tian
paper_authors.type: user
paper_authors.user: StarBurger
paper_authors._id: 66cbf6dab395ddc37b56e9cb
paper_authors.name: Chaoyou Fu
paper_authors._id: 66cbf6dab395ddc37b56e9cc
paper_authors.name: Shuangqing Zhang
paper_authors._id: 66cbf6dab395ddc37b56e9cd
paper_authors.name: Junfei Wu
paper_authors._id: 66cbf6dab395ddc37b56e9ce
paper_authors.name: Feng Li
paper_authors._id: 66cbf6dab395ddc37b56e9cf
paper_authors.name: Kun Wang
paper_authors._id: 66cbf6dab395ddc37b56e9d0
paper_authors.name: Qingsong Wen
paper_authors._id: 66cbf6dab395ddc37b56e9d1
paper_authors.name: Zhang Zhang
paper_authors._id: 66cbf6dab395ddc37b56e9d2
paper_authors.name: Liang Wang
paper_authors._id: 66cbf6dab395ddc37b56e9d3
paper_authors.name: Rong Jin
paper_authors._id: 66cbf6dab395ddc37b56e9d4
paper_authors.name: Tieniu Tan
paper_title: MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution
    Real-World Scenarios that are Difficult for Humans?
paper_summary: Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has
  recently garnered widespread attention in the research community. However, we
  observe that existing benchmarks present several common barriers that make it
  difficult to measure the significant challenges that models face in the real
  world, including: 1) small data scale leads to a large performance variance; 2)
  reliance on model-based annotations results in restricted data quality; 3)
  insufficient task difficulty, especially caused by the limited image
  resolution. To tackle these issues, we introduce MME-RealWorld. Specifically,
  we collect more than 300K images from public datasets and the Internet,
  filtering 13,366 high-quality images for annotation. This involves the
  efforts of professional 25 annotators and 7 experts in MLLMs, contributing
  to 29,429 question-answer pairs that cover 43 subtasks across 5
  real-world scenarios, extremely challenging even for humans. As far as we know,
  MME-RealWorld is the largest manually annotated benchmark to date, featuring
  the highest resolution and a targeted focus on real-world applications. We
  further conduct a thorough evaluation involving 28 prominent MLLMs, such as
  GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the
  most advanced models struggle with our benchmarks, where none of them reach
  60% accuracy. The challenges of perceiving high-resolution images and
  understanding complex real-world scenarios remain urgent issues to be
  addressed. The data and evaluation code are released at
  https://mme-realworld.github.io/ .
paper_ai_keywords: Multimodal Large Language Models (MLLMs) MME-RealWorld data scale performance variance model-based annotations task difficulty image resolution high-quality images question-answer pairs subtasks real-world scenarios manually annotated benchmark -resolution images complex real-world scenarios perceiving high-resolution images understanding complex real-world scenarios