title: MIEB: Massive Image Embedding Benchmark
summary: Image representations are often evaluated through disjointed, task-specific
  protocols, leading to a fragmented understanding of model capabilities. For
  instance, it is unclear whether an image embedding model adept at clustering
  images is equally good at retrieving relevant images given a piece of text. We
  introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the
  performance of image and image-text embedding models across the broadest
  spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we
  group into 8 high-level categories. We benchmark 50 models across our
  benchmark, finding that no single method dominates across all task categories.
  We reveal hidden capabilities in advanced vision models such as their accurate
  visual representation of texts, and their yet limited capabilities in
  interleaved encodings and matching images and texts in the presence of
  confounders. We also show that the performance of vision encoders on MIEB
  correlates highly with their performance when used in multimodal large language
  models. Our code, dataset, and leaderboard are publicly available at
  https://github.com/embeddings-benchmark/mteb.
paper_authors._id: 67fe0b5b45d0155d68b3c8c9
paper_authors.name: Chenghao Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:19.869Z
paper_authors._id: 63108cc834c7d77420b0fd68
paper_authors.avatarUrl: /avatars/2721e573a417a8ec0b81ee048c4b42ba.svg
paper_authors.fullname: chenghao xiao
paper_authors.type: user
paper_authors.user: gowitheflow
paper_authors._id: 67fe0b5b45d0155d68b3c8ca
paper_authors.name: Isaac Chung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:26.862Z
paper_authors._id: 64cc0e80a257a3212c0c4b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64cc0e80a257a3212c0c4b24/wqs6WZN8-3OQthcnQXgN7.png
paper_authors.fullname: Isaac Chung
paper_authors.type: user
paper_authors.user: isaacchung
paper_authors._id: 67fe0b5b45d0155d68b3c8cb
paper_authors.name: Imene Kerboua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:14:14.590Z
paper_authors._id: 62610f8040e04009e81047e9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62610f8040e04009e81047e9/iqGTYB7OMmS0jkFEB7cEF.jpeg
paper_authors.fullname: Imene Kerboua
paper_authors.type: user
paper_authors.user: imenelydiaker
paper_authors._id: 67fe0b5b45d0155d68b3c8cc
paper_authors.name: Jamie Stirling
paper_authors._id: 67fe0b5b45d0155d68b3c8cd
paper_authors.name: Xin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:52:33.782Z
paper_authors._id: 63b6dbc8ccebeadccc888456
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673396893898-63b6dbc8ccebeadccc888456.jpeg
paper_authors.fullname: Xin Zhang
paper_authors.type: user
paper_authors.user: izhx
paper_authors._id: 67fe0b5b45d0155d68b3c8ce
paper_authors.name: Márton Kardos
paper_authors._id: 67fe0b5b45d0155d68b3c8cf
paper_authors.name: Roman Solomatin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:24.757Z
paper_authors._id: 61af4544d691b3aadd1f62b6
paper_authors.avatarUrl: /avatars/7a4067accdd1005f78c3c4adad3ee0a5.svg
paper_authors.fullname: Solomatin Roman
paper_authors.type: user
paper_authors.user: Samoed
paper_authors._id: 67fe0b5b45d0155d68b3c8d0
paper_authors.name: Noura Al Moubayed
paper_authors._id: 67fe0b5b45d0155d68b3c8d1
paper_authors.name: Kenneth Enevoldsen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T09:23:22.425Z
paper_authors._id: 5ff5943752c26e9bc240bada
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5ff5943752c26e9bc240bada/Exyzf3C_gJ2KdsL4K5_cq.png
paper_authors.fullname: Kenneth C. Enevoldsen
paper_authors.type: user
paper_authors.user: KennethEnevoldsen
paper_authors._id: 67fe0b5b45d0155d68b3c8d2
paper_authors.name: Niklas Muennighoff
paper_title: MIEB: Massive Image Embedding Benchmark
paper_summary: Image representations are often evaluated through disjointed, task-specific
  protocols, leading to a fragmented understanding of model capabilities. For
  instance, it is unclear whether an image embedding model adept at clustering
  images is equally good at retrieving relevant images given a piece of text. We
  introduce the Massive Image Embedding Benchmark (MIEB) to evaluate the
  performance of image and image-text embedding models across the broadest
  spectrum to date. MIEB spans 38 languages across 130 individual tasks, which we
  group into 8 high-level categories. We benchmark 50 models across our
  benchmark, finding that no single method dominates across all task categories.
  We reveal hidden capabilities in advanced vision models such as their accurate
  visual representation of texts, and their yet limited capabilities in
  interleaved encodings and matching images and texts in the presence of
  confounders. We also show that the performance of vision encoders on MIEB
  correlates highly with their performance when used in multimodal large language
  models. Our code, dataset, and leaderboard are publicly available at
  https://github.com/embeddings-benchmark/mteb.
paper_ai_summary: The Massive Image Embedding Benchmark (MIEB) evaluates image and image-text embedding models across various tasks, revealing hidden capabilities and performance correlations with multimodal large language models.
paper_ai_keywords: Massive Image Embedding Benchmark MIEB image embedding model text retrieval vision models image-text embedding multilingual task categories visual representation interleaved encodings multimodal large language models