title: GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
summary: Large multimodal models (LMMs) have exhibited proficiencies across many
  visual tasks. Although numerous well-known benchmarks exist to evaluate model
  performance, they increasingly have insufficient headroom. As such, there is a
  pressing need for a new generation of benchmarks challenging enough for the
  next generation of LMMs. One area that LMMs show potential is graph analysis,
  specifically, the tasks an analyst might typically perform when interpreting
  figures such as estimating the mean, intercepts or correlations of functions
  and data series. In this work, we introduce GRAB, a graph analysis benchmark,
  fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
  ensuring high-quality, noise-free questions. GRAB is comprised of 2170
  questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
  GRAB, finding it to be a challenging benchmark, with the highest performing
  model attaining a score of just 21.7%. Finally, we conduct various ablations to
  investigate where the models succeed and struggle. We release GRAB to encourage
  progress in this important, growing domain.
paper_authors._id: 66c6eb81e41514386f743ce6
paper_authors.name: Jonathan Roberts
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:17.177Z
paper_authors._id: 632456d21ed511c0c5231afd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
paper_authors.fullname: Jonathan Roberts
paper_authors.type: user
paper_authors.user: jonathan-roberts1
paper_authors._id: 66c6eb81e41514386f743ce7
paper_authors.name: Kai Han
paper_authors._id: 66c6eb81e41514386f743ce8
paper_authors.name: Samuel Albanie
paper_title: GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
paper_summary: Large multimodal models (LMMs) have exhibited proficiencies across many
  visual tasks. Although numerous well-known benchmarks exist to evaluate model
  performance, they increasingly have insufficient headroom. As such, there is a
  pressing need for a new generation of benchmarks challenging enough for the
  next generation of LMMs. One area that LMMs show potential is graph analysis,
  specifically, the tasks an analyst might typically perform when interpreting
  figures such as estimating the mean, intercepts or correlations of functions
  and data series. In this work, we introduce GRAB, a graph analysis benchmark,
  fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
  ensuring high-quality, noise-free questions. GRAB is comprised of 2170
  questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
  GRAB, finding it to be a challenging benchmark, with the highest performing
  model attaining a score of just 21.7%. Finally, we conduct various ablations to
  investigate where the models succeed and struggle. We release GRAB to encourage
  progress in this important, growing domain.
paper_ai_keywords: multimodal models (LMMs) graph analysis benchmark (GRAB) NP ablations