title: LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
summary: The large-scale training of multi-modal models on data scraped from the web
  has shown outstanding utility in infusing these models with the required world
  knowledge to perform effectively on multiple downstream tasks. However, one
  downside of scraping data from the web can be the potential sacrifice of the
  benchmarks on which the abilities of these models are often evaluated. To
  safeguard against test data contamination and to truly test the abilities of
  these foundation models we propose LiveXiv: A scalable evolving live benchmark
  based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts
  at any given timestamp and proposes to automatically generate visual
  question-answer pairs (VQA). This is done without any human-in-the-loop, using
  the multi-modal content in the manuscripts, like graphs, charts, and tables.
  Moreover, we introduce an efficient evaluation approach that estimates the
  performance of all models on the evolving benchmark using evaluations of only a
  subset of models. This significantly reduces the overall evaluation cost. We
  benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the
  first version of our benchmark, showing its challenging nature and exposing the
  models true abilities, avoiding contamination. Lastly, in our commitment to
  high quality, we have collected and evaluated a manually verified subset. By
  comparing its overall results to our automatic annotations, we have found that
  the performance variance is indeed minimal (<2.5%). Our dataset is available
  online on HuggingFace, and our code will be available here.
paper_authors._id: 670e0604ae37bc424b3bb0c3
paper_authors.name: Nimrod Shabtay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T13:06:04.321Z
paper_authors._id: 62bedff7304b82a773bf8c1b
paper_authors.avatarUrl: /avatars/f9e79dc196caa95c220127c6212e9944.svg
paper_authors.fullname: Nimrod Shabtay
paper_authors.type: user
paper_authors.user: NimrodShabtay1986
paper_authors._id: 670e0604ae37bc424b3bb0c4
paper_authors.name: Felipe Maia Polo
paper_authors._id: 670e0604ae37bc424b3bb0c5
paper_authors.name: Sivan Doveh
paper_authors._id: 670e0604ae37bc424b3bb0c6
paper_authors.name: Wei Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:46.917Z
paper_authors._id: 64b99a3fd4463c3d2183739d
paper_authors.avatarUrl: /avatars/9b0fddbf6b13dd1baa7cfcee3ff4f0b9.svg
paper_authors.fullname: Wei Lin
paper_authors.type: user
paper_authors.user: wlin21at
paper_authors._id: 670e0604ae37bc424b3bb0c7
paper_authors.name: M. Jehanzeb Mirza
paper_authors._id: 670e0604ae37bc424b3bb0c8
paper_authors.name: Leshem Chosen
paper_authors._id: 670e0604ae37bc424b3bb0c9
paper_authors.name: Mikhail Yurochkin
paper_authors._id: 670e0604ae37bc424b3bb0ca
paper_authors.name: Yuekai Sun
paper_authors._id: 670e0604ae37bc424b3bb0cb
paper_authors.name: Assaf Arbelle
paper_authors._id: 670e0604ae37bc424b3bb0cc
paper_authors.name: Leonid Karlinsky
paper_authors._id: 670e0604ae37bc424b3bb0cd
paper_authors.name: Raja Giryes
paper_title: LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
paper_summary: The large-scale training of multi-modal models on data scraped from the web
  has shown outstanding utility in infusing these models with the required world
  knowledge to perform effectively on multiple downstream tasks. However, one
  downside of scraping data from the web can be the potential sacrifice of the
  benchmarks on which the abilities of these models are often evaluated. To
  safeguard against test data contamination and to truly test the abilities of
  these foundation models we propose LiveXiv: A scalable evolving live benchmark
  based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts
  at any given timestamp and proposes to automatically generate visual
  question-answer pairs (VQA). This is done without any human-in-the-loop, using
  the multi-modal content in the manuscripts, like graphs, charts, and tables.
  Moreover, we introduce an efficient evaluation approach that estimates the
  performance of all models on the evolving benchmark using evaluations of only a
  subset of models. This significantly reduces the overall evaluation cost. We
  benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the
  first version of our benchmark, showing its challenging nature and exposing the
  models true abilities, avoiding contamination. Lastly, in our commitment to
  high quality, we have collected and evaluated a manually verified subset. By
  comparing its overall results to our automatic annotations, we have found that
  the performance variance is indeed minimal (<2.5%). Our dataset is available
  online on HuggingFace, and our code will be available here.
paper_ai_keywords: multi-modal models test data contamination. visual question-answer pairs (VQA). graphs. charts. tables. scalable evolving live benchmark. multi-modal content. Large Multi-modal Models (LMMs). manually verified subset. automatic annotations.