title: arXiVeri: Automatic table verification with GPT
summary: Without accurate transcription of numerical data in scientific documents, a
  scientist cannot draw accurate conclusions. Unfortunately, the process of
  copying numerical data from one paper to another is prone to human error. In
  this paper, we propose to meet this challenge through the novel task of
  automatic table verification (AutoTV), in which the objective is to verify the
  accuracy of numerical data in tables by cross-referencing cited sources. To
  support this task, we propose a new benchmark, arXiVeri, which comprises
  tabular data drawn from open-access academic papers on arXiv. We introduce
  metrics to evaluate the performance of a table verifier in two key areas: (i)
  table matching, which aims to identify the source table in a cited document
  that corresponds to a target table, and (ii) cell matching, which aims to
  locate shared cells between a target and source table and identify their row
  and column indices accurately. By leveraging the flexible capabilities of
  modern large language models (LLMs), we propose simple baselines for table
  verification. Our findings highlight the complexity of this task, even for
  state-of-the-art LLMs like OpenAI's GPT-4. The code and benchmark will be made
  publicly available.
paper_authors._id: 64891cbe563d6345d8ea4b66
paper_authors.name: Gyungin Shin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T11:47:17.339Z
paper_authors._id: 6238c507254d95e12bd1952b
paper_authors.avatarUrl: /avatars/a70517208abecc1d6f245ebc63cb4ed6.svg
paper_authors.fullname: Gyungin Shin
paper_authors.type: user
paper_authors.user: noelshin
paper_authors._id: 64891cbe563d6345d8ea4b67
paper_authors.name: Weidi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T11:47:52.962Z
paper_authors._id: 63f1841f2f7c0152e8748ec4
paper_authors.avatarUrl: /avatars/5398658c7d0cf556531a625a4ca5d18a.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Weidi
paper_authors._id: 64891cbe563d6345d8ea4b68
paper_authors.name: Samuel Albanie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T11:51:18.735Z
paper_authors._id: 62f3efefd6ba2ee26651f44a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
paper_authors.fullname: Samuel Albanie
paper_authors.type: user
paper_authors.user: albanie
paper_title: arXiVeri: Automatic table verification with GPT
paper_summary: Without accurate transcription of numerical data in scientific documents, a
  scientist cannot draw accurate conclusions. Unfortunately, the process of
  copying numerical data from one paper to another is prone to human error. In
  this paper, we propose to meet this challenge through the novel task of
  automatic table verification (AutoTV), in which the objective is to verify the
  accuracy of numerical data in tables by cross-referencing cited sources. To
  support this task, we propose a new benchmark, arXiVeri, which comprises
  tabular data drawn from open-access academic papers on arXiv. We introduce
  metrics to evaluate the performance of a table verifier in two key areas: (i)
  table matching, which aims to identify the source table in a cited document
  that corresponds to a target table, and (ii) cell matching, which aims to
  locate shared cells between a target and source table and identify their row
  and column indices accurately. By leveraging the flexible capabilities of
  modern large language models (LLMs), we propose simple baselines for table
  verification. Our findings highlight the complexity of this task, even for
  state-of-the-art LLMs like OpenAI's GPT-4. The code and benchmark will be made
  publicly available.
paper_ai_keywords: automatic table verification (AutoTV) arXiVeri table matching cell matching row and column indices large language models (LLMs) GPT-4