title: Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
summary: Large language models (LLMs) have showcased remarkable reasoning
  capabilities, yet they remain susceptible to errors, particularly in temporal
  reasoning tasks involving complex temporal logic. Existing research has
  explored LLM performance on temporal reasoning using diverse datasets and
  benchmarks. However, these studies often rely on real-world data that LLMs may
  have encountered during pre-training or employ anonymization techniques that
  can inadvertently introduce factual inconsistencies. In this work, we address
  these limitations by introducing novel synthetic datasets specifically designed
  to assess LLM temporal reasoning abilities in various scenarios. The diversity
  of question types across these datasets enables systematic investigation into
  the impact of the problem structure, size, question type, fact order, and other
  factors on LLM performance. Our findings provide valuable insights into the
  strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster
  further research in this area, we are open-sourcing the datasets and evaluation
  framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
paper_authors._id: 666bb0c8848be12c2a559cf9
paper_authors.name: Bahare Fatemi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:22:39.108Z
paper_authors._id: 652def9065a4619fb5d7788e
paper_authors.avatarUrl: /avatars/aef5deb8774dee96e8d832be621a710b.svg
paper_authors.fullname: Bahare
paper_authors.type: user
paper_authors.user: baharef
paper_authors._id: 666bb0c8848be12c2a559cfa
paper_authors.name: Mehran Kazemi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:16:53.409Z
paper_authors._id: 66675877c610850071487bd5
paper_authors.avatarUrl: /avatars/a70e49e319f9716bead091dd8d6ca4de.svg
paper_authors.fullname: Mehran Kazemi
paper_authors.type: user
paper_authors.user: mehrankazemi
paper_authors._id: 666bb0c8848be12c2a559cfb
paper_authors.name: Anton Tsitsulin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:16:59.260Z
paper_authors._id: 63ab8e5c8a357d5057ae659e
paper_authors.avatarUrl: /avatars/0eed94fcdb6d7c86259bc5b72b5bd41e.svg
paper_authors.fullname: Anton Tsitsulin
paper_authors.type: user
paper_authors.user: xgfs
paper_authors._id: 666bb0c8848be12c2a559cfc
paper_authors.name: Karishma Malkan
paper_authors._id: 666bb0c8848be12c2a559cfd
paper_authors.name: Jinyeong Yim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:17:08.343Z
paper_authors._id: 63a08a503c8841cfe2c7298d
paper_authors.avatarUrl: /avatars/f8e912ce8c69474414d0daf1efb93efa.svg
paper_authors.fullname: Jinyeong Yim
paper_authors.type: user
paper_authors.user: jinyeong
paper_authors._id: 666bb0c8848be12c2a559cfe
paper_authors.name: John Palowitch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:17:13.988Z
paper_authors._id: 665ca85e8d6e9206bb932ba7
paper_authors.avatarUrl: /avatars/6252ed2e80ae2bcb006be123e7764672.svg
paper_authors.fullname: John Palowitch
paper_authors.type: user
paper_authors.user: jpalowitch
paper_authors._id: 666bb0c8848be12c2a559cff
paper_authors.name: Sungyong Seo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T07:45:59.272Z
paper_authors._id: 666c6aa0ea11ab8f28c6a94e
paper_authors.avatarUrl: /avatars/b95dd78742252b032f8702e9618a41af.svg
paper_authors.fullname: Sungyong Seo
paper_authors.type: user
paper_authors.user: sungyong12
paper_authors._id: 666bb0c8848be12c2a559d00
paper_authors.name: Jonathan Halcrow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:17:23.808Z
paper_authors._id: 64a844a9f972a36c53a09396
paper_authors.avatarUrl: /avatars/8476944468bee3d75c2429bbf0f08b2f.svg
paper_authors.fullname: Jonathan Halcrow
paper_authors.type: user
paper_authors.user: jhalcrow
paper_authors._id: 666bb0c8848be12c2a559d01
paper_authors.name: Bryan Perozzi
paper_title: Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
paper_summary: Large language models (LLMs) have showcased remarkable reasoning
  capabilities, yet they remain susceptible to errors, particularly in temporal
  reasoning tasks involving complex temporal logic. Existing research has
  explored LLM performance on temporal reasoning using diverse datasets and
  benchmarks. However, these studies often rely on real-world data that LLMs may
  have encountered during pre-training or employ anonymization techniques that
  can inadvertently introduce factual inconsistencies. In this work, we address
  these limitations by introducing novel synthetic datasets specifically designed
  to assess LLM temporal reasoning abilities in various scenarios. The diversity
  of question types across these datasets enables systematic investigation into
  the impact of the problem structure, size, question type, fact order, and other
  factors on LLM performance. Our findings provide valuable insights into the
  strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster
  further research in this area, we are open-sourcing the datasets and evaluation
  framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
paper_ai_summary: Synthetic datasets and an evaluation framework are introduced to assess Large Language Models' capabilities in various temporal reasoning scenarios.
paper_ai_keywords: large language models temporal reasoning complex temporal logic synthetic datasets problem structure question types fact order evaluation framework