Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Large language models (LLMs) have showcased remarkable reasoning
capabilities, yet they remain susceptible to errors, particularly in temporal
reasoning tasks involving complex temporal logic. Existing research has
explored LLM performance on temporal reasoning using diverse datasets and
benchmarks. However, these studies often rely on real-world data that LLMs may
have encountered during pre-training or employ anonymization techniques that
can inadvertently introduce factual inconsistencies. In this work, we address
these limitations by introducing novel synthetic datasets specifically designed
to assess LLM temporal reasoning abilities in various scenarios. The diversity
of question types across these datasets enables systematic investigation into
the impact of the problem structure, size, question type, fact order, and other
factors on LLM performance. Our findings provide valuable insights into the
strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster
further research in this area, we are open-sourcing the datasets and evaluation
framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
666bb0c8848be12c2a559cf9
Bahare Fatemi
claimed_verified
2024-06-14T07:22:39.108Z
652def9065a4619fb5d7788e
/avatars/aef5deb8774dee96e8d832be621a710b.svg
Bahare
user
baharef
666bb0c8848be12c2a559cfa
Mehran Kazemi
admin_assigned
2024-06-14T10:16:53.409Z
66675877c610850071487bd5
/avatars/a70e49e319f9716bead091dd8d6ca4de.svg
Mehran Kazemi
user
mehrankazemi
666bb0c8848be12c2a559cfb
Anton Tsitsulin
admin_assigned
2024-06-14T10:16:59.260Z
63ab8e5c8a357d5057ae659e
/avatars/0eed94fcdb6d7c86259bc5b72b5bd41e.svg
Anton Tsitsulin
user
xgfs
666bb0c8848be12c2a559cfc
Karishma Malkan
666bb0c8848be12c2a559cfd
Jinyeong Yim
admin_assigned
2024-06-14T10:17:08.343Z
63a08a503c8841cfe2c7298d
/avatars/f8e912ce8c69474414d0daf1efb93efa.svg
Jinyeong Yim
user
jinyeong
666bb0c8848be12c2a559cfe
John Palowitch
admin_assigned
2024-06-14T10:17:13.988Z
665ca85e8d6e9206bb932ba7
/avatars/6252ed2e80ae2bcb006be123e7764672.svg
John Palowitch
user
jpalowitch
666bb0c8848be12c2a559cff
Sungyong Seo
claimed_verified
2024-06-15T07:45:59.272Z
666c6aa0ea11ab8f28c6a94e
/avatars/b95dd78742252b032f8702e9618a41af.svg
Sungyong Seo
user
sungyong12
666bb0c8848be12c2a559d00
Jonathan Halcrow
admin_assigned
2024-06-14T10:17:23.808Z
64a844a9f972a36c53a09396
/avatars/8476944468bee3d75c2429bbf0f08b2f.svg
Jonathan Halcrow
user
jhalcrow
666bb0c8848be12c2a559d01
Bryan Perozzi
Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning
Large language models (LLMs) have showcased remarkable reasoning
capabilities, yet they remain susceptible to errors, particularly in temporal
reasoning tasks involving complex temporal logic. Existing research has
explored LLM performance on temporal reasoning using diverse datasets and
benchmarks. However, these studies often rely on real-world data that LLMs may
have encountered during pre-training or employ anonymization techniques that
can inadvertently introduce factual inconsistencies. In this work, we address
these limitations by introducing novel synthetic datasets specifically designed
to assess LLM temporal reasoning abilities in various scenarios. The diversity
of question types across these datasets enables systematic investigation into
the impact of the problem structure, size, question type, fact order, and other
factors on LLM performance. Our findings provide valuable insights into the
strengths and weaknesses of current LLMs in temporal reasoning tasks. To foster
further research in this area, we are open-sourcing the datasets and evaluation
framework used in our experiments: https://huggingface.co/datasets/baharef/ToT.
Synthetic datasets and an evaluation framework are introduced to assess Large Language Models' capabilities in various temporal reasoning scenarios.
large language models temporal reasoning complex temporal logic synthetic datasets problem structure question types fact order evaluation framework