title: MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
    in LLMs
summary: Recent large language models (LLMs) have demonstrated versatile capabilities
  in long-context scenarios. Although some recent benchmarks have been developed
  to evaluate the long-context capabilities of LLMs, there is a lack of
  benchmarks evaluating the mathematical reasoning abilities of LLMs over long
  contexts, which is crucial for LLMs' application in real-world scenarios. In
  this paper, we introduce MathHay, an automated benchmark designed to assess the
  long-context mathematical reasoning capabilities of LLMs. Unlike previous
  benchmarks like Needle in a Haystack, which focus primarily on information
  retrieval within long texts, MathHay demands models with both
  information-seeking and complex mathematical reasoning abilities. We conduct
  extensive experiments on MathHay to assess the long-context mathematical
  reasoning abilities of eight top-performing LLMs. Even the best-performing
  model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
  long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
  the significant room for improvement on the MathHay benchmark.
paper_authors._id: 6704d8f0e8d5909d35d32eea
paper_authors.name: Lei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:18:43.902Z
paper_authors._id: 646def60df618b303b419323
paper_authors.avatarUrl: /avatars/97aa761d5255abf230304cfeade87835.svg
paper_authors.fullname: Lei Wang
paper_authors.type: user
paper_authors.user: demolei
paper_authors._id: 6704d8f0e8d5909d35d32eeb
paper_authors.name: Shan Dong
paper_authors._id: 6704d8f0e8d5909d35d32eec
paper_authors.name: Yuhui Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:34:06.566Z
paper_authors._id: 6602869253a0518b2a98cafd
paper_authors.avatarUrl: /avatars/c14b5953a716f42c83ad28147f8308ae.svg
paper_authors.fullname: Yuhui Xu
paper_authors.type: user
paper_authors.user: yuhuixu
paper_authors._id: 6704d8f0e8d5909d35d32eed
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:33:48.011Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 6704d8f0e8d5909d35d32eee
paper_authors.name: Yalu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:33:41.725Z
paper_authors._id: 656741fca637eb616aff92ba
paper_authors.avatarUrl: /avatars/6a0c7ac44948406b191d27fd2157d829.svg
paper_authors.fullname: Yalu Wang
paper_authors.type: user
paper_authors.user: lunshi
paper_authors._id: 6704d8f0e8d5909d35d32eef
paper_authors.name: Amrita Saha
paper_authors._id: 6704d8f0e8d5909d35d32ef0
paper_authors.name: Ee-Peng Lim
paper_authors._id: 6704d8f0e8d5909d35d32ef1
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:33:30.216Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 6704d8f0e8d5909d35d32ef2
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:33:24.218Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_title: MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
    in LLMs
paper_summary: Recent large language models (LLMs) have demonstrated versatile capabilities
  in long-context scenarios. Although some recent benchmarks have been developed
  to evaluate the long-context capabilities of LLMs, there is a lack of
  benchmarks evaluating the mathematical reasoning abilities of LLMs over long
  contexts, which is crucial for LLMs' application in real-world scenarios. In
  this paper, we introduce MathHay, an automated benchmark designed to assess the
  long-context mathematical reasoning capabilities of LLMs. Unlike previous
  benchmarks like Needle in a Haystack, which focus primarily on information
  retrieval within long texts, MathHay demands models with both
  information-seeking and complex mathematical reasoning abilities. We conduct
  extensive experiments on MathHay to assess the long-context mathematical
  reasoning abilities of eight top-performing LLMs. Even the best-performing
  model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
  long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
  the significant room for improvement on the MathHay benchmark.
paper_ai_summary: MathHay is an automated benchmark assessing LLMs' long-context mathematical reasoning abilities, revealing significant areas for improvement.
paper_ai_keywords: large language models long-context mathematical reasoning MathHay information-seeking complex mathematical reasoning Gemini-1.5-Pro-002