title: U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills
    in LLMs
summary: The current evaluation of mathematical skills in LLMs is limited, as existing
  benchmarks are either relatively small, primarily focus on elementary and
  high-school problems, or lack diversity in topics. Additionally, the inclusion
  of visual elements in tasks remains largely under-explored.
    To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
  unpublished open-ended university-level problems sourced from teaching
  materials. It is balanced across six core subjects, with 20% of multimodal
  problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
  judge the correctness of generated solutions. To this end, we release
  mu-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
    The evaluation of general domain, math-specific, and multimodal LLMs
  highlights the challenges presented by U-MATH. Our findings reveal that LLMs
  achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
  on visual problems. The solution assessment proves challenging for LLMs, with
  the best LLM judge having an F1-score of 80% on mu-MATH.
paper_authors._id: 675156f747111c6a7bcbbdac
paper_authors.name: Konstantin Chernyshev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T14:29:22.375Z
paper_authors._id: 6338929107f5708a1aca1390
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6338929107f5708a1aca1390/Bq_1CDfE8rb7SHTRE8CUU.png
paper_authors.fullname: Konstantin Chernyshev 
paper_authors.type: user
paper_authors.user: k4black
paper_authors._id: 675156f747111c6a7bcbbdad
paper_authors.name: Vitaliy Polshkov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T12:13:59.577Z
paper_authors._id: 650238063e61bc019201e3e2
paper_authors.avatarUrl: /avatars/5268b67a116dc24f2b0228c3d87c7cd5.svg
paper_authors.fullname: Vitaliy Polshkov
paper_authors.type: user
paper_authors.user: cogwheelhead
paper_authors._id: 675156f747111c6a7bcbbdae
paper_authors.name: Ekaterina Artemova
paper_authors._id: 675156f747111c6a7bcbbdaf
paper_authors.name: Alex Myasnikov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:29.584Z
paper_authors._id: 66da29f728520589aaa1274c
paper_authors.avatarUrl: /avatars/8bf855bf3c82bbdfa4241e339feaab75.svg
paper_authors.fullname: Alex Myasnikov
paper_authors.type: user
paper_authors.user: aseplus
paper_authors._id: 675156f747111c6a7bcbbdb0
paper_authors.name: Vlad Stepanov
paper_authors._id: 675156f747111c6a7bcbbdb1
paper_authors.name: Alexei Miasnikov
paper_authors._id: 675156f747111c6a7bcbbdb2
paper_authors.name: Sergei Tilga
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T12:22:24.344Z
paper_authors._id: 63ef69f4f338cc3b7774422b
paper_authors.avatarUrl: /avatars/3975dca73d0941898f637d16e3115872.svg
paper_authors.fullname: Sergei Tilga
paper_authors.type: user
paper_authors.user: tilgasergey
paper_title: U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills
    in LLMs
paper_summary: The current evaluation of mathematical skills in LLMs is limited, as existing
  benchmarks are either relatively small, primarily focus on elementary and
  high-school problems, or lack diversity in topics. Additionally, the inclusion
  of visual elements in tasks remains largely under-explored.
    To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
  unpublished open-ended university-level problems sourced from teaching
  materials. It is balanced across six core subjects, with 20% of multimodal
  problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
  judge the correctness of generated solutions. To this end, we release
  mu-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
    The evaluation of general domain, math-specific, and multimodal LLMs
  highlights the challenges presented by U-MATH. Our findings reveal that LLMs
  achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
  on visual problems. The solution assessment proves challenging for LLMs, with
  the best LLM judge having an F1-score of 80% on mu-MATH.
paper_ai_summary: U-MATH, a new benchmark of university-level math problems, challenges LLMs in both text-based and multimodal tasks, revealing significant limitations in their accuracy and solution assessment capabilities.
paper_ai_keywords: benchmark university-level problems multimodal problems LLM general domain math-specific F1-score