title: DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical
    Reasoning Robustness of Vision Language Models
summary: The rapid advancements in Vision-Language Models (VLMs) have shown great
  potential in tackling mathematical reasoning tasks that involve visual context.
  Unlike humans who can reliably apply solution steps to similar problems with
  minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
  in these scenarios, revealing limitations in their mathematical reasoning
  capabilities. In this paper, we investigate the mathematical reasoning
  robustness in VLMs and evaluate how well these models perform under different
  variants of the same question, such as changes in visual numerical values or
  function graphs. While several vision-based math benchmarks have been developed
  to assess VLMs' problem-solving capabilities, these benchmarks contain only
  static sets of problems and cannot easily evaluate mathematical reasoning
  robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
  benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
  high-quality, multi-topic seed questions, each represented as a Python program.
  Those programs are carefully designed and annotated to enable the automatic
  generation of a much larger set of concrete questions, including many different
  types of visual and textual variations. DynaMath allows us to evaluate the
  generalization ability of VLMs, by assessing their performance under varying
  input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
  generated concrete questions. Our results show that the worst-case model
  accuracy, defined as the percentage of correctly answered seed questions in all
  10 variants, is significantly lower than the average-case accuracy. Our
  analysis emphasizes the need to study the robustness of VLMs' reasoning
  abilities, and DynaMath provides valuable insights to guide the development of
  more reliable models for mathematical reasoning.
paper_authors._id: 67297e6f6cc63cc1ba29886a
paper_authors.name: Chengke Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:03:14.097Z
paper_authors._id: 660a37f54fa3a72a97b09546
paper_authors.avatarUrl: /avatars/d6306e87e9343abfd16c1fe08a2f9358.svg
paper_authors.fullname: Chengke Zou
paper_authors.type: user
paper_authors.user: OwenZou
paper_authors._id: 67297e6f6cc63cc1ba29886b
paper_authors.name: Xingang Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:38:25.601Z
paper_authors._id: 6524837973a0f19d06fdc851
paper_authors.avatarUrl: /avatars/2bc59dddf66ff05afe83bb0711a8a5a3.svg
paper_authors.fullname: Xingang Guo
paper_authors.type: user
paper_authors.user: optizer
paper_authors._id: 67297e6f6cc63cc1ba29886c
paper_authors.name: Rui Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:58:11.859Z
paper_authors._id: 64d45451c34a346181b130dd
paper_authors.avatarUrl: /avatars/9bb8205b889337df5d321539c9b5d69d.svg
paper_authors.fullname: Rui Yang
paper_authors.type: user
paper_authors.user: Ray2333
paper_authors._id: 67297e6f6cc63cc1ba29886d
paper_authors.name: Junyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:03:42.928Z
paper_authors._id: 6719bfd07c6e6c83a388aeae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6719bfd07c6e6c83a388aeae/jHxryk04dzHo23TX5F5sz.png
paper_authors.fullname: Junyu Zhang
paper_authors.type: user
paper_authors.user: jyzhang1208
paper_authors._id: 67297e6f6cc63cc1ba29886e
paper_authors.name: Bin Hu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-17T22:52:47.594Z
paper_authors._id: 67b3bd862f8e45db9fc2eadf
paper_authors.avatarUrl: /avatars/d53454493c22978d13e9ca410a0159c7.svg
paper_authors.fullname: Bin Hu
paper_authors.type: user
paper_authors.user: huxxx221
paper_authors._id: 67297e6f6cc63cc1ba29886f
paper_authors.name: Huan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:04:05.546Z
paper_authors._id: 6719d581a6cad13741b8bc7f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
paper_authors.fullname: Huan Zhang
paper_authors.type: user
paper_authors.user: huanzhang12
paper_title: DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical
    Reasoning Robustness of Vision Language Models
paper_summary: The rapid advancements in Vision-Language Models (VLMs) have shown great
  potential in tackling mathematical reasoning tasks that involve visual context.
  Unlike humans who can reliably apply solution steps to similar problems with
  minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
  in these scenarios, revealing limitations in their mathematical reasoning
  capabilities. In this paper, we investigate the mathematical reasoning
  robustness in VLMs and evaluate how well these models perform under different
  variants of the same question, such as changes in visual numerical values or
  function graphs. While several vision-based math benchmarks have been developed
  to assess VLMs' problem-solving capabilities, these benchmarks contain only
  static sets of problems and cannot easily evaluate mathematical reasoning
  robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
  benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
  high-quality, multi-topic seed questions, each represented as a Python program.
  Those programs are carefully designed and annotated to enable the automatic
  generation of a much larger set of concrete questions, including many different
  types of visual and textual variations. DynaMath allows us to evaluate the
  generalization ability of VLMs, by assessing their performance under varying
  input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
  generated concrete questions. Our results show that the worst-case model
  accuracy, defined as the percentage of correctly answered seed questions in all
  10 variants, is significantly lower than the average-case accuracy. Our
  analysis emphasizes the need to study the robustness of VLMs' reasoning
  abilities, and DynaMath provides valuable insights to guide the development of
  more reliable models for mathematical reasoning.
paper_ai_keywords: Vision-Language Models (VLMs) mathematical reasoning tasks visual context solution steps mathematical reasoning robustness visual numerical values function graphs static sets of problems DynaMath dynamic visual math benchmark high-quality, multi-topic seed questions Python program automatic generation generalization ability worst-case model accuracy average-case accuracy neural learning models (SOTA VLMs)