DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical
  Reasoning Robustness of Vision Language Models
The rapid advancements in Vision-Language Models (VLMs) have shown great
potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with
minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
in these scenarios, revealing limitations in their mathematical reasoning
capabilities. In this paper, we investigate the mathematical reasoning
robustness in VLMs and evaluate how well these models perform under different
variants of the same question, such as changes in visual numerical values or
function graphs. While several vision-based math benchmarks have been developed
to assess VLMs' problem-solving capabilities, these benchmarks contain only
static sets of problems and cannot easily evaluate mathematical reasoning
robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
high-quality, multi-topic seed questions, each represented as a Python program.
Those programs are carefully designed and annotated to enable the automatic
generation of a much larger set of concrete questions, including many different
types of visual and textual variations. DynaMath allows us to evaluate the
generalization ability of VLMs, by assessing their performance under varying
input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
generated concrete questions. Our results show that the worst-case model
accuracy, defined as the percentage of correctly answered seed questions in all
10 variants, is significantly lower than the average-case accuracy. Our
analysis emphasizes the need to study the robustness of VLMs' reasoning
abilities, and DynaMath provides valuable insights to guide the development of
more reliable models for mathematical reasoning.
67297e6f6cc63cc1ba29886a
Chengke Zou
admin_assigned
2024-11-05T14:03:14.097Z
660a37f54fa3a72a97b09546
/avatars/d6306e87e9343abfd16c1fe08a2f9358.svg
Chengke Zou
user
OwenZou
67297e6f6cc63cc1ba29886b
Xingang Guo
claimed_verified
2024-11-15T09:38:25.601Z
6524837973a0f19d06fdc851
/avatars/2bc59dddf66ff05afe83bb0711a8a5a3.svg
Xingang Guo
user
optizer
67297e6f6cc63cc1ba29886c
Rui Yang
claimed_verified
2024-11-05T07:58:11.859Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
user
Ray2333
67297e6f6cc63cc1ba29886d
Junyu Zhang
admin_assigned
2024-11-05T14:03:42.928Z
6719bfd07c6e6c83a388aeae
https://cdn-avatars.huggingface.co/v1/production/uploads/6719bfd07c6e6c83a388aeae/jHxryk04dzHo23TX5F5sz.png
Junyu Zhang
user
jyzhang1208
67297e6f6cc63cc1ba29886e
Bin Hu
extracted_pending
2025-02-17T22:52:47.594Z
67b3bd862f8e45db9fc2eadf
/avatars/d53454493c22978d13e9ca410a0159c7.svg
Bin Hu
user
huxxx221
67297e6f6cc63cc1ba29886f
Huan Zhang
admin_assigned
2024-11-05T14:04:05.546Z
6719d581a6cad13741b8bc7f
https://cdn-avatars.huggingface.co/v1/production/uploads/6719d581a6cad13741b8bc7f/w4EttqfXRgWZJc6HpYOS9.jpeg
Huan Zhang
user
huanzhang12
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical
  Reasoning Robustness of Vision Language Models
The rapid advancements in Vision-Language Models (VLMs) have shown great
potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with
minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
in these scenarios, revealing limitations in their mathematical reasoning
capabilities. In this paper, we investigate the mathematical reasoning
robustness in VLMs and evaluate how well these models perform under different
variants of the same question, such as changes in visual numerical values or
function graphs. While several vision-based math benchmarks have been developed
to assess VLMs' problem-solving capabilities, these benchmarks contain only
static sets of problems and cannot easily evaluate mathematical reasoning
robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
high-quality, multi-topic seed questions, each represented as a Python program.
Those programs are carefully designed and annotated to enable the automatic
generation of a much larger set of concrete questions, including many different
types of visual and textual variations. DynaMath allows us to evaluate the
generalization ability of VLMs, by assessing their performance under varying
input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
generated concrete questions. Our results show that the worst-case model
accuracy, defined as the percentage of correctly answered seed questions in all
10 variants, is significantly lower than the average-case accuracy. Our
analysis emphasizes the need to study the robustness of VLMs' reasoning
abilities, and DynaMath provides valuable insights to guide the development of
more reliable models for mathematical reasoning.
DynaMath, a dynamic visual math benchmark, evaluates the robustness of Vision-Language Models in mathematical reasoning across various input conditions, revealing significant accuracy differences between average and worst-case scenarios.
Vision-Language Models VLMs DynaMath mathematical reasoning generalization ability seed questions concrete questions visual variations textual variations worst-case model accuracy average-case accuracy