The Jumping Reasoning Curve? Tracking the Evolution of Reasoning
  Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles
The releases of OpenAI's o1 and o3 mark a significant paradigm shift in Large
Language Models towards advanced reasoning capabilities. Notably, o3
outperformed humans in novel problem-solving and skill acquisition on the
Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI).
However, this benchmark is limited to symbolic patterns, whereas humans often
perceive and reason about multimodal scenarios involving both vision and
language data. Thus, there is an urgent need to investigate advanced reasoning
capabilities in multimodal tasks. To this end, we track the evolution of the
GPT-[n] and o-[n] series models on challenging multimodal puzzles, requiring
fine-grained visual perception with abstract or algorithmic reasoning. The
superior performance of o1 comes at nearly 750 times the computational cost of
GPT-4o, raising concerns about its efficiency. Our results reveal a clear
upward trend in reasoning capabilities across model iterations, with notable
performance jumps across GPT-series models and subsequently to o1. Nonetheless,
we observe that the o1 model still struggles with simple multimodal puzzles
requiring abstract reasoning. Furthermore, its performance in algorithmic
puzzles remains poor. We plan to continuously track new models in the series
and update our results in this paper accordingly. All resources used in this
evaluation are openly available https://github.com/declare-lab/LLM-PuzzleTest.
67a1a56d83c3565727d22f0c
Vernon Y. H. Toh
claimed_verified
2025-03-06T13:34:27.733Z
638488b3be729f36b2ec46d4
https://cdn-avatars.huggingface.co/v1/production/uploads/638488b3be729f36b2ec46d4/p0FJyzBKWlql6QJ7OtN_i.jpeg
Vernon Y.H. Toh
user
vernontoh
67a1a56d83c3565727d22f0d
Yew Ken Chia
67a1a56d83c3565727d22f0e
Deepanway Ghosal
admin_assigned
2025-02-04T17:40:36.537Z
62eced5e7e89e8d34df1a1ea
/avatars/99c9d8ba7e7722b7524d5d687cf96a25.svg
Deepanway Ghosal
user
dghosal
67a1a56d83c3565727d22f0f
Soujanya Poria
admin_assigned
2025-02-04T17:40:30.882Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
Soujanya Poria
user
soujanyaporia
The Jumping Reasoning Curve? Tracking the Evolution of Reasoning
  Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles
The releases of OpenAI's o1 and o3 mark a significant paradigm shift in Large
Language Models towards advanced reasoning capabilities. Notably, o3
outperformed humans in novel problem-solving and skill acquisition on the
Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI).
However, this benchmark is limited to symbolic patterns, whereas humans often
perceive and reason about multimodal scenarios involving both vision and
language data. Thus, there is an urgent need to investigate advanced reasoning
capabilities in multimodal tasks. To this end, we track the evolution of the
GPT-[n] and o-[n] series models on challenging multimodal puzzles, requiring
fine-grained visual perception with abstract or algorithmic reasoning. The
superior performance of o1 comes at nearly 750 times the computational cost of
GPT-4o, raising concerns about its efficiency. Our results reveal a clear
upward trend in reasoning capabilities across model iterations, with notable
performance jumps across GPT-series models and subsequently to o1. Nonetheless,
we observe that the o1 model still struggles with simple multimodal puzzles
requiring abstract reasoning. Furthermore, its performance in algorithmic
puzzles remains poor. We plan to continuously track new models in the series
and update our results in this paper accordingly. All resources used in this
evaluation are openly available https://github.com/declare-lab/LLM-PuzzleTest.
Advancements in OpenAI's o-series models show improved reasoning capabilities compared to GPT models, particularly on symbolic pattern tasks like ARC-AGI, but they face challenges in multimodal and algorithmic reasoning puzzles.
Large Language Models advanced reasoning capabilities Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) multimodal tasks GPT-series models o-series models fine-grained visual perception algorithmic reasoning