title: The Jumping Reasoning Curve? Tracking the Evolution of Reasoning
    Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles
summary: The releases of OpenAI's o1 and o3 mark a significant paradigm shift in Large
  Language Models towards advanced reasoning capabilities. Notably, o3
  outperformed humans in novel problem-solving and skill acquisition on the
  Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI).
  However, this benchmark is limited to symbolic patterns, whereas humans often
  perceive and reason about multimodal scenarios involving both vision and
  language data. Thus, there is an urgent need to investigate advanced reasoning
  capabilities in multimodal tasks. To this end, we track the evolution of the
  GPT-[n] and o-[n] series models on challenging multimodal puzzles, requiring
  fine-grained visual perception with abstract or algorithmic reasoning. The
  superior performance of o1 comes at nearly 750 times the computational cost of
  GPT-4o, raising concerns about its efficiency. Our results reveal a clear
  upward trend in reasoning capabilities across model iterations, with notable
  performance jumps across GPT-series models and subsequently to o1. Nonetheless,
  we observe that the o1 model still struggles with simple multimodal puzzles
  requiring abstract reasoning. Furthermore, its performance in algorithmic
  puzzles remains poor. We plan to continuously track new models in the series
  and update our results in this paper accordingly. All resources used in this
  evaluation are openly available https://github.com/declare-lab/LLM-PuzzleTest.
paper_authors._id: 67a1a56d83c3565727d22f0c
paper_authors.name: Vernon Y. H. Toh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T13:34:27.733Z
paper_authors._id: 638488b3be729f36b2ec46d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638488b3be729f36b2ec46d4/p0FJyzBKWlql6QJ7OtN_i.jpeg
paper_authors.fullname: Vernon Y.H. Toh
paper_authors.type: user
paper_authors.user: vernontoh
paper_authors._id: 67a1a56d83c3565727d22f0d
paper_authors.name: Yew Ken Chia
paper_authors._id: 67a1a56d83c3565727d22f0e
paper_authors.name: Deepanway Ghosal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:40:36.537Z
paper_authors._id: 62eced5e7e89e8d34df1a1ea
paper_authors.avatarUrl: /avatars/99c9d8ba7e7722b7524d5d687cf96a25.svg
paper_authors.fullname: Deepanway Ghosal
paper_authors.type: user
paper_authors.user: dghosal
paper_authors._id: 67a1a56d83c3565727d22f0f
paper_authors.name: Soujanya Poria
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:40:30.882Z
paper_authors._id: 626b626405fe1cb65725aca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/aa-Lata46I3fXOmMetvXH.jpeg
paper_authors.fullname: Soujanya Poria
paper_authors.type: user
paper_authors.user: soujanyaporia
paper_title: The Jumping Reasoning Curve? Tracking the Evolution of Reasoning
    Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles
paper_summary: The releases of OpenAI's o1 and o3 mark a significant paradigm shift in Large
  Language Models towards advanced reasoning capabilities. Notably, o3
  outperformed humans in novel problem-solving and skill acquisition on the
  Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI).
  However, this benchmark is limited to symbolic patterns, whereas humans often
  perceive and reason about multimodal scenarios involving both vision and
  language data. Thus, there is an urgent need to investigate advanced reasoning
  capabilities in multimodal tasks. To this end, we track the evolution of the
  GPT-[n] and o-[n] series models on challenging multimodal puzzles, requiring
  fine-grained visual perception with abstract or algorithmic reasoning. The
  superior performance of o1 comes at nearly 750 times the computational cost of
  GPT-4o, raising concerns about its efficiency. Our results reveal a clear
  upward trend in reasoning capabilities across model iterations, with notable
  performance jumps across GPT-series models and subsequently to o1. Nonetheless,
  we observe that the o1 model still struggles with simple multimodal puzzles
  requiring abstract reasoning. Furthermore, its performance in algorithmic
  puzzles remains poor. We plan to continuously track new models in the series
  and update our results in this paper accordingly. All resources used in this
  evaluation are openly available https://github.com/declare-lab/LLM-PuzzleTest.
paper_ai_summary: Advancements in OpenAI's o-series models show improved reasoning capabilities compared to GPT models, particularly on symbolic pattern tasks like ARC-AGI, but they face challenges in multimodal and algorithmic reasoning puzzles.
paper_ai_keywords: Large Language Models advanced reasoning capabilities Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) multimodal tasks GPT-series models o-series models fine-grained visual perception algorithmic reasoning