title: AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via
    GRPO
summary: Large Language Models (LLMs) have demonstrated impressive capabilities in
  language processing, yet they often struggle with tasks requiring genuine
  visual spatial reasoning. In this paper, we introduce a novel two-stage
  training framework designed to equip standard LLMs with visual reasoning
  abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT)
  on a curated dataset of tokenized maze representations to teach the model to
  predict step-by-step movement commands. Next, we apply Group Relative Policy
  Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted
  reward function to refine the model's sequential decision-making and encourage
  emergent chain-of-thought behaviors. Experimental results on synthetically
  generated mazes show that while a baseline model fails to navigate the maze,
  the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning
  boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more
  robust and self-corrective reasoning, highlighting the potential of our
  approach to bridge the gap between language models and visual spatial tasks.
  These findings offer promising implications for applications in robotics,
  autonomous navigation, and other domains that require integrated visual and
  sequential reasoning.
paper_authors._id: 67b7eeddaf9f1b1bd95b878b
paper_authors.name: Alan Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T15:03:59.165Z
paper_authors._id: 62d7b2339b629105a5d6888a
paper_authors.avatarUrl: /avatars/c3f164fde6b8f9a671890e08ce8a3e75.svg
paper_authors.fullname: Alan Dao
paper_authors.type: user
paper_authors.user: alandao
paper_authors._id: 67b7eeddaf9f1b1bd95b878c
paper_authors.name: Dinh Bach Vu
paper_title: AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via
    GRPO
paper_summary: Large Language Models (LLMs) have demonstrated impressive capabilities in
  language processing, yet they often struggle with tasks requiring genuine
  visual spatial reasoning. In this paper, we introduce a novel two-stage
  training framework designed to equip standard LLMs with visual reasoning
  abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT)
  on a curated dataset of tokenized maze representations to teach the model to
  predict step-by-step movement commands. Next, we apply Group Relative Policy
  Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted
  reward function to refine the model's sequential decision-making and encourage
  emergent chain-of-thought behaviors. Experimental results on synthetically
  generated mazes show that while a baseline model fails to navigate the maze,
  the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning
  boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more
  robust and self-corrective reasoning, highlighting the potential of our
  approach to bridge the gap between language models and visual spatial tasks.
  These findings offer promising implications for applications in robotics,
  autonomous navigation, and other domains that require integrated visual and
  sequential reasoning.
paper_ai_keywords: Supervised Fine Tuning (SFT) Group Relative Policy Optimization (GRPO) Schwart's technique (used in DeepSeekR1) tokenized maze representations step-by-step movement commands sequentially decision-making chain-of-thought behaviors robust and self-corrective behavior