title: LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
summary: Multi-step spatial reasoning entails understanding and reasoning about
  spatial relationships across multiple sequential steps, which is crucial for
  tackling complex real-world applications, such as robotic manipulation,
  autonomous navigation, and automated assembly. To assess how well current
  Multimodal Large Language Models (MLLMs) have acquired this fundamental
  capability, we introduce LEGO-Puzzles, a scalable benchmark designed
  to evaluate both spatial understanding and sequential
  reasoning in MLLMs through LEGO-based tasks. LEGO-Puzzles consists of 1,100
  carefully curated visual question-answering (VQA) samples spanning 11 distinct
  tasks, ranging from basic spatial understanding to complex multi-step
  reasoning. Based on LEGO-Puzzles, we conduct a comprehensive evaluation of
  state-of-the-art MLLMs and uncover significant limitations in their spatial
  reasoning capabilities: even the most powerful MLLMs can answer only about half
  of the test cases, whereas human participants achieve over 90\% accuracy. In
  addition to VQA tasks, we evaluate MLLMs' abilities to generate LEGO images
  following assembly illustrations. Our experiments show that only
  Gemini-2.0-Flash and GPT-4o exhibit a limited ability to follow these
  instructions, while other MLLMs either replicate the input image or generate
  completely irrelevant outputs. Overall, LEGO-Puzzles exposes critical
  deficiencies in existing MLLMs' spatial understanding and sequential reasoning
  capabilities, and underscores the need for further advancements in multimodal
  spatial reasoning.
paper_authors._id: 67e4d3df7e97884ba4150ec0
paper_authors.name: Kexian Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:46:12.696Z
paper_authors._id: 662516d72419feed62fb3a0a
paper_authors.avatarUrl: /avatars/24c4157829e70a4e346aa984885aa5ad.svg
paper_authors.fullname: Kexian Tang
paper_authors.type: user
paper_authors.user: KexianTang
paper_authors._id: 67e4d3df7e97884ba4150ec1
paper_authors.name: Junyao Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:03:15.061Z
paper_authors._id: 64a6ae0e0437599198cf3a98
paper_authors.avatarUrl: /avatars/6635432cc0589ba12dc170cad6986d6d.svg
paper_authors.fullname: Junyao Gao
paper_authors.type: user
paper_authors.user: favourisnotyou
paper_authors._id: 67e4d3df7e97884ba4150ec2
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:46:24.775Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 67e4d3df7e97884ba4150ec3
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:00:33.733Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 67e4d3df7e97884ba4150ec4
paper_authors.name: Yanan Sun
paper_authors._id: 67e4d3df7e97884ba4150ec5
paper_authors.name: Zhening Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:00:36.341Z
paper_authors._id: 62fb2a9dc95d426ff8f74c8d
paper_authors.avatarUrl: /avatars/25c1a68ee7b7d0cc7e9f56bde37f4914.svg
paper_authors.fullname: Zhening Xing
paper_authors.type: user
paper_authors.user: Leoxing
paper_authors._id: 67e4d3df7e97884ba4150ec6
paper_authors.name: Wenran Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:46:45.866Z
paper_authors._id: 6385f8598b5acae8d24caf16
paper_authors.avatarUrl: /avatars/9d261f95d24e882157b987b8827098be.svg
paper_authors.fullname: liuwenran
paper_authors.type: user
paper_authors.user: lwrshi1965
paper_authors._id: 67e4d3df7e97884ba4150ec7
paper_authors.name: Kaifeng Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T09:46:52.341Z
paper_authors._id: 6414230a0fcefcf72e5085dd
paper_authors.avatarUrl: /avatars/3a38dc8c84b0f27af846184d1c19f6ef.svg
paper_authors.fullname: Kaifeng Lyu
paper_authors.type: user
paper_authors.user: vfleaking
paper_authors._id: 67e4d3df7e97884ba4150ec8
paper_authors.name: Kai Chen
paper_title: LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?
paper_summary: Multi-step spatial reasoning entails understanding and reasoning about
  spatial relationships across multiple sequential steps, which is crucial for
  tackling complex real-world applications, such as robotic manipulation,
  autonomous navigation, and automated assembly. To assess how well current
  Multimodal Large Language Models (MLLMs) have acquired this fundamental
  capability, we introduce LEGO-Puzzles, a scalable benchmark designed
  to evaluate both spatial understanding and sequential
  reasoning in MLLMs through LEGO-based tasks. LEGO-Puzzles consists of 1,100
  carefully curated visual question-answering (VQA) samples spanning 11 distinct
  tasks, ranging from basic spatial understanding to complex multi-step
  reasoning. Based on LEGO-Puzzles, we conduct a comprehensive evaluation of
  state-of-the-art MLLMs and uncover significant limitations in their spatial
  reasoning capabilities: even the most powerful MLLMs can answer only about half
  of the test cases, whereas human participants achieve over 90\% accuracy. In
  addition to VQA tasks, we evaluate MLLMs' abilities to generate LEGO images
  following assembly illustrations. Our experiments show that only
  Gemini-2.0-Flash and GPT-4o exhibit a limited ability to follow these
  instructions, while other MLLMs either replicate the input image or generate
  completely irrelevant outputs. Overall, LEGO-Puzzles exposes critical
  deficiencies in existing MLLMs' spatial understanding and sequential reasoning
  capabilities, and underscores the need for further advancements in multimodal
  spatial reasoning.
paper_ai_summary: LEGO-Puzzles evaluates the spatial understanding and sequential reasoning capabilities of Multimodal Large Language Models (MLLMs) through LEGO-based tasks, revealing significant limitations compared to human performance.
paper_ai_keywords: spatial reasoning sequential reasoning Multimodal Large Language Models (MLLMs) LEGO-Puzzles visual question-answering (VQA) spatial understanding Gemini-2.0-Flash GPT-4o