title: MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
summary: We present MetaSpatial, the first reinforcement learning (RL)-based framework
  designed to enhance 3D spatial reasoning in vision-language models (VLMs),
  enabling real-time 3D scene generation without the need for hard-coded
  optimizations. MetaSpatial addresses two core challenges: (i) the lack of
  internalized 3D spatial reasoning in VLMs, which limits their ability to
  generate realistic layouts, and (ii) the inefficiency of traditional supervised
  fine-tuning (SFT) for layout generation tasks, as perfect ground truth
  annotations are unavailable. Our key innovation is a multi-turn RL-based
  optimization mechanism that integrates physics-aware constraints and rendered
  image evaluations, ensuring generated 3D layouts are coherent, physically
  plausible, and aesthetically consistent. Methodologically, MetaSpatial
  introduces an adaptive, iterative reasoning process, where the VLM refines
  spatial arrangements over multiple turns by analyzing rendered outputs,
  improving scene coherence progressively. Empirical evaluations demonstrate that
  MetaSpatial significantly enhances the spatial consistency and formatting
  stability of various scale models. Post-training, object placements are more
  realistic, aligned, and functionally coherent, validating the effectiveness of
  RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game
  development applications. Our code, data, and training pipeline are publicly
  available at https://github.com/PzySeere/MetaSpatial.
paper_authors._id: 67e23d7ddb11e1d38226cafd
paper_authors.name: Zhenyu Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:19:54.569Z
paper_authors._id: 669794c5813d96b4eb0b3fd6
paper_authors.avatarUrl: /avatars/b4e2bb7b07cc281932d783dcbb64f211.svg
paper_authors.fullname: Zhenyu Pan
paper_authors.type: user
paper_authors.user: zhenyupan
paper_authors._id: 67e23d7ddb11e1d38226cafe
paper_authors.name: Han Liu
paper_title: MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
paper_summary: We present MetaSpatial, the first reinforcement learning (RL)-based framework
  designed to enhance 3D spatial reasoning in vision-language models (VLMs),
  enabling real-time 3D scene generation without the need for hard-coded
  optimizations. MetaSpatial addresses two core challenges: (i) the lack of
  internalized 3D spatial reasoning in VLMs, which limits their ability to
  generate realistic layouts, and (ii) the inefficiency of traditional supervised
  fine-tuning (SFT) for layout generation tasks, as perfect ground truth
  annotations are unavailable. Our key innovation is a multi-turn RL-based
  optimization mechanism that integrates physics-aware constraints and rendered
  image evaluations, ensuring generated 3D layouts are coherent, physically
  plausible, and aesthetically consistent. Methodologically, MetaSpatial
  introduces an adaptive, iterative reasoning process, where the VLM refines
  spatial arrangements over multiple turns by analyzing rendered outputs,
  improving scene coherence progressively. Empirical evaluations demonstrate that
  MetaSpatial significantly enhances the spatial consistency and formatting
  stability of various scale models. Post-training, object placements are more
  realistic, aligned, and functionally coherent, validating the effectiveness of
  RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game
  development applications. Our code, data, and training pipeline are publicly
  available at https://github.com/PzySeere/MetaSpatial.
paper_ai_summary: MetaSpatial is a reinforcement learning framework that enhances 3D spatial reasoning in vision-language models, enabling realistic and coherent 3D scene generation through iterative refinement and physics-aware constraints.
paper_ai_keywords: reinforcement learning vision-language models 3D spatial reasoning real-time 3D scene generation hard-coded optimizations multi-turn RL physics-aware constraints rendered image evaluations adaptive reasoning process empirical evaluations metaverse AR/VR digital twins game development