title: VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D
    Reconstruction
summary: The rapid advancement of Large Multimodal Models (LMMs) for 2D images and
  videos has motivated extending these models to understand 3D scenes, aiming for
  human-like visual-spatial intelligence. Nevertheless, achieving deep spatial
  understanding comparable to human capabilities poses significant challenges in
  model encoding and data acquisition. Existing methods frequently depend on
  external depth sensors for geometry capture or utilize off-the-shelf algorithms
  for pre-constructing 3D maps, thereby limiting their scalability, especially
  with prevalent monocular video inputs and for time-sensitive applications. In
  this work, we introduce VLM-3R, a unified framework for Vision-Language Models
  (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes
  monocular video frames by employing a geometry encoder to derive implicit 3D
  tokens that represent spatial understanding. Leveraging our Spatial-Visual-View
  Fusion and over 200K curated 3D reconstructive instruction tuning
  question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial
  context with language instructions. This enables monocular 3D spatial
  assistance and embodied reasoning. To facilitate the evaluation of temporal
  reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark,
  featuring over 138.6K QA pairs across five distinct tasks focused on evolving
  spatial relationships. Extensive experiments demonstrate that our model,
  VLM-3R, not only facilitates robust visual-spatial reasoning but also enables
  the understanding of temporal 3D context changes, excelling in both accuracy
  and scalability.
paper_authors._id: 6837611a3216d45c6288350c
paper_authors.name: Zhiwen Fan
paper_authors._id: 6837611a3216d45c6288350d
paper_authors.name: Jian Zhang
paper_authors._id: 6837611a3216d45c6288350e
paper_authors.name: Renjie Li
paper_authors._id: 6837611a3216d45c6288350f
paper_authors.name: Junge Zhang
paper_authors._id: 6837611a3216d45c62883510
paper_authors.name: Runjin Chen
paper_authors._id: 6837611a3216d45c62883511
paper_authors.name: Hezhen Hu
paper_authors._id: 6837611a3216d45c62883512
paper_authors.name: Kevin Wang
paper_authors._id: 6837611a3216d45c62883513
paper_authors.name: Huaizhi Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:30.840Z
paper_authors._id: 6621ea7224210791d94ebde4
paper_authors.avatarUrl: /avatars/6510b1faa5042755ba80cf3c54b08781.svg
paper_authors.fullname: Huaizhi Qu
paper_authors.type: user
paper_authors.user: qhz991029
paper_authors._id: 6837611a3216d45c62883514
paper_authors.name: Dilin Wang
paper_authors._id: 6837611a3216d45c62883515
paper_authors.name: Zhicheng Yan
paper_authors._id: 6837611a3216d45c62883516
paper_authors.name: Hongyu Xu
paper_authors._id: 6837611a3216d45c62883517
paper_authors.name: Justin Theiss
paper_authors._id: 6837611a3216d45c62883518
paper_authors.name: Tianlong Chen
paper_authors._id: 6837611a3216d45c62883519
paper_authors.name: Jiachen Li
paper_authors._id: 6837611a3216d45c6288351a
paper_authors.name: Zhengzhong Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:50:56.517Z
paper_authors._id: 62548d5fef3debb2ddf91217
paper_authors.avatarUrl: /avatars/14975b45568f9c399c92c3986b6ce83e.svg
paper_authors.fullname: Zhengzhong Tu
paper_authors.type: user
paper_authors.user: vztu
paper_authors._id: 6837611a3216d45c6288351b
paper_authors.name: Zhangyang Wang
paper_authors._id: 6837611a3216d45c6288351c
paper_authors.name: Rakesh Ranjan
paper_title: VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D
    Reconstruction
paper_summary: The rapid advancement of Large Multimodal Models (LMMs) for 2D images and
  videos has motivated extending these models to understand 3D scenes, aiming for
  human-like visual-spatial intelligence. Nevertheless, achieving deep spatial
  understanding comparable to human capabilities poses significant challenges in
  model encoding and data acquisition. Existing methods frequently depend on
  external depth sensors for geometry capture or utilize off-the-shelf algorithms
  for pre-constructing 3D maps, thereby limiting their scalability, especially
  with prevalent monocular video inputs and for time-sensitive applications. In
  this work, we introduce VLM-3R, a unified framework for Vision-Language Models
  (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes
  monocular video frames by employing a geometry encoder to derive implicit 3D
  tokens that represent spatial understanding. Leveraging our Spatial-Visual-View
  Fusion and over 200K curated 3D reconstructive instruction tuning
  question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial
  context with language instructions. This enables monocular 3D spatial
  assistance and embodied reasoning. To facilitate the evaluation of temporal
  reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark,
  featuring over 138.6K QA pairs across five distinct tasks focused on evolving
  spatial relationships. Extensive experiments demonstrate that our model,
  VLM-3R, not only facilitates robust visual-spatial reasoning but also enables
  the understanding of temporal 3D context changes, excelling in both accuracy
  and scalability.
paper_ai_summary: VLM-3R, a framework for Vision-Language Models, incorporates 3D reconstructive instruction tuning to process monocular video frames and perform embodied reasoning with robust visual-spatial and temporal contextual understanding.
paper_ai_keywords: Large Multimodal Models Vision-Language Models 3D Reconstructive instruction tuning geometry encoder implicit 3D tokens Spatial-Visual-View Fusion Vision-Spatial-Temporal Intelligence benchmark embodied reasoning