title: Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision
    Geometry Priors
summary: Previous research has investigated the application of Multimodal Large
  Language Models (MLLMs) in understanding 3D scenes by interpreting them as
  videos. These approaches generally depend on comprehensive 3D data inputs, such
  as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research,
  we advance this field by enhancing the capability of MLLMs to understand and
  reason in 3D spaces directly from video data, without the need for additional
  3D input. We propose a novel and efficient method, the Video-3D Geometry Large
  Language Model (VG LLM). Our approach employs a 3D visual geometry encoder that
  extracts 3D prior information from video sequences. This information is
  integrated with visual tokens and fed into the MLLM. Extensive experiments have
  shown that our method has achieved substantial improvements in various tasks
  related to 3D scene understanding and spatial reasoning, all directly learned
  from video sources. Impressively, our 4B model, which does not rely on explicit
  3D data inputs, achieves competitive results compared to existing
  state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the
  VSI-Bench evaluations.
paper_authors._id: 683e5569fce31842c60675d7
paper_authors.name: Duo Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:46:45.196Z
paper_authors._id: 646e2fcaf813cfe153f1af6c
paper_authors.avatarUrl: /avatars/2f87d0e5c071000990da29cd744bc03d.svg
paper_authors.fullname: Duo Zheng
paper_authors.type: user
paper_authors.user: zd11024
paper_authors._id: 683e5569fce31842c60675d8
paper_authors.name: Shijia Huang
paper_authors._id: 683e5569fce31842c60675d9
paper_authors.name: Yanyang Li
paper_authors._id: 683e5569fce31842c60675da
paper_authors.name: Liwei Wang
paper_title: Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision
    Geometry Priors
paper_summary: Previous research has investigated the application of Multimodal Large
  Language Models (MLLMs) in understanding 3D scenes by interpreting them as
  videos. These approaches generally depend on comprehensive 3D data inputs, such
  as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research,
  we advance this field by enhancing the capability of MLLMs to understand and
  reason in 3D spaces directly from video data, without the need for additional
  3D input. We propose a novel and efficient method, the Video-3D Geometry Large
  Language Model (VG LLM). Our approach employs a 3D visual geometry encoder that
  extracts 3D prior information from video sequences. This information is
  integrated with visual tokens and fed into the MLLM. Extensive experiments have
  shown that our method has achieved substantial improvements in various tasks
  related to 3D scene understanding and spatial reasoning, all directly learned
  from video sources. Impressively, our 4B model, which does not rely on explicit
  3D data inputs, achieves competitive results compared to existing
  state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the
  VSI-Bench evaluations.
paper_ai_summary: A novel Video-3D Geometry Large Language Model (VG LLM) extracts 3D information directly from video sequences to enhance 3D scene understanding without additional 3D data, achieving competitive results in various tasks.
paper_ai_keywords: MLLMs Video-3D Geometry Large Language Model VG LLM 3D visual geometry encoder VSI-Bench