title: 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large
    Language Models
summary: Learning 4D language fields to enable time-sensitive, open-ended language
  queries in dynamic scenes is essential for many real-world applications. While
  LangSplat successfully grounds CLIP features into 3D Gaussian representations,
  achieving precision and efficiency in 3D static scenes, it lacks the ability to
  handle dynamic 4D fields as CLIP, designed for static image-text tasks, cannot
  capture temporal dynamics in videos. Real-world environments are inherently
  dynamic, with object semantics evolving over time. Building a precise 4D
  language field necessitates obtaining pixel-aligned, object-wise video
  features, which current vision models struggle to achieve. To address these
  challenges, we propose 4D LangSplat, which learns 4D language fields to handle
  time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes
  efficiently. 4D LangSplat bypasses learning the language field from vision
  features and instead learns directly from text generated from object-wise video
  captions via Multimodal Large Language Models (MLLMs). Specifically, we propose
  a multimodal object-wise video prompting method, consisting of visual and text
  prompts that guide MLLMs to generate detailed, temporally consistent,
  high-quality captions for objects throughout a video. These captions are
  encoded using a Large Language Model into high-quality sentence embeddings,
  which then serve as pixel-aligned, object-specific feature supervision,
  facilitating open-vocabulary text queries through shared embedding spaces.
  Recognizing that objects in 4D scenes exhibit smooth transitions across states,
  we further propose a status deformable network to model these continuous
  changes over time effectively. Our results across multiple benchmarks
  demonstrate that 4D LangSplat attains precise and efficient results for both
  time-sensitive and time-agnostic open-vocabulary queries.
paper_authors._id: 67d3adf57360ea908cf5f0bc
paper_authors.name: Wanhua Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:56:10.748Z
paper_authors._id: 658bb7e47459b6e471b9d2e6
paper_authors.avatarUrl: /avatars/efd8051b468b4dbcb5d149479de67c58.svg
paper_authors.fullname: Wanhua Li
paper_authors.type: user
paper_authors.user: EthanTaylor
paper_authors._id: 67d3adf57360ea908cf5f0bd
paper_authors.name: Renping Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T08:56:08.159Z
paper_authors._id: 66f0d2036a483077eed42bfb
paper_authors.avatarUrl: /avatars/f7f3f726842c26b8e52c9bdd48774b8e.svg
paper_authors.fullname: Renping Zhou
paper_authors.type: user
paper_authors.user: rpzhou
paper_authors._id: 67d3adf57360ea908cf5f0be
paper_authors.name: Jiawei Zhou
paper_authors._id: 67d3adf57360ea908cf5f0bf
paper_authors.name: Yingwei Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:56:39.206Z
paper_authors._id: 6578f49ba87010c9f8a30e2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9XvodI5B4-ghh1dJr84lQ.jpeg
paper_authors.fullname: Yingwei Song
paper_authors.type: user
paper_authors.user: wrencanfly
paper_authors._id: 67d3adf57360ea908cf5f0c0
paper_authors.name: Johannes Herter
paper_authors._id: 67d3adf57360ea908cf5f0c1
paper_authors.name: Minghan Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:56:49.663Z
paper_authors._id: 642002b51ccd411979d72b18
paper_authors.avatarUrl: /avatars/0a1e7b632fb30ad39e363c3838117322.svg
paper_authors.fullname: Minghan Qin
paper_authors.type: user
paper_authors.user: Qmh
paper_authors._id: 67d3adf57360ea908cf5f0c2
paper_authors.name: Gao Huang
paper_authors._id: 67d3adf57360ea908cf5f0c3
paper_authors.name: Hanspeter Pfister
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:56:55.960Z
paper_authors._id: 62acc69e36f7c7b7f65fccca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62acc69e36f7c7b7f65fccca/S8o0XE6TaQwLU8q3QPkct.png
paper_authors.fullname: Hanspeter Pfister
paper_authors.type: user
paper_authors.user: hpfister
paper_title: 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large
    Language Models
paper_summary: Learning 4D language fields to enable time-sensitive, open-ended language
  queries in dynamic scenes is essential for many real-world applications. While
  LangSplat successfully grounds CLIP features into 3D Gaussian representations,
  achieving precision and efficiency in 3D static scenes, it lacks the ability to
  handle dynamic 4D fields as CLIP, designed for static image-text tasks, cannot
  capture temporal dynamics in videos. Real-world environments are inherently
  dynamic, with object semantics evolving over time. Building a precise 4D
  language field necessitates obtaining pixel-aligned, object-wise video
  features, which current vision models struggle to achieve. To address these
  challenges, we propose 4D LangSplat, which learns 4D language fields to handle
  time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes
  efficiently. 4D LangSplat bypasses learning the language field from vision
  features and instead learns directly from text generated from object-wise video
  captions via Multimodal Large Language Models (MLLMs). Specifically, we propose
  a multimodal object-wise video prompting method, consisting of visual and text
  prompts that guide MLLMs to generate detailed, temporally consistent,
  high-quality captions for objects throughout a video. These captions are
  encoded using a Large Language Model into high-quality sentence embeddings,
  which then serve as pixel-aligned, object-specific feature supervision,
  facilitating open-vocabulary text queries through shared embedding spaces.
  Recognizing that objects in 4D scenes exhibit smooth transitions across states,
  we further propose a status deformable network to model these continuous
  changes over time effectively. Our results across multiple benchmarks
  demonstrate that 4D LangSplat attains precise and efficient results for both
  time-sensitive and time-agnostic open-vocabulary queries.
paper_ai_summary: A new method, 4D LangSplat, uses multimodal object-wise video captions and a status deformable network to generate precise and efficient 4D language fields for dynamic scenes.
paper_ai_keywords: LangSplat CLIP features 3D Gaussian representations 4D language fields dynamic 4D fields MLLMs multimodal object-wise video prompting Large Language Model sentence embeddings pixel-aligned object-specific feature supervision status deformable network open-vocabulary queries