title: Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile
    Gaussian Feature Fields
summary: Recent advancements in 2D and multimodal models have achieved remarkable
  success by leveraging large-scale training on extensive datasets. However,
  extending these achievements to enable free-form interactions and high-level
  semantic operations with complex 3D/4D scenes remains challenging. This
  difficulty stems from the limited availability of large-scale, annotated 3D/4D
  or multi-view datasets, which are crucial for generalizable vision and language
  tasks such as open-vocabulary and prompt-based segmentation, language-guided
  editing, and visual question answering (VQA). In this paper, we introduce
  Feature4X, a universal framework designed to extend any functionality from 2D
  vision foundation model into the 4D realm, using only monocular video input,
  which is widely available from user-generated content. The "X" in Feature4X
  represents its versatility, enabling any task through adaptable,
  model-conditioned 4D feature field distillation. At the core of our framework
  is a dynamic optimization strategy that unifies multiple model capabilities
  into a single representation. Additionally, to the best of our knowledge,
  Feature4X is the first method to distill and lift the features of video
  foundation models (e.g. SAM2, InternVideo2) into an explicit 4D feature field
  using Gaussian Splatting. Our experiments showcase novel view segment anything,
  geometric and appearance scene editing, and free-form VQA across all time
  steps, empowered by LLMs in feedback loops. These advancements broaden the
  scope of agentic AI applications by providing a foundation for scalable,
  contextually and spatiotemporally aware systems capable of immersive dynamic 4D
  scene interaction.
paper_authors._id: 67e6019af54a34f8a989d8eb
paper_authors.name: Shijie Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:37:59.022Z
paper_authors._id: 642a276516d4d8293c9a47e8
paper_authors.avatarUrl: /avatars/80e6db8bc2544f3486b11b57858a8692.svg
paper_authors.fullname: Shijie Zhou
paper_authors.type: user
paper_authors.user: shijiezhou
paper_authors._id: 67e6019af54a34f8a989d8ec
paper_authors.name: Hui Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:12.621Z
paper_authors._id: 65c30a5ffc592f2e565ea69f
paper_authors.avatarUrl: /avatars/5b7b42fb96cc88dbeca530fedf3c2571.svg
paper_authors.fullname: Hui Ren
paper_authors.type: user
paper_authors.user: rhfeiyang
paper_authors._id: 67e6019af54a34f8a989d8ed
paper_authors.name: Yijia Weng
paper_authors._id: 67e6019af54a34f8a989d8ee
paper_authors.name: Shuwang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:13:30.134Z
paper_authors._id: 67b3c26f3d0f54ab3805954d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/beyNv649ZZ5A29CH3m8mj.png
paper_authors.fullname: Shuwang Zhang
paper_authors.type: user
paper_authors.user: ShuwangZhang00
paper_authors._id: 67e6019af54a34f8a989d8ef
paper_authors.name: Zhen Wang
paper_authors._id: 67e6019af54a34f8a989d8f0
paper_authors.name: Dejia Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:13:40.451Z
paper_authors._id: 62f687a0c58915315c4ff75d
paper_authors.avatarUrl: /avatars/b657180c7666735062782edd4f6a69c9.svg
paper_authors.fullname: Dejia Xu
paper_authors.type: user
paper_authors.user: ir1d
paper_authors._id: 67e6019af54a34f8a989d8f1
paper_authors.name: Zhiwen Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T09:16:41.739Z
paper_authors._id: 6526386e1c6a09292d8d0a22
paper_authors.avatarUrl: /avatars/471de830de2d775d35368678c1579f87.svg
paper_authors.fullname: fan
paper_authors.type: user
paper_authors.user: Fanzhiwen
paper_authors._id: 67e6019af54a34f8a989d8f2
paper_authors.name: Suya You
paper_authors._id: 67e6019af54a34f8a989d8f3
paper_authors.name: Zhangyang Wang
paper_authors._id: 67e6019af54a34f8a989d8f4
paper_authors.name: Leonidas Guibas
paper_authors._id: 67e6019af54a34f8a989d8f5
paper_authors.name: Achuta Kadambi
paper_title: Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile
    Gaussian Feature Fields
paper_summary: Recent advancements in 2D and multimodal models have achieved remarkable
  success by leveraging large-scale training on extensive datasets. However,
  extending these achievements to enable free-form interactions and high-level
  semantic operations with complex 3D/4D scenes remains challenging. This
  difficulty stems from the limited availability of large-scale, annotated 3D/4D
  or multi-view datasets, which are crucial for generalizable vision and language
  tasks such as open-vocabulary and prompt-based segmentation, language-guided
  editing, and visual question answering (VQA). In this paper, we introduce
  Feature4X, a universal framework designed to extend any functionality from 2D
  vision foundation model into the 4D realm, using only monocular video input,
  which is widely available from user-generated content. The "X" in Feature4X
  represents its versatility, enabling any task through adaptable,
  model-conditioned 4D feature field distillation. At the core of our framework
  is a dynamic optimization strategy that unifies multiple model capabilities
  into a single representation. Additionally, to the best of our knowledge,
  Feature4X is the first method to distill and lift the features of video
  foundation models (e.g. SAM2, InternVideo2) into an explicit 4D feature field
  using Gaussian Splatting. Our experiments showcase novel view segment anything,
  geometric and appearance scene editing, and free-form VQA across all time
  steps, empowered by LLMs in feedback loops. These advancements broaden the
  scope of agentic AI applications by providing a foundation for scalable,
  contextually and spatiotemporally aware systems capable of immersive dynamic 4D
  scene interaction.
paper_ai_summary: Feature4X extends 2D vision foundation models to 4D scene understanding using monocular video, enabling tasks like segmentation, editing, and VQA through Gaussian Splatting and dynamic optimization.
paper_ai_keywords: Feature4X dynamic optimization strategy 4D feature field Gaussian Splatting SAM2 InternVideo2 novel view segmentation geometric and appearance scene editing free-form VQA agentic AI scalable contextually aware spatiotemporally aware immersive dynamic 4D scene interaction