LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with
  TriMap Video Diffusion
Recovering 3D structures with open-vocabulary scene understanding from 2D
images is a fundamental but daunting task. Recent developments have achieved
this by performing per-scene optimization with embedded language information.
However, they heavily rely on the calibrated dense-view reconstruction
paradigm, thereby suffering from severe rendering artifacts and implausible
semantic synthesis when limited views are available. In this paper, we
introduce a novel generative framework, coined LangScene-X, to unify and
generate 3D consistent multi-modality information for reconstruction and
understanding. Powered by the generative capability of creating more consistent
novel observations, we can build generalizable 3D language-embedded scenes from
only sparse views. Specifically, we first train a TriMap video diffusion model
that can generate appearance (RGBs), geometry (normals), and semantics
(segmentation maps) from sparse inputs through progressive knowledge
integration. Furthermore, we propose a Language Quantized Compressor (LQC),
trained on large-scale image datasets, to efficiently encode language
embeddings, enabling cross-scene generalization without per-scene retraining.
Finally, we reconstruct the language surface fields by aligning language
information onto the surface of 3D scenes, enabling open-ended language
queries. Extensive experiments on real-world data demonstrate the superiority
of our LangScene-X over state-of-the-art methods in terms of quality and
generalizability. Project Page: https://liuff19.github.io/LangScene-X.
686735e69db35afc9c304ce1
Fangfu Liu
686735e69db35afc9c304ce2
Hao Li
686735e69db35afc9c304ce3
Jiawei Chi
claimed_verified
2025-07-07T11:24:57.924Z
66e14cf2fb009ef598305fe5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/etohfcD9GKaRPir1HRP-5.jpeg
Jiawei Chi
user
chijw
686735e69db35afc9c304ce4
Hanyang Wang
claimed_verified
2025-07-04T07:49:31.644Z
65c38f6c137aba2aee524989
/avatars/a93e29f55876df3e65e2532972e057e4.svg
Hanyang Wang
user
hanyang-21
686735e69db35afc9c304ce5
Minghui Yang
686735e69db35afc9c304ce6
Fudong Wang
686735e69db35afc9c304ce7
Yueqi Duan
LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with
  TriMap Video Diffusion
Recovering 3D structures with open-vocabulary scene understanding from 2D
images is a fundamental but daunting task. Recent developments have achieved
this by performing per-scene optimization with embedded language information.
However, they heavily rely on the calibrated dense-view reconstruction
paradigm, thereby suffering from severe rendering artifacts and implausible
semantic synthesis when limited views are available. In this paper, we
introduce a novel generative framework, coined LangScene-X, to unify and
generate 3D consistent multi-modality information for reconstruction and
understanding. Powered by the generative capability of creating more consistent
novel observations, we can build generalizable 3D language-embedded scenes from
only sparse views. Specifically, we first train a TriMap video diffusion model
that can generate appearance (RGBs), geometry (normals), and semantics
(segmentation maps) from sparse inputs through progressive knowledge
integration. Furthermore, we propose a Language Quantized Compressor (LQC),
trained on large-scale image datasets, to efficiently encode language
embeddings, enabling cross-scene generalization without per-scene retraining.
Finally, we reconstruct the language surface fields by aligning language
information onto the surface of 3D scenes, enabling open-ended language
queries. Extensive experiments on real-world data demonstrate the superiority
of our LangScene-X over state-of-the-art methods in terms of quality and
generalizability. Project Page: https://liuff19.github.io/LangScene-X.
LangScene-X uses a TriMap video diffusion model and Language Quantized Compressor to generate 3D consistent multi-modality information from sparse 2D views, enabling open-ended language queries and superior generalization.
TriMap video diffusion model Language Quantized Compressor appearance geometry semantics progressive knowledge integration language embeddings language surface fields open-ended language queries