Aligning Text, Images, and 3D Structure Token-by-Token
Creating machines capable of understanding the world in 3D is essential in
assisting designers that build and edit 3D environments and robots navigating
and interacting within a three-dimensional space. Inspired by advances in
language and image modeling, we investigate the potential of autoregressive
models for a new modality: structured 3D scenes. To this end, we propose a
unified LLM framework that aligns language, images, and 3D scenes and provide a
detailed ''cookbook'' outlining critical design choices for achieving optimal
training and performance addressing key questions related to data
representation, modality-specific objectives, and more. We evaluate performance
across four core 3D tasks -- rendering, recognition, instruction-following, and
question-answering -- and four 3D datasets, synthetic and real-world. We extend
our approach to reconstruct complex 3D object shapes by enriching our 3D
modality with quantized shape encodings, and show our model's effectiveness on
real-world 3D object recognition tasks. Project webpage:
https://glab-caltech.github.io/kyvo/
6848f8c242e4f9106973f2f6
Aadarsh Sahoo
6848f8c242e4f9106973f2f7
Vansh Tibrewal
6848f8c242e4f9106973f2f8
Georgia Gkioxari
claimed_verified
2025-06-29T10:14:23.078Z
634ededbd049354d7ee4b557
https://cdn-avatars.huggingface.co/v1/production/uploads/634ededbd049354d7ee4b557/XJS3qqpExVDFCKjT0NMi8.jpeg
Georgia Gkioxari
user
gkioxari
Aligning Text, Images, and 3D Structure Token-by-Token
Creating machines capable of understanding the world in 3D is essential in
assisting designers that build and edit 3D environments and robots navigating
and interacting within a three-dimensional space. Inspired by advances in
language and image modeling, we investigate the potential of autoregressive
models for a new modality: structured 3D scenes. To this end, we propose a
unified LLM framework that aligns language, images, and 3D scenes and provide a
detailed ''cookbook'' outlining critical design choices for achieving optimal
training and performance addressing key questions related to data
representation, modality-specific objectives, and more. We evaluate performance
across four core 3D tasks -- rendering, recognition, instruction-following, and
question-answering -- and four 3D datasets, synthetic and real-world. We extend
our approach to reconstruct complex 3D object shapes by enriching our 3D
modality with quantized shape encodings, and show our model's effectiveness on
real-world 3D object recognition tasks. Project webpage:
https://glab-caltech.github.io/kyvo/
A unified language, image, and 3D scene model framework is proposed, achieving optimal training and performance across various 3D tasks and datasets.
autoregressive models LLM framework data representation modality-specific objectives 3D rendering 3D recognition instruction-following question-answering 3D datasets quantized shape encodings