3D-VLA: A 3D Vision-Language-Action Generative World Model
Recent vision-language-action (VLA) models rely on 2D inputs, lacking
integration with the broader realm of the 3D physical world. Furthermore, they
perform action prediction by learning a direct mapping from perception to
action, neglecting the vast dynamics of the world and the relations between
actions and dynamics. In contrast, human beings are endowed with world models
that depict imagination about future scenarios to plan actions accordingly. To
this end, we propose 3D-VLA by introducing a new family of embodied foundation
models that seamlessly link 3D perception, reasoning, and action through a
generative world model. Specifically, 3D-VLA is built on top of a 3D-based
large language model (LLM), and a set of interaction tokens is introduced to
engage with the embodied environment. Furthermore, to inject generation
abilities into the model, we train a series of embodied diffusion models and
align them into the LLM for predicting the goal images and point clouds. To
train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
extracting vast 3D-related information from existing robotics datasets. Our
experiments on held-in datasets demonstrate that 3D-VLA significantly improves
the reasoning, multimodal generation, and planning capabilities in embodied
environments, showcasing its potential in real-world applications.
65f3c9dc5f86a2581bcf2d37
Haoyu Zhen
admin_assigned
2024-03-15T13:15:41.732Z
6437c7dae282b4a48eaf065e
https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
Haoyu Zhen
user
anyeZHY
65f3c9dc5f86a2581bcf2d38
Xiaowen Qiu
admin_assigned
2024-03-15T13:15:49.401Z
654568380856885240110e80
/avatars/9dc8451a7d33eb12ff80e70026558ec4.svg
Xiaowen Qiu
user
miscalculation
65f3c9dc5f86a2581bcf2d39
Peihao Chen
admin_assigned
2024-03-15T13:17:11.437Z
64bfe895c874dda773303b44
/avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
PeihaoChen
user
PeihaoChen
65f3c9dc5f86a2581bcf2d3a
Jincheng Yang
65f3c9dc5f86a2581bcf2d3b
Xin Yan
claimed_verified
2024-03-15T16:08:41.305Z
62f1fcbee2c9b4f83b4a6276
https://cdn-avatars.huggingface.co/v1/production/uploads/62f1fcbee2c9b4f83b4a6276/Vb648pHEskN7U0rZouYha.jpeg
Xin Yan
user
Cakeyan
65f3c9dc5f86a2581bcf2d3c
Yilun Du
admin_assigned
2024-03-15T13:17:32.731Z
63c9bd445fdc575773c732fe
/avatars/def472d1ab3fbf751225357c0932ae7e.svg
Yilun Du
user
yilundu
65f3c9dc5f86a2581bcf2d3d
Yining Hong
admin_assigned
2024-03-15T13:17:39.270Z
6431b64df76c34519e93d1ba
/avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
Yining Hong
user
evelynhong
65f3c9dc5f86a2581bcf2d3e
Chuang Gan
3D-VLA: A 3D Vision-Language-Action Generative World Model
Recent vision-language-action (VLA) models rely on 2D inputs, lacking
integration with the broader realm of the 3D physical world. Furthermore, they
perform action prediction by learning a direct mapping from perception to
action, neglecting the vast dynamics of the world and the relations between
actions and dynamics. In contrast, human beings are endowed with world models
that depict imagination about future scenarios to plan actions accordingly. To
this end, we propose 3D-VLA by introducing a new family of embodied foundation
models that seamlessly link 3D perception, reasoning, and action through a
generative world model. Specifically, 3D-VLA is built on top of a 3D-based
large language model (LLM), and a set of interaction tokens is introduced to
engage with the embodied environment. Furthermore, to inject generation
abilities into the model, we train a series of embodied diffusion models and
align them into the LLM for predicting the goal images and point clouds. To
train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
extracting vast 3D-related information from existing robotics datasets. Our
experiments on held-in datasets demonstrate that 3D-VLA significantly improves
the reasoning, multimodal generation, and planning capabilities in embodied
environments, showcasing its potential in real-world applications.
3D-VLA integrates 3D perception, reasoning, and action through a generative world model, improving capabilities in embodied environments with the help of embodied diffusion models.
3D-VLA embodied foundation models 3D-based large language model interaction tokens embodied diffusion models 3D embodied instruction dataset multimodal generation