title: 3D-VLA: A 3D Vision-Language-Action Generative World Model
summary: Recent vision-language-action (VLA) models rely on 2D inputs, lacking
  integration with the broader realm of the 3D physical world. Furthermore, they
  perform action prediction by learning a direct mapping from perception to
  action, neglecting the vast dynamics of the world and the relations between
  actions and dynamics. In contrast, human beings are endowed with world models
  that depict imagination about future scenarios to plan actions accordingly. To
  this end, we propose 3D-VLA by introducing a new family of embodied foundation
  models that seamlessly link 3D perception, reasoning, and action through a
  generative world model. Specifically, 3D-VLA is built on top of a 3D-based
  large language model (LLM), and a set of interaction tokens is introduced to
  engage with the embodied environment. Furthermore, to inject generation
  abilities into the model, we train a series of embodied diffusion models and
  align them into the LLM for predicting the goal images and point clouds. To
  train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
  extracting vast 3D-related information from existing robotics datasets. Our
  experiments on held-in datasets demonstrate that 3D-VLA significantly improves
  the reasoning, multimodal generation, and planning capabilities in embodied
  environments, showcasing its potential in real-world applications.
paper_authors._id: 65f3c9dc5f86a2581bcf2d37
paper_authors.name: Haoyu Zhen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:15:41.732Z
paper_authors._id: 6437c7dae282b4a48eaf065e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
paper_authors.fullname: Haoyu Zhen
paper_authors.type: user
paper_authors.user: anyeZHY
paper_authors._id: 65f3c9dc5f86a2581bcf2d38
paper_authors.name: Xiaowen Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:15:49.401Z
paper_authors._id: 654568380856885240110e80
paper_authors.avatarUrl: /avatars/9dc8451a7d33eb12ff80e70026558ec4.svg
paper_authors.fullname: Xiaowen Qiu
paper_authors.type: user
paper_authors.user: miscalculation
paper_authors._id: 65f3c9dc5f86a2581bcf2d39
paper_authors.name: Peihao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:17:11.437Z
paper_authors._id: 64bfe895c874dda773303b44
paper_authors.avatarUrl: /avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
paper_authors.fullname: PeihaoChen
paper_authors.type: user
paper_authors.user: PeihaoChen
paper_authors._id: 65f3c9dc5f86a2581bcf2d3a
paper_authors.name: Jincheng Yang
paper_authors._id: 65f3c9dc5f86a2581bcf2d3b
paper_authors.name: Xin Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T16:08:41.305Z
paper_authors._id: 62f1fcbee2c9b4f83b4a6276
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f1fcbee2c9b4f83b4a6276/Vb648pHEskN7U0rZouYha.jpeg
paper_authors.fullname: Xin Yan
paper_authors.type: user
paper_authors.user: Cakeyan
paper_authors._id: 65f3c9dc5f86a2581bcf2d3c
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:17:32.731Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 65f3c9dc5f86a2581bcf2d3d
paper_authors.name: Yining Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:17:39.270Z
paper_authors._id: 6431b64df76c34519e93d1ba
paper_authors.avatarUrl: /avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
paper_authors.fullname: Yining Hong
paper_authors.type: user
paper_authors.user: evelynhong
paper_authors._id: 65f3c9dc5f86a2581bcf2d3e
paper_authors.name: Chuang Gan
paper_title: 3D-VLA: A 3D Vision-Language-Action Generative World Model
paper_summary: Recent vision-language-action (VLA) models rely on 2D inputs, lacking
  integration with the broader realm of the 3D physical world. Furthermore, they
  perform action prediction by learning a direct mapping from perception to
  action, neglecting the vast dynamics of the world and the relations between
  actions and dynamics. In contrast, human beings are endowed with world models
  that depict imagination about future scenarios to plan actions accordingly. To
  this end, we propose 3D-VLA by introducing a new family of embodied foundation
  models that seamlessly link 3D perception, reasoning, and action through a
  generative world model. Specifically, 3D-VLA is built on top of a 3D-based
  large language model (LLM), and a set of interaction tokens is introduced to
  engage with the embodied environment. Furthermore, to inject generation
  abilities into the model, we train a series of embodied diffusion models and
  align them into the LLM for predicting the goal images and point clouds. To
  train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
  extracting vast 3D-related information from existing robotics datasets. Our
  experiments on held-in datasets demonstrate that 3D-VLA significantly improves
  the reasoning, multimodal generation, and planning capabilities in embodied
  environments, showcasing its potential in real-world applications.
paper_ai_summary: 3D-VLA integrates 3D perception, reasoning, and action through a generative world model, improving capabilities in embodied environments with the help of embodied diffusion models.
paper_ai_keywords: 3D-VLA embodied foundation models 3D-based large language model interaction tokens embodied diffusion models 3D embodied instruction dataset multimodal generation