title: ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and
    Understanding
summary: Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to
  growing appreciation for native multimodal large language models. However, its
  multimodal capabilities remain confined to images and text. Yet beyond images,
  the ability to understand and generate 3D content is equally crucial. To
  address this gap, we propose ShapeLLM-Omni-a native 3D large language model
  capable of understanding and generating 3D assets and text in any sequence.
  First, we train a 3D vector-quantized variational autoencoder (VQVAE), which
  maps 3D objects into a discrete latent space to achieve efficient and accurate
  shape representation and reconstruction. Building upon the 3D-aware discrete
  tokens, we innovatively construct a large-scale continuous training dataset
  named 3D-Alpaca, encompassing generation, comprehension, and editing, thus
  providing rich resources for future research and training. Finally, by
  performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on
  the 3D-Alpaca dataset. Our work provides an effective attempt at extending
  multimodal models with basic 3D capabilities, which contributes to future
  research in 3D-native AI. Project page:
  https://github.com/JAMESYJL/ShapeLLM-Omni
paper_authors._id: 683e671483a130f817c4937a
paper_authors.name: Junliang Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:36:53.278Z
paper_authors._id: 65a420cd90e65dc39a6abe9e
paper_authors.avatarUrl: /avatars/81ac5b749043e899f5017782409f9e28.svg
paper_authors.fullname: yejunliang
paper_authors.type: user
paper_authors.user: yejunliang23
paper_authors._id: 683e671483a130f817c4937b
paper_authors.name: Zhengyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:37:23.401Z
paper_authors._id: 634e15aec1ce28f1de91c470
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
paper_authors.fullname: Zhengyi Wang
paper_authors.type: user
paper_authors.user: Zhengyi
paper_authors._id: 683e671483a130f817c4937c
paper_authors.name: Ruowen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:46:03.565Z
paper_authors._id: 6522e4fbd89bc7773ddc4b58
paper_authors.avatarUrl: /avatars/3e9b158af52c5f738a3eae72dcbb3824.svg
paper_authors.fullname: Ruowen Zhao
paper_authors.type: user
paper_authors.user: zzzrw
paper_authors._id: 683e671483a130f817c4937d
paper_authors.name: Shenghao Xie
paper_authors._id: 683e671483a130f817c4937e
paper_authors.name: Jun Zhu
paper_title: ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and
    Understanding
paper_summary: Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to
  growing appreciation for native multimodal large language models. However, its
  multimodal capabilities remain confined to images and text. Yet beyond images,
  the ability to understand and generate 3D content is equally crucial. To
  address this gap, we propose ShapeLLM-Omni-a native 3D large language model
  capable of understanding and generating 3D assets and text in any sequence.
  First, we train a 3D vector-quantized variational autoencoder (VQVAE), which
  maps 3D objects into a discrete latent space to achieve efficient and accurate
  shape representation and reconstruction. Building upon the 3D-aware discrete
  tokens, we innovatively construct a large-scale continuous training dataset
  named 3D-Alpaca, encompassing generation, comprehension, and editing, thus
  providing rich resources for future research and training. Finally, by
  performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on
  the 3D-Alpaca dataset. Our work provides an effective attempt at extending
  multimodal models with basic 3D capabilities, which contributes to future
  research in 3D-native AI. Project page:
  https://github.com/JAMESYJL/ShapeLLM-Omni
paper_ai_summary: A native 3D large language model, ShapeLLM-Omni, is proposed to understand and generate 3D assets and text, trained using a 3D vector-quantized variational autoencoder and a new 3D-Alpaca dataset.
paper_ai_keywords: 3D vector-quantized variational autoencoder (VQVAE) discrete latent space instruction-based training 3D-Alpaca dataset 3D-native AI