ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and
  Understanding
Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to
growing appreciation for native multimodal large language models. However, its
multimodal capabilities remain confined to images and text. Yet beyond images,
the ability to understand and generate 3D content is equally crucial. To
address this gap, we propose ShapeLLM-Omni-a native 3D large language model
capable of understanding and generating 3D assets and text in any sequence.
First, we train a 3D vector-quantized variational autoencoder (VQVAE), which
maps 3D objects into a discrete latent space to achieve efficient and accurate
shape representation and reconstruction. Building upon the 3D-aware discrete
tokens, we innovatively construct a large-scale continuous training dataset
named 3D-Alpaca, encompassing generation, comprehension, and editing, thus
providing rich resources for future research and training. Finally, by
performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on
the 3D-Alpaca dataset. Our work provides an effective attempt at extending
multimodal models with basic 3D capabilities, which contributes to future
research in 3D-native AI. Project page:
https://github.com/JAMESYJL/ShapeLLM-Omni
683e671483a130f817c4937a
Junliang Ye
admin_assigned
2025-06-03T12:36:53.278Z
65a420cd90e65dc39a6abe9e
/avatars/81ac5b749043e899f5017782409f9e28.svg
yejunliang
user
yejunliang23
683e671483a130f817c4937b
Zhengyi Wang
admin_assigned
2025-06-03T12:37:23.401Z
634e15aec1ce28f1de91c470
https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
Zhengyi Wang
user
Zhengyi
683e671483a130f817c4937c
Ruowen Zhao
claimed_verified
2025-06-03T08:46:03.565Z
6522e4fbd89bc7773ddc4b58
/avatars/3e9b158af52c5f738a3eae72dcbb3824.svg
Ruowen Zhao
user
zzzrw
683e671483a130f817c4937d
Shenghao Xie
683e671483a130f817c4937e
Jun Zhu
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and
  Understanding
Recently, the powerful text-to-image capabilities of ChatGPT-4o have led to
growing appreciation for native multimodal large language models. However, its
multimodal capabilities remain confined to images and text. Yet beyond images,
the ability to understand and generate 3D content is equally crucial. To
address this gap, we propose ShapeLLM-Omni-a native 3D large language model
capable of understanding and generating 3D assets and text in any sequence.
First, we train a 3D vector-quantized variational autoencoder (VQVAE), which
maps 3D objects into a discrete latent space to achieve efficient and accurate
shape representation and reconstruction. Building upon the 3D-aware discrete
tokens, we innovatively construct a large-scale continuous training dataset
named 3D-Alpaca, encompassing generation, comprehension, and editing, thus
providing rich resources for future research and training. Finally, by
performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on
the 3D-Alpaca dataset. Our work provides an effective attempt at extending
multimodal models with basic 3D capabilities, which contributes to future
research in 3D-native AI. Project page:
https://github.com/JAMESYJL/ShapeLLM-Omni
A native 3D large language model, ShapeLLM-Omni, is proposed to understand and generate 3D assets and text, trained using a 3D vector-quantized variational autoencoder and a new 3D-Alpaca dataset.
3D vector-quantized variational autoencoder (VQVAE) discrete latent space instruction-based training 3D-Alpaca dataset 3D-native AI