SAR3D: Autoregressive 3D Object Generation and Understanding via
  Multi-scale 3D VQVAE
Autoregressive models have demonstrated remarkable success across various
fields, from large language models (LLMs) to large multimodal models (LMMs) and
2D content generation, moving closer to artificial general intelligence (AGI).
Despite these advances, applying autoregressive approaches to 3D object
generation and understanding remains largely unexplored. This paper introduces
Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale
3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for
efficient autoregressive generation and detailed understanding. By predicting
the next scale in a multi-scale latent representation instead of the next
single token, SAR3D reduces generation time significantly, achieving fast 3D
object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the
tokens enriched with hierarchical 3D-aware information, we finetune a
pretrained LLM on them, enabling multimodal comprehension of 3D content. Our
experiments show that SAR3D surpasses current 3D generation methods in both
speed and quality and allows LLMs to interpret and caption 3D models
comprehensively.
674693666b75c045a3f54a54
Yongwei Chen
claimed_verified
2024-11-27T22:20:09.353Z
6458c809b1202755ec5cf33a
/avatars/0cd92dbb4e85484926c65e5582d89e55.svg
cyw-3d
user
cyw-3d
674693666b75c045a3f54a55
Yushi Lan
674693666b75c045a3f54a56
Shangchen Zhou
claimed_verified
2025-06-11T16:03:10.320Z
62e57662ae9d3f10acbb1b1b
https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
Shangchen Zhou
user
sczhou
674693666b75c045a3f54a57
Tengfei Wang
674693666b75c045a3f54a58
XIngang Pan
SAR3D: Autoregressive 3D Object Generation and Understanding via
  Multi-scale 3D VQVAE
Autoregressive models have demonstrated remarkable success across various
fields, from large language models (LLMs) to large multimodal models (LMMs) and
2D content generation, moving closer to artificial general intelligence (AGI).
Despite these advances, applying autoregressive approaches to 3D object
generation and understanding remains largely unexplored. This paper introduces
Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale
3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for
efficient autoregressive generation and detailed understanding. By predicting
the next scale in a multi-scale latent representation instead of the next
single token, SAR3D reduces generation time significantly, achieving fast 3D
object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the
tokens enriched with hierarchical 3D-aware information, we finetune a
pretrained LLM on them, enabling multimodal comprehension of 3D content. Our
experiments show that SAR3D surpasses current 3D generation methods in both
speed and quality and allows LLMs to interpret and caption 3D models
comprehensively.
Multi-scale autoregressive generation using a 3D vector-quantized variational autoencoder (VQVAE) framework achieves fast and high-quality 3D object generation and supports multimodal comprehension by fine-tuning LLMs.
autoregressive models large language models (LLMs) large multimodal models (LMMs) 2D content generation artificial general intelligence (AGI) 3D object generation 3D object understanding Scale AutoRegressive 3D (SAR3D) multi-scale 3D vector-quantized variational autoencoder (VQVAE) hierarchical 3D-aware information multimodal comprehension