title: SAR3D: Autoregressive 3D Object Generation and Understanding via
    Multi-scale 3D VQVAE
summary: Autoregressive models have demonstrated remarkable success across various
  fields, from large language models (LLMs) to large multimodal models (LMMs) and
  2D content generation, moving closer to artificial general intelligence (AGI).
  Despite these advances, applying autoregressive approaches to 3D object
  generation and understanding remains largely unexplored. This paper introduces
  Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale
  3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for
  efficient autoregressive generation and detailed understanding. By predicting
  the next scale in a multi-scale latent representation instead of the next
  single token, SAR3D reduces generation time significantly, achieving fast 3D
  object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the
  tokens enriched with hierarchical 3D-aware information, we finetune a
  pretrained LLM on them, enabling multimodal comprehension of 3D content. Our
  experiments show that SAR3D surpasses current 3D generation methods in both
  speed and quality and allows LLMs to interpret and caption 3D models
  comprehensively.
paper_authors._id: 674693666b75c045a3f54a54
paper_authors.name: Yongwei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:09.353Z
paper_authors._id: 6458c809b1202755ec5cf33a
paper_authors.avatarUrl: /avatars/0cd92dbb4e85484926c65e5582d89e55.svg
paper_authors.fullname: cyw-3d
paper_authors.type: user
paper_authors.user: cyw-3d
paper_authors._id: 674693666b75c045a3f54a55
paper_authors.name: Yushi Lan
paper_authors._id: 674693666b75c045a3f54a56
paper_authors.name: Shangchen Zhou
paper_authors._id: 674693666b75c045a3f54a57
paper_authors.name: Tengfei Wang
paper_authors._id: 674693666b75c045a3f54a58
paper_authors.name: XIngang Pan
paper_title: SAR3D: Autoregressive 3D Object Generation and Understanding via
    Multi-scale 3D VQVAE
paper_summary: Autoregressive models have demonstrated remarkable success across various
  fields, from large language models (LLMs) to large multimodal models (LMMs) and
  2D content generation, moving closer to artificial general intelligence (AGI).
  Despite these advances, applying autoregressive approaches to 3D object
  generation and understanding remains largely unexplored. This paper introduces
  Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale
  3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for
  efficient autoregressive generation and detailed understanding. By predicting
  the next scale in a multi-scale latent representation instead of the next
  single token, SAR3D reduces generation time significantly, achieving fast 3D
  object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the
  tokens enriched with hierarchical 3D-aware information, we finetune a
  pretrained LLM on them, enabling multimodal comprehension of 3D content. Our
  experiments show that SAR3D surpasses current 3D generation methods in both
  speed and quality and allows LLMs to interpret and caption 3D models
  comprehensively.
paper_ai_summary: Multi-scale autoregressive generation using a 3D vector-quantized variational autoencoder (VQVAE) framework achieves fast and high-quality 3D object generation and supports multimodal comprehension by fine-tuning LLMs.
paper_ai_keywords: autoregressive models large language models (LLMs) large multimodal models (LMMs) 2D content generation artificial general intelligence (AGI) 3D object generation 3D object understanding Scale AutoRegressive 3D (SAR3D) multi-scale 3D vector-quantized variational autoencoder (VQVAE) hierarchical 3D-aware information multimodal comprehension