title: Pushing Auto-regressive Models for 3D Shape Generation at Capacity and
    Scalability
summary: Auto-regressive models have achieved impressive results in 2D image
  generation by modeling joint distributions in grid space. In this paper, we
  extend auto-regressive models to 3D domains, and seek a stronger ability of 3D
  shape generation by improving auto-regressive models at capacity and
  scalability simultaneously. Firstly, we leverage an ensemble of publicly
  available 3D datasets to facilitate the training of large-scale models. It
  consists of a comprehensive collection of approximately 900,000 objects, with
  multiple properties of meshes, points, voxels, rendered images, and text
  captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our
  model to learn from a wide range of object variations. However, directly
  applying 3D auto-regression encounters critical challenges of high
  computational demands on volumetric grids and ambiguous auto-regressive order
  along grid dimensions, resulting in inferior quality of 3D shapes. To this end,
  we then present a novel framework Argus3D in terms of capacity. Concretely, our
  approach introduces discrete representation learning based on a latent vector
  instead of volumetric grids, which not only reduces computational costs but
  also preserves essential geometric details by learning the joint distributions
  in a more tractable order. The capacity of conditional generation can thus be
  realized by simply concatenating various conditioning inputs to the latent
  vector, such as point clouds, categories, images, and texts. In addition,
  thanks to the simplicity of our model architecture, we naturally scale up our
  approach to a larger model with an impressive 3.6 billion parameters, further
  enhancing the quality of versatile 3D generation. Extensive experiments on four
  generation tasks demonstrate that Argus3D can synthesize diverse and faithful
  shapes across multiple categories, achieving remarkable performance.
paper_authors._id: 65d424d0c3df2fd80eebd14b
paper_authors.name: Xuelin Qian
paper_authors._id: 65d424d0c3df2fd80eebd14c
paper_authors.name: Yu Wang
paper_authors._id: 65d424d0c3df2fd80eebd14d
paper_authors.name: Simian Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:59:54.705Z
paper_authors._id: 652a4f174d0eef4a2bf85096
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ofYN04WWtD60oeDP45pQN.jpeg
paper_authors.fullname: Simian Luo
paper_authors.type: user
paper_authors.user: SimianLuo
paper_authors._id: 65d424d0c3df2fd80eebd14e
paper_authors.name: Yinda Zhang
paper_authors._id: 65d424d0c3df2fd80eebd14f
paper_authors.name: Ying Tai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T11:00:21.165Z
paper_authors._id: 65734004769f3ee9bde1af10
paper_authors.avatarUrl: /avatars/d6310ed861972fd691687d8f47413f33.svg
paper_authors.fullname: Ying Tai
paper_authors.type: user
paper_authors.user: yingtai
paper_authors._id: 65d424d0c3df2fd80eebd150
paper_authors.name: Zhenyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T11:01:42.415Z
paper_authors._id: 642e6cc9be01b88c9446fe3d
paper_authors.avatarUrl: /avatars/43a8f9bf35bea54bcda547174d921aa4.svg
paper_authors.fullname: Zhenyu Zhang
paper_authors.type: user
paper_authors.user: JesseZhang
paper_authors._id: 65d424d0c3df2fd80eebd151
paper_authors.name: Chengjie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T11:01:49.947Z
paper_authors._id: 65729087379284b904a6d81d
paper_authors.avatarUrl: /avatars/0e7968b1d9221d523a54e3ac787e449d.svg
paper_authors.fullname: Chengjie Wang
paper_authors.type: user
paper_authors.user: chengjie-wang
paper_authors._id: 65d424d0c3df2fd80eebd152
paper_authors.name: Xiangyang Xue
paper_authors._id: 65d424d0c3df2fd80eebd153
paper_authors.name: Bo Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:38:49.386Z
paper_authors._id: 64acafac1aee69ece03af05a
paper_authors.avatarUrl: /avatars/6380777852546dbd432fed5544d5db0b.svg
paper_authors.fullname: Bo Zhao
paper_authors.type: user
paper_authors.user: Z-LIRA
paper_authors._id: 65d424d0c3df2fd80eebd154
paper_authors.name: Tiejun Huang
paper_authors._id: 65d424d0c3df2fd80eebd155
paper_authors.name: Yunsheng Wu
paper_authors._id: 65d424d0c3df2fd80eebd156
paper_authors.name: Yanwei Fu
paper_title: Pushing Auto-regressive Models for 3D Shape Generation at Capacity and
    Scalability
paper_summary: Auto-regressive models have achieved impressive results in 2D image
  generation by modeling joint distributions in grid space. In this paper, we
  extend auto-regressive models to 3D domains, and seek a stronger ability of 3D
  shape generation by improving auto-regressive models at capacity and
  scalability simultaneously. Firstly, we leverage an ensemble of publicly
  available 3D datasets to facilitate the training of large-scale models. It
  consists of a comprehensive collection of approximately 900,000 objects, with
  multiple properties of meshes, points, voxels, rendered images, and text
  captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our
  model to learn from a wide range of object variations. However, directly
  applying 3D auto-regression encounters critical challenges of high
  computational demands on volumetric grids and ambiguous auto-regressive order
  along grid dimensions, resulting in inferior quality of 3D shapes. To this end,
  we then present a novel framework Argus3D in terms of capacity. Concretely, our
  approach introduces discrete representation learning based on a latent vector
  instead of volumetric grids, which not only reduces computational costs but
  also preserves essential geometric details by learning the joint distributions
  in a more tractable order. The capacity of conditional generation can thus be
  realized by simply concatenating various conditioning inputs to the latent
  vector, such as point clouds, categories, images, and texts. In addition,
  thanks to the simplicity of our model architecture, we naturally scale up our
  approach to a larger model with an impressive 3.6 billion parameters, further
  enhancing the quality of versatile 3D generation. Extensive experiments on four
  generation tasks demonstrate that Argus3D can synthesize diverse and faithful
  shapes across multiple categories, achieving remarkable performance.
paper_ai_summary: Argus3D framework uses discrete representation learning with a latent vector to enhance 3D shape generation capacity and scalability, significantly improving performance across various generation tasks.
paper_ai_keywords: auto-regressive models volumetric grids latent vector discrete representation learning conditional generation point clouds categories images texts Argus3D