title: CLAY: A Controllable Large-scale Generative Model for Creating
    High-quality 3D Assets
summary: In the realm of digital creativity, our potential to craft intricate 3D
  worlds from imagination is often hampered by the limitations of existing
  digital tools, which demand extensive expertise and efforts. To narrow this
  disparity, we introduce CLAY, a 3D geometry and material generator designed to
  effortlessly transform human imagination into intricate 3D digital structures.
  CLAY supports classic text or image inputs as well as 3D-aware controls from
  diverse primitives (multi-view images, voxels, bounding boxes, point clouds,
  implicit representations, etc). At its core is a large-scale generative model
  composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic
  latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a
  diverse range of 3D geometries. Specifically, it adopts neural fields to
  represent continuous and complete surfaces and uses a geometry generative
  module with pure transformer blocks in latent space. We present a progressive
  training scheme to train CLAY on an ultra large 3D model dataset obtained
  through a carefully designed processing pipeline, resulting in a 3D native
  geometry generator with 1.5 billion parameters. For appearance generation, CLAY
  sets out to produce physically-based rendering (PBR) textures by employing a
  multi-view material diffusion model that can generate 2K resolution textures
  with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for
  a range of controllable 3D asset creations, from sketchy conceptual designs to
  production ready assets with intricate details. Even first time users can
  easily use CLAY to bring their vivid 3D imaginations to life, unleashing
  unlimited creativity.
paper_authors._id: 66750403b3bf319fd9608248
paper_authors.name: Longwen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:41:05.873Z
paper_authors._id: 636d12455aaed143cd665607
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679399015950-636d12455aaed143cd665607.png
paper_authors.fullname: ZLW
paper_authors.type: user
paper_authors.user: ZarkLngeW
paper_authors._id: 66750403b3bf319fd9608249
paper_authors.name: Ziyu Wang
paper_authors._id: 66750403b3bf319fd960824a
paper_authors.name: Qixuan Zhang
paper_authors._id: 66750403b3bf319fd960824b
paper_authors.name: Qiwei Qiu
paper_authors._id: 66750403b3bf319fd960824c
paper_authors.name: Anqi Pang
paper_authors._id: 66750403b3bf319fd960824d
paper_authors.name: Haoran Jiang
paper_authors._id: 66750403b3bf319fd960824e
paper_authors.name: Wei Yang
paper_authors._id: 66750403b3bf319fd960824f
paper_authors.name: Lan Xu
paper_authors._id: 66750403b3bf319fd9608250
paper_authors.name: Jingyi Yu
paper_title: CLAY: A Controllable Large-scale Generative Model for Creating
    High-quality 3D Assets
paper_summary: In the realm of digital creativity, our potential to craft intricate 3D
  worlds from imagination is often hampered by the limitations of existing
  digital tools, which demand extensive expertise and efforts. To narrow this
  disparity, we introduce CLAY, a 3D geometry and material generator designed to
  effortlessly transform human imagination into intricate 3D digital structures.
  CLAY supports classic text or image inputs as well as 3D-aware controls from
  diverse primitives (multi-view images, voxels, bounding boxes, point clouds,
  implicit representations, etc). At its core is a large-scale generative model
  composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic
  latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a
  diverse range of 3D geometries. Specifically, it adopts neural fields to
  represent continuous and complete surfaces and uses a geometry generative
  module with pure transformer blocks in latent space. We present a progressive
  training scheme to train CLAY on an ultra large 3D model dataset obtained
  through a carefully designed processing pipeline, resulting in a 3D native
  geometry generator with 1.5 billion parameters. For appearance generation, CLAY
  sets out to produce physically-based rendering (PBR) textures by employing a
  multi-view material diffusion model that can generate 2K resolution textures
  with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for
  a range of controllable 3D asset creations, from sketchy conceptual designs to
  production ready assets with intricate details. Even first time users can
  easily use CLAY to bring their vivid 3D imaginations to life, unleashing
  unlimited creativity.
paper_ai_summary: CLAY, a 3D geometry and material generator, uses a combination of VAE, DiT, and neural fields to produce high-quality 3D models and textures from various inputs, requiring minimal user expertise.
paper_ai_keywords: Multi-resolution Variational Autoencoder (VAE) latent Diffusion Transformer (DiT) neural fields geometry generative module progressive training ultra large 3D model dataset physically-based rendering (PBR) multi-view material diffusion model 2K resolution textures diffuse modalities roughness modalities metallic modalities