title: Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
    Aligned Latent Representation
summary: We present a novel alignment-before-generation approach to tackle the
  challenging task of generating general 3D shapes based on 2D images or texts.
  Directly learning a conditional generative model from images or texts to 3D
  shapes is prone to producing inconsistent results with the conditions because
  3D shapes have an additional dimension whose distribution significantly differs
  from that of 2D images and texts. To bridge the domain gap among the three
  modalities and facilitate multi-modal-conditioned 3D shape generation, we
  explore representing 3D shapes in a shape-image-text-aligned space. Our
  framework comprises two models: a Shape-Image-Text-Aligned Variational
  Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
  (ASLDM). The former model encodes the 3D shapes into the shape latent space
  aligned to the image and text and reconstructs the fine-grained 3D neural
  fields corresponding to given shape embeddings via the transformer-based
  decoder. The latter model learns a probabilistic mapping function from the
  image or text space to the latent shape space. Our extensive experiments
  demonstrate that our proposed approach can generate higher-quality and more
  diverse 3D shapes that better semantically conform to the visual or textural
  conditional inputs, validating the effectiveness of the
  shape-image-text-aligned space for cross-modality 3D shape generation.
paper_authors._id: 649e49ba8f9890dcc266e1b9
paper_authors.name: Zibo Zhao
paper_authors._id: 649e49ba8f9890dcc266e1ba
paper_authors.name: Wen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:32:41.634Z
paper_authors._id: 63198e8802fb322037332f2d
paper_authors.avatarUrl: /avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
paper_authors.fullname: Wen Liu
paper_authors.type: user
paper_authors.user: doubility123
paper_authors._id: 649e49ba8f9890dcc266e1bb
paper_authors.name: Xin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T08:21:46.538Z
paper_authors._id: 63b543b2bda8d44adf2d7d0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: ChrisChenXin
paper_authors._id: 649e49ba8f9890dcc266e1bc
paper_authors.name: Xianfang Zeng
paper_authors._id: 649e49ba8f9890dcc266e1bd
paper_authors.name: Rui Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-12T08:09:22.865Z
paper_authors._id: 630443483926de1f7ec6bda0
paper_authors.avatarUrl: /avatars/e8e95ab0d03f58c61846ccc1318502fe.svg
paper_authors.fullname: Rui Wang
paper_authors.type: user
paper_authors.user: budui
paper_authors._id: 649e49ba8f9890dcc266e1be
paper_authors.name: Pei Cheng
paper_authors._id: 649e49ba8f9890dcc266e1bf
paper_authors.name: Bin Fu
paper_authors._id: 649e49ba8f9890dcc266e1c0
paper_authors.name: Tao Chen
paper_authors._id: 649e49ba8f9890dcc266e1c1
paper_authors.name: Gang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-30T10:23:20.237Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_authors._id: 649e49ba8f9890dcc266e1c2
paper_authors.name: Shenghua Gao
paper_title: Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
    Aligned Latent Representation
paper_summary: We present a novel alignment-before-generation approach to tackle the
  challenging task of generating general 3D shapes based on 2D images or texts.
  Directly learning a conditional generative model from images or texts to 3D
  shapes is prone to producing inconsistent results with the conditions because
  3D shapes have an additional dimension whose distribution significantly differs
  from that of 2D images and texts. To bridge the domain gap among the three
  modalities and facilitate multi-modal-conditioned 3D shape generation, we
  explore representing 3D shapes in a shape-image-text-aligned space. Our
  framework comprises two models: a Shape-Image-Text-Aligned Variational
  Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model
  (ASLDM). The former model encodes the 3D shapes into the shape latent space
  aligned to the image and text and reconstructs the fine-grained 3D neural
  fields corresponding to given shape embeddings via the transformer-based
  decoder. The latter model learns a probabilistic mapping function from the
  image or text space to the latent shape space. Our extensive experiments
  demonstrate that our proposed approach can generate higher-quality and more
  diverse 3D shapes that better semantically conform to the visual or textural
  conditional inputs, validating the effectiveness of the
  shape-image-text-aligned space for cross-modality 3D shape generation.
paper_ai_summary: A novel alignment-before-generation approach using SITA-VAE and ASLDM for generating high-quality 3D shapes from 2D images or texts leverages a shape-image-text-aligned space.
paper_ai_keywords: Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) Aligned Shape Latent Diffusion Model (ASLDM) shape latent space transformer-based decoder semantic conforming