title: CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
summary: This paper aims to design a unified Computer-Aided Design (CAD) generation
  system that can easily generate CAD models based on the user's inputs in the
  form of textual description, images, point clouds, or even a combination of
  them. Towards this goal, we introduce the CAD-MLLM, the first system capable of
  generating parametric CAD models conditioned on the multimodal input.
  Specifically, within the CAD-MLLM framework, we leverage the command sequences
  of CAD models and then employ advanced large language models (LLMs) to align
  the feature space across these diverse multi-modalities data and CAD models'
  vectorized representations. To facilitate the model training, we design a
  comprehensive data construction and annotation pipeline that equips each CAD
  model with corresponding multimodal data. Our resulting dataset, named
  Omni-CAD, is the first multimodal CAD dataset that contains textual
  description, multi-view images, points, and command sequence for each CAD
  model. It contains approximately 450K instances and their CAD construction
  sequences. To thoroughly evaluate the quality of our generated CAD models, we
  go beyond current evaluation metrics that focus on reconstruction quality by
  introducing additional metrics that assess topology quality and surface
  enclosure extent. Extensive experimental results demonstrate that CAD-MLLM
  significantly outperforms existing conditional generative methods and remains
  highly robust to noises and missing points. The project page and more
  visualizations can be found at: https://cad-mllm.github.io/
paper_authors._id: 672da1e862371d59e7eb69b6
paper_authors.name: Jingwei Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-11T18:11:55.179Z
paper_authors._id: 672da19bb2f2dc21e176b0de
paper_authors.avatarUrl: /avatars/a942ad1c467b865cb9530927fe13f2b7.svg
paper_authors.fullname: Jingwei Xu
paper_authors.type: user
paper_authors.user: jingwei-xu-00
paper_authors._id: 672da1e862371d59e7eb69b7
paper_authors.name: Chenyu Wang
paper_authors._id: 672da1e862371d59e7eb69b8
paper_authors.name: Zibo Zhao
paper_authors._id: 672da1e862371d59e7eb69b9
paper_authors.name: Wen Liu
paper_authors._id: 672da1e862371d59e7eb69ba
paper_authors.name: Yi Ma
paper_authors._id: 672da1e862371d59e7eb69bb
paper_authors.name: Shenghua Gao
paper_title: CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
paper_summary: This paper aims to design a unified Computer-Aided Design (CAD) generation
  system that can easily generate CAD models based on the user's inputs in the
  form of textual description, images, point clouds, or even a combination of
  them. Towards this goal, we introduce the CAD-MLLM, the first system capable of
  generating parametric CAD models conditioned on the multimodal input.
  Specifically, within the CAD-MLLM framework, we leverage the command sequences
  of CAD models and then employ advanced large language models (LLMs) to align
  the feature space across these diverse multi-modalities data and CAD models'
  vectorized representations. To facilitate the model training, we design a
  comprehensive data construction and annotation pipeline that equips each CAD
  model with corresponding multimodal data. Our resulting dataset, named
  Omni-CAD, is the first multimodal CAD dataset that contains textual
  description, multi-view images, points, and command sequence for each CAD
  model. It contains approximately 450K instances and their CAD construction
  sequences. To thoroughly evaluate the quality of our generated CAD models, we
  go beyond current evaluation metrics that focus on reconstruction quality by
  introducing additional metrics that assess topology quality and surface
  enclosure extent. Extensive experimental results demonstrate that CAD-MLLM
  significantly outperforms existing conditional generative methods and remains
  highly robust to noises and missing points. The project page and more
  visualizations can be found at: https://cad-mllm.github.io/
paper_ai_keywords: CAD-MLLM parametric CAD models large language models (LLMs) feature space multimodal data vectorized representations CAD construction sequences multimodal CAD dataset topology quality surface enclosure extent