CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
This paper aims to design a unified Computer-Aided Design (CAD) generation
system that can easily generate CAD models based on the user's inputs in the
form of textual description, images, point clouds, or even a combination of
them. Towards this goal, we introduce the CAD-MLLM, the first system capable of
generating parametric CAD models conditioned on the multimodal input.
Specifically, within the CAD-MLLM framework, we leverage the command sequences
of CAD models and then employ advanced large language models (LLMs) to align
the feature space across these diverse multi-modalities data and CAD models'
vectorized representations. To facilitate the model training, we design a
comprehensive data construction and annotation pipeline that equips each CAD
model with corresponding multimodal data. Our resulting dataset, named
Omni-CAD, is the first multimodal CAD dataset that contains textual
description, multi-view images, points, and command sequence for each CAD
model. It contains approximately 450K instances and their CAD construction
sequences. To thoroughly evaluate the quality of our generated CAD models, we
go beyond current evaluation metrics that focus on reconstruction quality by
introducing additional metrics that assess topology quality and surface
enclosure extent. Extensive experimental results demonstrate that CAD-MLLM
significantly outperforms existing conditional generative methods and remains
highly robust to noises and missing points. The project page and more
visualizations can be found at: https://cad-mllm.github.io/
672da1e862371d59e7eb69b6
Jingwei Xu
admin_assigned
2024-11-11T18:11:55.179Z
672da19bb2f2dc21e176b0de
/avatars/a942ad1c467b865cb9530927fe13f2b7.svg
Jingwei Xu
user
jingwei-xu-00
672da1e862371d59e7eb69b7
Chenyu Wang
claimed_verified
2025-04-22T10:17:21.033Z
641bd539aebaa27e07540613
/avatars/000e8057e7280b9eba9bec5116d14718.svg
chenyu wang
user
ch-chenyu
672da1e862371d59e7eb69b8
Zibo Zhao
672da1e862371d59e7eb69b9
Wen Liu
672da1e862371d59e7eb69ba
Yi Ma
672da1e862371d59e7eb69bb
Shenghua Gao
CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
This paper aims to design a unified Computer-Aided Design (CAD) generation
system that can easily generate CAD models based on the user's inputs in the
form of textual description, images, point clouds, or even a combination of
them. Towards this goal, we introduce the CAD-MLLM, the first system capable of
generating parametric CAD models conditioned on the multimodal input.
Specifically, within the CAD-MLLM framework, we leverage the command sequences
of CAD models and then employ advanced large language models (LLMs) to align
the feature space across these diverse multi-modalities data and CAD models'
vectorized representations. To facilitate the model training, we design a
comprehensive data construction and annotation pipeline that equips each CAD
model with corresponding multimodal data. Our resulting dataset, named
Omni-CAD, is the first multimodal CAD dataset that contains textual
description, multi-view images, points, and command sequence for each CAD
model. It contains approximately 450K instances and their CAD construction
sequences. To thoroughly evaluate the quality of our generated CAD models, we
go beyond current evaluation metrics that focus on reconstruction quality by
introducing additional metrics that assess topology quality and surface
enclosure extent. Extensive experimental results demonstrate that CAD-MLLM
significantly outperforms existing conditional generative methods and remains
highly robust to noises and missing points. The project page and more
visualizations can be found at: https://cad-mllm.github.io/
A unified CAD generation system, CAD-MLLM, uses advanced large language models to align multimodal data and generates high-quality parametric CAD models conditioned on textual, image, and point cloud inputs.
large language models multimodal input parametric CAD models multimodal data data construction annotation pipeline multimodal CAD dataset Omni-CAD topology quality surface enclosure extent conditional generative methods